Facing the threat of domestic violence, surviving sexual assault, living with depression, coping with thoughts of suicide: all of these things can have major impacts on a person’s health. But a new study suggests many Americans aren’t talking to their primary health care providers about these important topics.

The study was done using two internet survey companies, Amazon’s Mechanical Turk and Survey Sampling International. Respondents were asked whether they were living with one or more of these threats, and if so, if they had disclosed that information to their healthcare provider. A shocking percentage of the more than 4,500 people surveyed had not: almost half of the participants identified as living with one or more threats, but between 40 and 47.5 percent of all participants said they had withheld that information from their provider.

Most people can probably guess the most significant reasons people gave—embarrassment, not wanting to feel judged or lectured, not wanting a medical intervention that might make things worse, and not wanting the information to appear in their records.

“The thing that is so awesome about a primary care provider—the relationship, the trust-building, seeing each other frequently—can also be a hindrance,” says study author Angela Fagerlin, a psychologist at the University of Utah’s Department of Population Health Sciences. “Because if you admit something bad, you’re going to have to see them over and over and over again.”

These study results can’t tell us the whole story. For one thing, Fagerlin says, she and her colleagues have no way of knowing whether participants’ healthcare providers have actually asked them about these threats outright. Perhaps some of the people surveyed didn’t feel comfortable bringing up the subject of depression or domestic abuse but would have been honest if their doctors had asked. “Any time you do a survey, there are limitations,” says Fagerlin, “especially with internet samples.”

Beyond the fact that the people who took the survey have self-selected to do so, rather than being completely random, on the four sensitive subjects this study looks at, “people might not be disclosing, even to us,” she says.

Still, the results suggest that many people don’t tell their providers about these stigmatized events and circumstances that can have a huge effect on their health, says Kathryn Pollak, a Duke University social psychologist who works on clinician-patient communication. Someone who has been sexually assaulted is at a heightened risk for developing post-traumatic stress disorder and may be in need of an STI or pregnancy test. Someone threatened by domestic abuse may have injuries they allow to go untreated.

“All those things are associated with shame,” says Pollak. People are especially worried about being judged by doctors and other medical professionals, she says, because they’re seen as authority figures. “When I’m coaching doctors, I teach them to literally say, ‘I am not going to judge you,’” Pollak says.

An earlier study using the same survey data suggests many patients fail to disclose diet- and exercise-related concerns to providers as well. Although this information is less sensitive, it adds weight to the idea that doctors and patients need to communicate better.

The primary care relationship is a two-way street, Pollak says, and learning how to help both parties talk to each other means helping everyone feel safe. While doctors need to let patients know they won’t be judged (and actually refrain from passing judgment), patients also need to know this stuff is medically relevant, and their doctor may be able to help. “We want to start to figure out how to change the dialogue,” Fagerlin says. “There’s work that we need to do on both sides.”