In late March, a team of German-based researchers published a study that found that eating a bar of chocolate every day, when combined with a low-carb diet, helped participants lose weight. As is typical for a study like this, the story got picked up by a number of service-based publications in Europe and the U.S., doing write-ups with splashy headlines and suggestive images of women eating chocolate.
But unlike many similar studies, this one was intentionally fake. John Bohannon is a science journalist with a PhD, focused on the molecular biology of bacteria. He teamed up with German documentary filmmakers to conduct a purposely terrible study and see how far it would go. Bohannon published his account of the scheme yesterday on iO9.
The study was real, and the results were real. But Bohannon and his collaborators used a bit of scientific trickery to come to those conclusions:
In other words, a study that uses a small sample size and many possible metrics means that the odds are stacked towards finding a result that is statistically significant; according to Bohannon’s calculations, they had a 60 percent chance of finding some result that was statistically significant.
The journal that published the study clearly didn’t send it out for peer review, publishing it two weeks after charging its authors about $650. The press release that Bohannon wrote didn’t mention any of the bad science and, when media outlets picked up the story, Bohannon writes that no one questioned him.
For Bohannon, the takeaway is that the conflicting information readers constantly find in even the most trustworthy publications makes it nearly impossible for them to know what to believe. The truth is that the science is far from settled. Reporters who write about dietary science need to be paying more attention to the actual science, Bohannon writes. They need to know how to read a scientific paper so that they can make sure that what they recommend to their readers has been appropriately vetted.
In this case, the readers did some of the vetting the reporters should have done, posting skeptical comments below many of the enthusiastic articles, Bohannon notes.
But in the time since Bohannon published his account, a number of leading science journalists have taken to social media to condemn his actions.
For science journalists, the heart of the matter is that Bohannon “fooled millions of people,” as he touts in his iO9 headline. “Peeing in the reservoir,” as some have called it, doesn’t illuminate the problem–it creates a whole different one.
In the midst of so much negative attention, the journal that published Bohannon’s paper said that it was only published by mistake and has retracted it.
