Headlines Impeding Health

Lisa Thornberg / E+/ Getty Images

Not long ago, a science journalist named John Bohannon spilled the cacao beans in a publication called io9 on a scam he perpetrated that is presumably exonerated under terms of ends justifying means.

As he tells it, Dr. Bohannon (he has a PhD in the molecular biology of bacteria) pretended to be a clinical researcher of human nutrition, named Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health- which didn’t exist until John made it up.

He tells the story very well, so I refer you to the original rather than attempting to replicate it here.  Suffice to say he ran a small study of chocolate for weight loss to see how readily a study intended to be bogus could be turned into a barrage of hyperbolic headlines.

His answer is a precautionary tale for us all: very, very easily, indeed.

The study itself did actually suggest a weight loss benefit from 1.5oz per day of dark chocolate.  This, in fact, is not out of the question.  Dark chocolate has nutrient properties that could plausibly influence everything from mood, to satiety, to hormonal balance, and even, perhaps energy expenditure, albeit quite modestly.  Any of these effects might contribute something to weight loss.

But the health benefits of dark chocolate, and the potential extension of those benefits into the realm of weight control, are not today’s topic.  Today’s topic is the impediment to health that occurs at the interface of weak science and strong headlines.

Bohannon and colleagues willfully ran a very weak study. They did not concern themselves with sample size, statistical power, controlling for multiple comparisons, or controlling for confounders.  There are methods for all of these, of course, and they are routinely built into any good study from the start, and subject to the scrutiny of peer review before publication in any reputable journal.

Bohannon and team neglected all of these vital concerns on purpose, and wound up with a study that showed a weight loss benefit from the daily addition of chocolate.  They went on to publish the paper in a journal claiming to be peer-reviewed, but requiring a fee for publication.  The paper was published as soon as the payment was made, far too fast for peer review- indicating that the claims of peer review were dubious, if not an outright confabulation.

And then, predictably, because the study was about not one media darling, but two (chocolate, and weight loss)- efforts to get media coverage were richly, and promptly rewarded.  Some headlines went as far as, “why you must eat chocolate daily.”

Bohannon did us all a favor by demonstrating two failings in the system we rely on to disseminate biomedical information: the generation of credible information in the first place, and the responsible translation of such science into the media of popular culture.  But since none of us has the power to shut down irresponsible journals feigning peer review, or to prohibit hyperbolic headlines, how do we apply the lesson?

There are two strategies that will serve us all well. 

The first is to recognize that science and sense are codependent, not mutually exclusive.  There are no good answers to bad questions, and sense is required to distinguish good questions from bad.  We are all advised to recall that what sounds too good to be true almost certainly is; and that the standards of science required to convince us should be the same for messages we don’t want, as for those we do.  I am impressed at how people who routinely fuss about methodologic deficiencies in research they don’t like, overlook exactly those same deficiencies in research they do like.  We all need to recognize, and manage, our native bias. 

Second, it is time to recognize that science is much more like the flow of a river into the ocean than it is like a game of Ping Pong.  We treat science like Ping Pong- with each swing of the news cycle reversing everything we thought we knew so far- at our peril.  Science slowly accumulates, and it is the weight of evidence that matters most, not the evidence that happened to generate the most recent headlines.  What we know most reliably, we know because it is a product of many studies replicating similar findings.  It is a product of diverse methods, diverse populations, and diverse research teams.  It works in the real world, stands the test of time, and runs on assessable mechanisms.

On the basis of just such evidence and consensus, we know what works for both controlling weight and promoting health.  Our prevailing problem is not that we don’t know what we should eat; it’s that we are disinclined to swallow it until it is dipped in chocolate, and turned into a confection.

There is, apparently, a thriving cottage industry selling just such candy where science-based guidance ought to be.  Approach science with sense, and what they’re selling can’t hurt you- because you don’t have to buy it.


Continue Reading