On the ethics of that "chocolate sting" study

By now many people must have heard of the so-called "chocolate sting" carried out by scientist and journalist John Bohannon. In a nutshell, what Bohannon did was to carry out a fake study on a very small sample of people that purported to investigate the effects of a chocolate-laced diet on weight. The study was actually perfectly done except for the statistics which were nicely massaged to conform to what's called 'p-hacking', the selective cherry-picking of results to correspond to statistical significance.

Bohannon and his colleagues then published the "study". Perhaps to nobody's surprise, both journalists and popular magazines jumped on it and proclaimed a new era of chocolate-enabled weight loss. Bohannon has written an article about his sting on the website iO9 which is now all over the Internet. The message from the article is that the gullibility of both journalists and the lay public is well and alive when it comes to swallowing incredulous health-related results. The result was especially depressing because one would think that even uninformed people would be skeptical about the supposedly beneficial effects of chocolate on weight.

Personally when I read about the sting and people's reactions to it I was immediately reminded of the Sokal hoax. Granted that the Sokal hoax did not perform experiments on human beings, but it too relied on deception and a sting operation to reveal the dirty truth. In that case the truth pertained to the tendency of people who called themselves "postmodernists" to dress up nonsensical notions in pseudoscientific language and try to present them as serious theories. Sokal wrote a deliberately nonsensical article and sent it to 'Social Text', the leading postmodernist journal. To his surprise it was enthusiastically published and touted as a novel insight into science and human nature. The emperor had been disrobed.

Bohannon's hoax applies to a different field but it is essentially in the same spirit. The goal of the hoax is to show that the emperor of uncritical thinking and statistical ignorance has no clothes. And the one thing the hoax managed to demonstrate is that emperors of this kind are running all over the place, and in fact each one of us has a bit of them rooted in our own thinking.

The study has drawn mixed reactions. For instance Ben Goldacre, slayer of bad science and medical studies, has praised the sting and said that it "deserves a prize" while Seth Mnookin, another slayer of bad science and campaigner against anti-vaxxers, has condemned it and called it "reprehensible". The most common reason why people are condemning the study is because they see no value coming out of it and because they see it as unethical: Bohannon did not seek approval from anything like an institutional review board (IRB) or inform his subjects about the goals of the study and get their approval (although he did lay out the general outline).

I myself got into a lively debate about the study on Twitter with two longstanding members of the science blogging community - Aatish Bhatia who blogs at Wired and Bethany Brookshire who blogs under the name Scicurious. Aatish and Bethany's concerns about the study were the same as those of Mnookin and others: the authors did not get their subjects' consent and approval from an IRB, and the value of the study was marginal at best. The thinking was, there have been several exposes of bad science and journalism in the past few years, so what exactly does this ethically dubious work demonstrate?

Here are my responses to these objections: First of all I agree with Aatish and Bethany that strictly speaking the study is not ethical. But then so aren't hundreds of other studies, especially in areas like psychology where detailed disclosure of the study goals themselves might change people's psychology and thwart those goals. Even the Sokal hoax with its explicit deception of journal editors and thousands of readers was not strictly ethical, and yet it is now regarded as a landmark critique of pseudoscience. In some cases, as long as no harm is being explicitly done to human subjects, it is only by withholding all the details that one can do a truly blinded and clean study. Bohannon's project was as much a psychology project as a nutrition project. In fact strictly speaking even most medical trials are unethical in the sense that they are run some time after the purported drug has shown a beneficial effect relative to the controls. If complete consent to every detail was a required condition for scientific experiments involving human beings, then most research projects in psychology and medicine would be deemed 'unethical.'

My bigger point though is that a discussion of ethics cannot be divorced from consequences, and any assessment of the ethical nature of research has to be done relative to the potential benefits and costs of the research. Viewed through this lens, the Bohannon study acquires a more interesting tinge. The most important benefit of the project would be an inculcation of a sense of skepticism and caution in the laypeople, journalists and magazines which were fooled. Aatish and Bethany are skeptical that such a sense would be ingrained in the victims. At the very least I agree that one would have to do a follow-up study to find out whether people have indeed become more enlightened and cautious and whether a sizable fraction of those who were fooled are now aware of the hoax.

But personally I am more hopeful. While P T Barnum's observation that a sucker is born every minute does ring true, it is also true that people often remember to be circumspect only after they have been shamed or embarrassed into doing so. Sometimes it's only a jolt of reality that can awaken people to reality. Granted that the jolt may be exaggerated, simplistic or disingenuous - in that sense Bohannan's work is what I would call the Michael Moore style of research - but it can lead to results that may not be acquired by a gentler approach. It's not the best approach, should be used sparingly and can be easily seen as cynical, but I would be surprised if it does not work. I would be surprised if at least a few of the magazines which fell hook line and sinker for Bohannon's ruse didn't think twice next time before enthusiastically publishing such results. Even if a few of them turn more skeptical I think the study would have had considerable value.

Another reasonable objection raised was that the ploy might cause people to become too skeptical and lead to a mistrust of even legitimate science from next time onwards. This is a valid point, but my response to it is to ask the question: would we rather err on the side of safety or on the side of caution (ideally we would rather not err at all, but that's a different story)? Aatish's answer was that we should try to err on the side of minimizing harm. I agree with this, but in my view harm is of different kinds. Aatish was presumably talking about the harm done to people's psyche that might cause them to mistrust honest science and journalism, but I believe that the harm done from ignorance of the scientific method and statistics is even greater, that it can lead to an equal or greater erosion of trust in science and rationality. If I were indeed to pick between the sides of overt skepticism and overt gullibility, even with my reservations I would pick the side of overt skepticism.

Ultimately my feeling about this study is that it's the kind of bitter medicine that should be administered occasionally for the health of a rational society. Strictly speaking it's not ethical, but its ethics should be balanced against its consequences. Its liberal use would indeed lead to a jaundiced populace that trusts nothing, but using it once in a while might actually cause a statistically significant increase in that skepticism that we all sorely need. And that's a good thing.