In this year's ScienceOnline conference I co-moderated a productive session on peer review in which I pointed out how overly conservative or agenda-driven peer reviews can prevent the publication of legitimate science. Now here's a case where the opposite seems to have occurred; highly questionable science making it through the filter of peer review as easily as particles of dust would make it through a sieve with penny-sized holes.
Thanks to the Freedom of Information Act, USA Today and a couple of other scientists got their hands on the reviews of the infamous #arseniclife paper. There were three reviewers of the study, and all of them approved the paper for publication.
What's interesting is how effortlessly the reviewers miss the forest for the trees. We of course have the benefit of hindsight here, but it's still striking how all three reviews simply swallow the flawed paper's basic and potentially textbook-changing paradigm - the substitution of arsenic for phosphorus - right off the bat. Once they accept this basic premise, all their other objections can simply be seen as nitpicking and window dressing. Only one reviewer asks questions that come close to questioning the absence of phosphorus in the medium, but even he or she quickly veers off course. Another calls the paper a "rare pleasure" to read, seemingly unaware that the pleasure which the paper has provided comes from an extraordinarily ambitious claim that needs to be vetted as closely as possible.
In fact the reviewers ask good questions about vacuoles seen in the bacterium, about better standards for some of the experiments, about better methods to quantify arsenic in its various forms. They even ask a few very chemical questions regarding bond distances. But all these questions are somewhat beside the point since they flow from a fundamentally flawed belief.
When I was in graduate school, the most important thing that my advisor taught me was to always question the assumptions behind a study. If you don't do this, it's easy to be seduced by the technical details of the experiment and to let these details convince you that the basic premise is validated. That's what seems to me to have happened here. All the reviewers seem to have been sucked into legitimate and interesting questions about minutiae. But all the time they forget that what really needs to be questioned is the giant assumption from which all the minutiae have been derived, an assumption that we now know does not stand up to scrutiny. There's an important lesson here.
Thanks to the Freedom of Information Act, USA Today and a couple of other scientists got their hands on the reviews of the infamous #arseniclife paper. There were three reviewers of the study, and all of them approved the paper for publication.
What's interesting is how effortlessly the reviewers miss the forest for the trees. We of course have the benefit of hindsight here, but it's still striking how all three reviews simply swallow the flawed paper's basic and potentially textbook-changing paradigm - the substitution of arsenic for phosphorus - right off the bat. Once they accept this basic premise, all their other objections can simply be seen as nitpicking and window dressing. Only one reviewer asks questions that come close to questioning the absence of phosphorus in the medium, but even he or she quickly veers off course. Another calls the paper a "rare pleasure" to read, seemingly unaware that the pleasure which the paper has provided comes from an extraordinarily ambitious claim that needs to be vetted as closely as possible.
In fact the reviewers ask good questions about vacuoles seen in the bacterium, about better standards for some of the experiments, about better methods to quantify arsenic in its various forms. They even ask a few very chemical questions regarding bond distances. But all these questions are somewhat beside the point since they flow from a fundamentally flawed belief.
When I was in graduate school, the most important thing that my advisor taught me was to always question the assumptions behind a study. If you don't do this, it's easy to be seduced by the technical details of the experiment and to let these details convince you that the basic premise is validated. That's what seems to me to have happened here. All the reviewers seem to have been sucked into legitimate and interesting questions about minutiae. But all the time they forget that what really needs to be questioned is the giant assumption from which all the minutiae have been derived, an assumption that we now know does not stand up to scrutiny. There's an important lesson here.
It's clear from the the authors' formal response to reviews that the DNA gel evidence (Fig. 2) wasn't even in the original manuscript (it was added at the revision stage).
ReplyDeleteThis gel analysis was certainly flawed, but without it how could the reviewers ever have accepted the claim that the cells had arsenic-backbone DNA?
True, that's really fundamental, so it's even worse than it appears.
Delete