On the impact of social media and Twitter on scientific peer review

I am very pleased to note that an my article on the impact of social media and especially of blogs and Twitter on peer review in chemistry in particular and science in general has just come out in a special issue of the journal 'Accountability in Research'. This project has been in the works for almost a year and I have spent quite a bit of time on it. The whole issue is open access and it was made possible by the dedicated and generous efforts of my colleague and friend, the eminent historian of chemistry Jeff Seeman. I am privileged to have my article appear along with those by Roald Hoffmann, William Schulz, Jeffrey Kovac and Sandra Titus. All their papers are highly readable.

Here in a nutshell is what I say. I have had a very dim view of Twitter recently as a vehicle for cogent science communication and rational debate, but in this article I find myself full of praise for the medium. This sentiment has been inspired by the use of Twitter in recent times for demolishing careless science and questioning shoddy or controversial papers in the scientific literature. In my opinion the most spectacular use of Twitter to this effect was Nature Chemistry editor Stuart Cantrill's stark highlighting of 'self-plagiarism' in a review article published by Ronald Breslow in JACS in 2012 (I hold forth on the concept of self-plagiarism itself in the article). As I say in my piece, to my knowledge this is the first and only instance I know in which Twitter - and Twitter alone - was used to point our errors in a paper published in a major journal. If Cantrill's analysis was not a resounding example of peer review in the age of social media, I don't know what is.

I have had a much more consistent and positive views of blogs as tools for instant and comprehensive peer review, and thanks to the vibrant chemistry blogosphere that I have been lucky to be a part of for almost eleven years, have witnessed the true coming of age of this medium. There is no doubt that peer review on blogs is here to stay, and in my article I address the pitfalls and promises inherent in this development. One of the most important concerns that a naive observer would have regarding the use of blogs or Twitter for peer review is the potential for public shaming and ad hominem attacks - and such an observer would find plenty of recent evidence in the general Twittersphere to support their suspicions. Yet I argue that, at least as far as the limited milieu of chemistry blogs is concerned, the signal to noise ratio has been very high and the debate remarkably forward-thinking and positive; in fact I think that, by and large, chemistry blogs could serve as models of civil and productive debate for blogs on more socially or politically contentious topics like evolution and climate change. I am proud to be part of this (largely) civil community.

What I aim to do in this piece is to view the positive role of Twitter and blogs in effecting rapid and comprehensive peer review through the lens of three major case studies which would be familiar to informed observers: the debacle of 'arsenic life', the fiasco of hexacyclinol and the curious case of self-plagiarism in the Breslow 'space dinosaurs' review. In each case I point out how blogs and Twitter were responsible for pointing out mistakes and issues with the relevant material far faster than official review ever could and how they circumvented problems with traditional peer review, some obvious and some more structural. The latter part of the review raises questions about the problems and possibilities inherent in the effective use of these tools, and I muse a bit about how the process could be made fairer and simpler.

Due to the sheer speed with which blogs and social media can turn our collective microscopes on the scientific literature and the sheer diversity of views which can be instantly brought to bear on a contentious topic, there is no doubt in my mind that this new tier of scientific appraisal is here to stay. In my opinion the future of completely open peer review is bright and beckons. How it can complement existing modalities of 'official' peer review is an open question. While I raise this question and offer some of my own thoughts I claim to provide no definitive answers. Those answers can only be provided by our community.

Which brings me to the crux of the article: although my name is printed on the first page of the piece it really is of, by and for the community. Hope there will be something of interest to everyone in it. I welcome your comments.

Arsenic DNA, chemistry and the problem of differing standards of proof in cross-disciplinary science

Arsenic-based linkages in DNA would be unstable and would quickly break, a fact suspected by chemists for years (Image: Johannes Wilbertz)

When the purported discovery of the now infamous “arsenic DNA” bacteria was published, a friend of mine who was studying astrobiology could not stop praising it as an exciting scientific advance. When I expressed reservations about the discovery mainly based on my understanding of the instability of biomolecules containing arsenic, she gushed, “But of course you will be skeptical; you are an organic chemist!"

She was right. As chemists me and many of my colleagues could not help but zero in on what we thought was the most questionable aspect of the whole discovery; the fact that somehow, contrary to everything we understood about basic chemistry, the “arsenic DNA” inside the bacteria was stably chugging along, replicating and performing its regular functions.

It turned out that the chemists were right. Measurements on arsenic DNA analogs made by researchers several months later found that the arsenic analogs differed in stability from their phosphate versions by a mind-boggling factor of 10¹⁷. Curiously, physicists, astronomers, geologists and even biologists were far more accommodating about the validity of the discovery. For some reason the standards used by these scientists were different from those used by chemists, and in the end the chemists’ standard turned out to be the “correct” one. This is not a triumph of chemists and a blemish on other sciences since there could well be cases where other sciences might have used the correct standards in nailing down the truth or falsehood of an unprecedented scientific finding.

The arsenic DNA fiasco thus illustrates a very interesting aspect of modern cross-disciplinary science – the need to reconcile what can be differing standards of evidence or proof between different sciences. This aspect is the focus of a short but thought-provoking piece by Steven Benner, William Bains and Sara Seager in the journal Astrobiology.

The article explains why it was that standards of proof that were acceptable to different degrees to geologists, physicists and biologists were unacceptable to chemists. The answer pertains to what we call “background knowledge”. In this case, chemists were compelled to ask how DNA with arsenic replacing phosphorus in its backbone could possibly be stable given everything they knew about the instability of arsenate esters. The latter had been studied for several decades, and while arsenic DNA itself had not been synthesized before, simpler arsenate esters were known to be highly unstable in water. The chemists were quite confident in extrapolating from these simple cases to questioning the stable existence of arsenic DNA; if arsenic DNA indeed were so stable, then almost everything they had known about arsenate esters for fifty years would have been wrong, a possibility that was highly unlikely. Thus for chemists, arsenic DNA was an extraordinary claim. And as Carl Sagan said, they needed to see extraordinary evidence before they could believe it, evidence that was ultimately not forthcoming.

For geologists however, it was much easier to buy into the claims. That is because as the article points out, there are several cases where elements in minerals are readily interchanged for other elements in the same column of the periodic table. Arsenic in particular is known to replace phosphorus in rocks bearing arsenate and phosphate minerals. Unlike chemists, geologists found the claim of arsenic replacing phosphorus quite consistent with their experiences. Physicists too bought readily into the idea. As the authors say, physicists are generally tuned to distinguishing two hypotheses from one another; in this case the hypothesis that DNA contains arsenic versus the hypothesis that it does not. The physicists thus found the many tests apparently indicating the presence of arsenate in the DNA to provide support for one hypothesis over another. Physicists did not appreciate that the key question to ask would be regarding the stability of arsenic DNA.

Like chemists biologists were also skeptical. Biologists usually check the validity of a claim for a new form of life by comparing it to existing forms. In this case, when the genetic sequence and lineage of the bacteria were inspected they were found to be very similar to garden variety, phosphate-containing bacteria. The biologists’ background knowledge thus compelled them to ask how it could possibly be that a bacterium that was otherwise similar to other existing bacterium could suddenly survive on arsenic instead of phosphorus.

In the end of course, none of the duplicated studies found the presence of arsenic in the GFAJ-1 bacteria. But this was probably the least surprising to chemists. The GFAJ-1 case thus shows that different sciences can have different standards for what they regard as “evidence”. What may be suitable for one field may be controversial or unacceptable for others. This fact helps answer at least one question for the GFAJ-1 paper: Why was it accepted in a prestigious journal like Science? The answer almost certainly concerns the shuttling of the manuscript to planetary scientists rather than chemists or biologists as reviewers. These scientists had different standards of evidence, and they enthusiastically recommended publication. One of the key lessons here is that any paper on cross-disciplinary topics must be sent to at least one specialist from each discipline comprising the field. Highly interdisciplinary fields like astrobiology, drug discovery, and social psychology are prime candidates for this kind of a policy.

Discipline-dependent standards of proof not only explain how occasionally bad science gets published or how promising results get rejected but it also goes into the deeper issue of what in fact constitutes “proof” in science. This question reminds me of the periodic debates about whether psychology or economics is a science. The fact is that many times the standard of proof in psychology or economics might be unacceptable to a physicist or statistician. As a simple example, it is often impossible to get correlations of better than 0.6 in a psychological experiment. And yet such standards can be accepted as proof in the psychological community, partly because an experiment on human beings is too complex to get more accurate numbers; after all, most human beings are not inclined planes or balls dropped from a tower. In addition one may not always need accurate correlations for discerning valuable trends and patterns. Statistical significance may not always be related to real world significance (researchers running clinical trials would be especially aware of this fact).

The article by Benner, Bains and Seager concludes by asking how conflicting standards of proof can be reconciled in highly cross-disciplinary sciences, and this is a question which is going to be increasingly important in an age of inherently cross-disciplinary research.

I think the GFAJ-1 fiasco itself provides one answer. In that case the most “obvious” objection was raised by chemists based on years of experience. In addition it was a “strong” objection in the sense that it really raised the stakes for their discipline; as noted before, if arsenic DNA exists then much of what chemists know about elementary chemical reactivity might have to be revised. In that sense it was really the kind of falsifiable, make-or-break test advocated by Karl Popper. So one cogent strategy might be to first consider these strong, obvious objections, no matter what discipline they may arise from. If a finding passes the test of these strong objections, then it could be subjected to less obvious and more relaxing criteria provided by other disciplines. If it passes every single criterion across the board then we might actually be able to claim a novel discovery, of the kind that rarely comes along and advances the entire field.

First published on the Scientific American Blog Network.

#Arseniclife reviews: Missing the forest for the trees

In this year's ScienceOnline conference I co-moderated a productive session on peer review in which I pointed out how overly conservative or agenda-driven peer reviews can prevent the publication of legitimate science. Now here's a case where the opposite seems to have occurred; highly questionable science making it through the filter of peer review as easily as particles of dust would make it through a sieve with penny-sized holes.

Thanks to the Freedom of Information Act, USA Today and a couple of other scientists got their hands on the reviews of the infamous #arseniclife paper. There were three reviewers of the study, and all of them approved the paper for publication.

What's interesting is how effortlessly the reviewers miss the forest for the trees. We of course have the benefit of hindsight here, but it's still striking how all three reviews simply swallow the flawed paper's basic and potentially textbook-changing paradigm - the substitution of arsenic for phosphorus - right off the bat. Once they accept this basic premise, all their other objections can simply be seen as nitpicking and window dressing. Only one reviewer asks questions that come close to questioning the absence of phosphorus in the medium, but even he or she quickly veers off course. Another calls the paper a "rare pleasure" to read, seemingly unaware that the pleasure which the paper has provided comes from an extraordinarily ambitious claim that needs to be vetted as closely as possible.

In fact the reviewers ask good questions about vacuoles seen in the bacterium, about better standards for some of the experiments, about better methods to quantify arsenic in its various forms. They even ask a few very chemical questions regarding bond distances. But all these questions are somewhat beside the point since they flow from a fundamentally flawed belief.

When I was in graduate school, the most important thing that my advisor taught me was to always question the assumptions behind a study. If you don't do this, it's easy to be seduced by the technical details of the experiment and to let these details convince you that the basic premise is validated. That's what seems to me to have happened here. All the reviewers seem to have been sucked into legitimate and interesting questions about minutiae. But all the time they forget that what really needs to be questioned is the giant assumption from which all the minutiae have been derived, an assumption that we now know does not stand up to scrutiny. There's an important lesson here.