On the impact of social media and Twitter on scientific peer review

I am very pleased to note that an my article on the impact of social media and especially of blogs and Twitter on peer review in chemistry in particular and science in general has just come out in a special issue of the journal 'Accountability in Research'. This project has been in the works for almost a year and I have spent quite a bit of time on it. The whole issue is open access and it was made possible by the dedicated and generous efforts of my colleague and friend, the eminent historian of chemistry Jeff Seeman. I am privileged to have my article appear along with those by Roald Hoffmann, William Schulz, Jeffrey Kovac and Sandra Titus. All their papers are highly readable.

Here in a nutshell is what I say. I have had a very dim view of Twitter recently as a vehicle for cogent science communication and rational debate, but in this article I find myself full of praise for the medium. This sentiment has been inspired by the use of Twitter in recent times for demolishing careless science and questioning shoddy or controversial papers in the scientific literature. In my opinion the most spectacular use of Twitter to this effect was Nature Chemistry editor Stuart Cantrill's stark highlighting of 'self-plagiarism' in a review article published by Ronald Breslow in JACS in 2012 (I hold forth on the concept of self-plagiarism itself in the article). As I say in my piece, to my knowledge this is the first and only instance I know in which Twitter - and Twitter alone - was used to point our errors in a paper published in a major journal. If Cantrill's analysis was not a resounding example of peer review in the age of social media, I don't know what is.

I have had a much more consistent and positive views of blogs as tools for instant and comprehensive peer review, and thanks to the vibrant chemistry blogosphere that I have been lucky to be a part of for almost eleven years, have witnessed the true coming of age of this medium. There is no doubt that peer review on blogs is here to stay, and in my article I address the pitfalls and promises inherent in this development. One of the most important concerns that a naive observer would have regarding the use of blogs or Twitter for peer review is the potential for public shaming and ad hominem attacks - and such an observer would find plenty of recent evidence in the general Twittersphere to support their suspicions. Yet I argue that, at least as far as the limited milieu of chemistry blogs is concerned, the signal to noise ratio has been very high and the debate remarkably forward-thinking and positive; in fact I think that, by and large, chemistry blogs could serve as models of civil and productive debate for blogs on more socially or politically contentious topics like evolution and climate change. I am proud to be part of this (largely) civil community.

What I aim to do in this piece is to view the positive role of Twitter and blogs in effecting rapid and comprehensive peer review through the lens of three major case studies which would be familiar to informed observers: the debacle of 'arsenic life', the fiasco of hexacyclinol and the curious case of self-plagiarism in the Breslow 'space dinosaurs' review. In each case I point out how blogs and Twitter were responsible for pointing out mistakes and issues with the relevant material far faster than official review ever could and how they circumvented problems with traditional peer review, some obvious and some more structural. The latter part of the review raises questions about the problems and possibilities inherent in the effective use of these tools, and I muse a bit about how the process could be made fairer and simpler.

Due to the sheer speed with which blogs and social media can turn our collective microscopes on the scientific literature and the sheer diversity of views which can be instantly brought to bear on a contentious topic, there is no doubt in my mind that this new tier of scientific appraisal is here to stay. In my opinion the future of completely open peer review is bright and beckons. How it can complement existing modalities of 'official' peer review is an open question. While I raise this question and offer some of my own thoughts I claim to provide no definitive answers. Those answers can only be provided by our community.

Which brings me to the crux of the article: although my name is printed on the first page of the piece it really is of, by and for the community. Hope there will be something of interest to everyone in it. I welcome your comments.

Solomon Snyder on academic publishing: ask for adequate, not exhaustive, documentation

Image: Corpus Callosum

Renowned neuropharmacologist Solomon Snyder has a thought-provoking take on what seems to be one of the two evils that has plagued modern academia: publication (the other one is the job market). I have previously blogged about the increasing conservatism of academic publishing myself, and in this case “conservatism” also translates to “excessive rigor”.

Snyder starts by lamenting the startling fact that the average duration for a modern American biomedical scientist to start his or her academic career is about the same as that for a neuro or cardiovascular surgeon, people whose specialty is usually considered to be in the top tier of their profession; the difference of course is that a cardiovascular surgeon starts making $500K right off the bat while a new assistant professor starts making $80K and almost never goes beyond $200k or so. The long trudge begins with graduate education, the average duration of which has stretched out over the last three decades (these days, a 5 year Ph.D. is considered relatively quick). Every part of the academic process, from getting a postdoctoral position to your first job to your first grant, has turned into a war of attrition. The “winners” who emerge at the end of it are often demoralized academics in their early 40s whose best years may be behind them. And the situation seems to only be getting worse.

But the article’s really about publishing papers. Snyder hits the nail when he says that academic publishing has become so rigorous in asking for exhaustive experimentation and documentation that it dissuades many authors from publishing their best ideas, ideas which are interesting and valid but which may not have been completely fleshed out. He points to reviewers’ insistence that authors perform a comprehensive set of experiments – often ranging over several months – that would qualify their manuscript for publication. Anyone who has tried to publish biomedical papers must be well aware of how tedious and demoralizing the experience can be. This long-drawn process significantly impacts the progress of science:

“Why does it take so much longer to move from test tube to the printed page? One element is a journal review process that is substantially lengthier, especially in terms of experiments required to address the concerns of referees. To anticipate such referee responses, scientists preemptively carry forward experimentation more exhaustively than is necessary to document their assertions. Yet, we can clone genes in a couple of days. Shouldn’t we be able to complete experiments to satisfy reviewers in a few weeks rather than the 7–12 months typically consumed in revision, not to mention the many years devoted to developing the original manuscript? If one spends 5 years accumulating the data for a manuscript and another year revising it to satisfy referees, benefits to the public are delayed for years.”

In contrast Snyder points to his postdoctoral advisor, another legendary scientist named Julius Axelrod at the NIH who churned out discovery after discovery in short order and won a Nobel Prize (the Axelrod dynasty is nicely charted out in Robert Kanigel’s book “Apprentice to Genius”). The point that Snyder is making is that in those days the reviewing process was much quicker but the quality of science doesn’t seem to have suffered in spite of this speedier turnaround. What has gone wrong since then?

Snyder partially places the blame at the feet of Cell founder Benjamin Lewin who wanted Cell to showcase papers that were essentially complete stories; from hypotheses to final products. But Lewin also made the process highly streamlined. Reviewers were warned to stay away from insults, stick to succinct criticism and suggest adequate but not unrealistic experiments and further studies. The objective was to get the best science out in a form that was interesting enough to spark further inquiry but which was not necessarily the last word.

Lewin understood the piecemeal nature of science where researchers build on each other’s discoveries. This understanding of the scientific process has since been subverted by academic reviewers, partially to cull a flood of proposals and ideas and partially to satisfy their own whims. Sometimes old boys’ networks can conspire to put sound science in a straitjacket. Expecting every research project to tell complete, final stories not only imposes unrealistic and demotivating standards on scientists but also ignores the always incomplete and provisional nature of science. Snyder asks that expectations for accepting papers be changed and points to recent developments like the journal eLIFE which incorporates some of his thinking. Blogger SciCurious suggests her own system of peer-review where a paper is simultaneously sent to a group of journals with different standards; after hearing back from reviewers, the authors can decide whether to push ahead with further experiments to satisfy the top-tier journals or whether to publish the paper in a lower-tier journal right away. But Snyder’s perspective points out that all journals – whether top tier or otherwise – should have a reviewing system that allows for rapid dissemination of results.

Reviews and authors need to seriously contemplate Snyder’s recommendations. Academic research has already turned into a long slog with its uncertain job market and draconian grant approval and does not to face need additional difficulties in the form of glacial and unrealistic reviewing standards. Let’s remember that the purpose of science is to generate ideas, not products. And it shouldn’t take very long for ideas to see the light of day.

First published on Scientific American Blogs.

#Arseniclife reviews: Missing the forest for the trees

In this year's ScienceOnline conference I co-moderated a productive session on peer review in which I pointed out how overly conservative or agenda-driven peer reviews can prevent the publication of legitimate science. Now here's a case where the opposite seems to have occurred; highly questionable science making it through the filter of peer review as easily as particles of dust would make it through a sieve with penny-sized holes.

Thanks to the Freedom of Information Act, USA Today and a couple of other scientists got their hands on the reviews of the infamous #arseniclife paper. There were three reviewers of the study, and all of them approved the paper for publication.

What's interesting is how effortlessly the reviewers miss the forest for the trees. We of course have the benefit of hindsight here, but it's still striking how all three reviews simply swallow the flawed paper's basic and potentially textbook-changing paradigm - the substitution of arsenic for phosphorus - right off the bat. Once they accept this basic premise, all their other objections can simply be seen as nitpicking and window dressing. Only one reviewer asks questions that come close to questioning the absence of phosphorus in the medium, but even he or she quickly veers off course. Another calls the paper a "rare pleasure" to read, seemingly unaware that the pleasure which the paper has provided comes from an extraordinarily ambitious claim that needs to be vetted as closely as possible.

In fact the reviewers ask good questions about vacuoles seen in the bacterium, about better standards for some of the experiments, about better methods to quantify arsenic in its various forms. They even ask a few very chemical questions regarding bond distances. But all these questions are somewhat beside the point since they flow from a fundamentally flawed belief.

When I was in graduate school, the most important thing that my advisor taught me was to always question the assumptions behind a study. If you don't do this, it's easy to be seduced by the technical details of the experiment and to let these details convince you that the basic premise is validated. That's what seems to me to have happened here. All the reviewers seem to have been sucked into legitimate and interesting questions about minutiae. But all the time they forget that what really needs to be questioned is the giant assumption from which all the minutiae have been derived, an assumption that we now know does not stand up to scrutiny. There's an important lesson here.