The anatomy of peer review: Why airing dirty laundry in public is important

Since everyone is talking about Ron Breslow, I thought I might bring readers' attention to a truly fascinating (from the point of view of the sociology of science) article published in Nature in 1992 regarding two of Breslow's papers. The article was written by Prof. Fred Menger (Emory) and Prof. Albert Haim (Stony Brook) and details what happened when the two tried to publish a rebuttal addressing errors in two Breslow articles from the Journal of the American Chemical Society (JACS).

First I want to emphasize the reason why I am writing about this episode. I write about it not because I want to add to Breslow's troubles. I don't want it to sound like I am kicking someone when they are down, nor do I think that I will ever have the capability to do this to someone like Breslow. As I have reiterated in other places, neither this account nor the recent controversy should blind us to an unambiguous fact; Breslow's contributions to research, service and education have been outstanding by any standards. Most scientists would be lucky if they could accomplish half of what he has done during an unusually long and productive career that has still not slowed down. We can all hope that people will continue to remember him by his superlative accomplishments.

So I am not writing this post because I want to add to further criticism of Breslow. It's because I have always thought that the Nature article is a unique document in the history and sociology of chemistry, a great illustration of the pitfalls and promises of peer review that deserves a wider audience. I think laymen (who don't have journal access) will find its contents very interesting, and there's lessons in there for fellow scientists too. I do want to add a disclaimer: Prof. Menger was on my Ph.D. committee and I have enormous respect for his research, but as someone interested in the way science functions, this paper would have been equally fascinating to me even if I had absolutely no connection to him. There were several other papers related to this incident, but the one in Nature stands out as a unique example of scrutiny.

The article is one of the most remarkable publications I have ever read, for two reasons. Firstly, because it presents a rare glimpse into what we might call the "anatomy" of peer review; and it does this in excruciating detail, with warts and all exposed. Secondly, because it makes you wonder whether a journal like Nature would ever again have the inclination to publish something like this.

The whole episode germinated when Breslow and a pair of graduate students (Eric Anslyn and Deeng-Lih Huang) published two papers in JACS dealing with the kinetics of an imidazole-catalysed nucleotide cleavage reaction. This is standard stuff in physical organic and bioorganic chemistry and is part of a large body of work going back 50 years. What was interesting was that the authors seemed to derive negative rate constants for some of the reactions. Now, even college students will recognize this as odd; a rate constant is supposed to indicate the speed of a reaction. If it's positive, the reaction proceeds. If it's zero, the reaction halts. But what is a negative rate constant supposed to mean? Of course the original authors had their own interpretation of their numbers. Independently, Menger and Haim investigated this question and they found out a few rather significant problems with the two papers. The technical details can only be appreciated by a physical organic chemist, but the main problem seemed to be in treating the background reactions with water in the absence of imidazole. It seemed that at least some of the "negative" rate constants were artifacts of the data fitting.

The publication saga started when Menger and Haim independently submitted papers to JACS criticizing the original paper and offering corrections. The editor of JACS at the time was Alan Bard, an internationally renowned chemist who served a distinguished stint as journal editor for several years. Menger's paper was rejected by the reviewers with some odd and rather self-contradictory commentary. On one hand, the reviewers acknowledged the errors in the Breslow papers, but on the other hand they inexplicably chose to reject the manuscript, "hoping" that Breslow and Huang would publish a more detailed explanation and correction. It wasn't clear why they would not let Menger himself publish a correction in the journal.

Haim's paper was also rejected by the journal with similar comments. At this point both Menger and Haim wrote to Bard and the associate editor in charge of the manuscripts. They appealed to JACS's editorial policy which encouraged the submission of manuscripts detailing major errors in published material. This appeal did not have much effect. Something strange also transpired at this juncture; as Menger details, Breslow sent a rather interesting note to Haim, saying: "I don't know what you are so excited about. Are you being led astray by a notoriously unstable individual?". 

Following their unsuccessful attempt at publication in JACS, both Menger and Haim did what scientists always do; try to publish in other places. Haim first send his manuscript to the Journal of Physical Chemistry (JPC). JPC's response was about as strange as JACS's; while acknowledging the problems with the original papers, they too chose not to publish Haim's manuscript. Menger in turn sent his paper to the Journal of Organic Chemistry (JOC). He seems to have found a much more sympathetic audience in the journal's chief and associate editors, Clayton Heathcock and Andrew Streitwieser, both leading researchers. The paper was accepted without much ado by Streitwieser who wondered how the original paper had made it past the JACS referees.

The story does not end there. Menger's JOC paper stimulated a number of largely favorable responses. An especially noteworthy response was from a Nobel Laureate, who sent a note to Bard saying, 

"It seems to me there is a long-established scientific etiquette which says that papers pointing out errors should be published in the same journal in which the original article appeared". 

But these encouraging exchanges were followed by a few calls for retraction of the JOC paper, one of these being from Breslow. At least some of these raised legitimate scientific questions. This prompted Heathcock to ask for clarification, which Menger duly provided. Streitwieser accepted Menger's responses and the paper was published. To the journal's credit, they even permitted a footnote in which Menger said that his attempts to publish his paper in JACS had been rejected.

Meanwhile, Haim was still trying to publish his correction in JACS. He sent a revised, more thorough manuscript to Bard, asking that the original associate editor who had rejected the manuscript be replaced by a fresh pair of eyes. Bard did not agree to this request, but asked Haim to recommend ten referees for the paper. The article was sent to four of these reviewers. Two of the reviewers again agreed that the original science contained errors but again asked that Breslow and his collaborators be given a chance to publish corrections. One reviewer's response was especially interesting; he or she thought that the matter should simply be dropped, ostensibly to protect the reputation of the author:

"It could lead to a reputation, rightly or wrongly, of the author being a nitpicker and Breslow would certainly fight back loudly. Who needs such things"

I find the part about being a nitpicker especially interesting. Firstly, the criticism was not just about a few minor details; it was about rather fundamental analyses and conclusions in the paper. But more importantly, a lot of science in fact is nitpicking because it's through nitpicking that one often uncovers the really important things. Science especially should provide a welcome refuge for nitpickers. 

In any case, after yet another rejection, Haim submitted another revised manuscript. It's worth noting that most reviewers' comments during the 11 months that Haim had tried to get his manuscript published had been favorable, and nobody had ever called Haim's basic analysis into serious question. Yet the paper kept on getting rejected for various reasons. Finally, Haim appealed to Bard and the paper suddenly and inexplicably got published.

Menger ends his account of the long saga with the following words:

"As the dust settles, it is comforting to reflect that the system ultimately worked. After all, both of us succeeded in getting our papers published. Yet this was accomplished only at the cost of considerable anguish to us. Few people, we presume, would be willing to go through this experience...Two problems are involved here. First is the mishandling of the original publications, which many people have come to regard as substandard. The second is the position taken by the associate editor after flaws were pointed out. That position can only be described as evasive and defensive. Without attributing motivation for his actions, we simply state that we believe them to have been inimical to the best interests of science"

He ends by hoping that the fear of open criticism would encourage scientists to police themselves better (At this point science bloggers should let out a collective hurrah, for reasons that will become apparent below). There is a postscript: Breslow replied to this lengthy report shortly (and gratifyingly, his response was published in the same journal) and described experiments that would clarify his earlier work, but he did not address the many questions about peer review raised in Menger's communication.

This fascinating account raises many important issues. For one thing, it was quite clear that the original paper had problems; even the reviewers consistently agreed with this part of the story. Thus the science seems pretty clear, and the ambiguity in the situation came from the human element. We will never know what went behind the scenes when the manuscripts were rejected even after the reviewers agreed with the rebuttal. Unfortunately motivations are hard to unravel, but one cannot help but suspect that there was prestige and influence at work here which thwarted efficient and open scientific revision. The fact that powerful people from the chemistry community (especially Breslow and Bard) were involved cannot be an inconsequential factor.

Secondly, it does seem important to me (although this is a relatively minor issue in my mind) for journals to publish corrections to papers in their own pages; at the very least, this underscores a culture of responsibility on the part of the journal and sends out a positive message. However this practice involves some interesting operational questions. Should the journal first allow the original authors to publish a correction? If so, how long should it wait before doing this? It seems clear to me that legitimate corrections should immediately be published, irrespective of the source.

The most remarkable fact about this account is that Nature published it, and in writing it Menger performed a unique and valuable public service. Personally I have never seen such a detailed dissection of peer review described in a major journal. Some people would deplore this public airing of dirty laundry. They would say that none of this can undo what happened, and the only effect of such articles is bad blood and destroyed reputations. I happen to disagree. I think journals should occasionally publish such analyses, because it alerts us to the very human aspect of science. It demonstrates to the public what science is truly like, how scientists can make mistakes, and how they can react when they are corrected or challenged. It sheds important light on the limitations of the peer review process, but also reaffirms faith in its ultimately self-correcting nature. Some people might think that this is a great example of how peer review should not be, but I would like to think that this is in fact exactly how the process works in the vast majority of cases; imperfect, ambiguous, influenced by human factors like reputations, biases and beliefs. If we want to understand science, we need to acknowledge its true workings instead of trying to fit it into our idealized worldview of perfect peer review.

In this day and age, blogs are performing the exact same function as Nature did in 1992, and this is clearly apparent from the latest Breslow brouhaha. Menger and Haim in 2012 would not have to test their patience by trying to publish in JACS for 11 months; instead they could upload their correction on a website and let the wonder of instant online dissemination work its magic. Blogs may not yet be as respectable as JACS, but the recent incident shows that they can be perfectly respectable outlets of criticism as long as the criticism is fair and rigorous. The growing ascendancy of blogs and their capacity to inflict instant harm on sloppy or unscrupulous science should hopefully result in much better self-policing, leading authors to be more careful about what they publish in "more respectable" venues. Thus, quite paradoxically, blogs could lead to the publication of better science in the very official sources which have largely neglected them until now. This would be a delightful irony.

Perhaps the greatest message that the public can take home from such incidents is that even great scientists can make mistakes and remain great scientists, and that science continues to progress in one way or another. No matter how bad this kind of stuff sounds, it's actually business as usual for the scientific process, and there's nothing wrong with it.