Fraud, not error, accounts for most scientific retractions

There's a new study in Proceedings of the National Academy of Sciences that should make the scientific community sit up and do a little pondering. Researchers from the University of Washington, the Albert Einstein College of Medicine and the firm MediCC! analyze retracted papers from 1977 onwards and investigate the reasons for their retractions. The authors focus on papers in the life sciences. And they find that about 67% of the 2047 retracted papers owed their retraction to plain old misconduct; only 21% or so can be traced back to error and honest mistakes. The misconduct can come in three forms - outright fraud, plagiarism and duplication. The other piece of bad news is that among these three, fraud contributes the most to the retraction with plagiarism and duplication tagging behind.

Number of retractions since 1977 and cause of retraction (Image source: Fang et al. PNAS 2012)

The study is disturbing, not just for the amount of misconduct it unearths but for the various trends it observes and for what it has trouble finding. The authors look at parameters such as the time to retraction, the impact factor of the journal in which the retracted papers were published and the geographic distribution of the papers. In each one of these categories it discovers something thought-provoking.

The first thing the study discovers is that fraud is not as easy to find as we think. There are several instances of papers where the retracted papers are still available and cited. Not only does this prevent the retraction from misleading other researchers  but it also depressingly means that, as the authors themselves confess, the study likely underestimates the number of retractions. Why this is so makes partial sense in at least a few of these cases where scientists seem to find something of value in the paper in spite of the retraction. Also opaque are the cases where the retraction is ambiguously communicated, giving the impression of error rather than misconduct. The problem is that the retraction notice is often published by the original authors and they are understandably reluctant to provide any hints of fraud. Thus, the notice for one particular study says that "results were derived from experiments that were found to have flaws in methodological execution and data analysis". What this note does not say - and what's confirmed later by an independent Office of Research Integrity inquiry - is that the "flaws" were actually a result of manipulation and willful fabrication. Retraction notices may thus clearly hide evidence of wrongdoing, and it seems that journals are not always correcting or modifying original notices wherever necessary.

An analysis of the geographic origins of retraction is also quite interesting since this seems to be split by the nature of the misconduct. The US, Germany, China and Japan accounted for 75% of cases of fraud or suspected fraud. Conversely, China and India accounted for a large percentage of plagiarism and duplication. Why this may be so takes us to the next step of the analysis; analyzing retraction by impact factor. The authors found that fraud contributed the most to retractions in high-impact journals like Science and Nature while plagiarism and duplication were responsible for retractions in lower-impact journals. This may be consistent with the geographic trends since the US, Germany and Japan publish in higher-impact journals compared to India or China.

Cause of retraction by geographic location (Image source: Fang et al. PNAS 2012)

Sadly and perhaps not surprisingly, this is also a case where a few bad apples give the whole field a bad name. Scientific controversies over the last few years are littered with the corpses of multiple papers published by the same author or the same lab. As the authors find out, a small number of authors were responsible for multiple retractions and nearly all the multiple retractions from the same author could be traced to outright fraud rather than plagiarism or duplication. A table in the paper has a rogues' gallery of infamous retractors, ranging from the infamous vaccination denialist Andrew Wakefield to Anil Potti, the Duke researcher who falsified data in multiple papers and then hired an "image consultant" to craft personal airbrushed websites with misleading information. Papers from these people have been cited a depressingly large number of times, and one factor which the study cannot gauge is the immense negative impact they must have had on other researchers' honest work before being retracted.

The authors also look at time-to-retraction and citation frequency of retracted articles and find both hope and disappointment. Over the years there's a general, gradual trend toward longer time-to-retraction that's independent of impact factor. As for citation frequency, while many retracted articles stop being cited right away, there are also several that continue to be cited for various reasons, ranging from a lack of transparency in the retraction notice to researchers still finding something of value in the papers.

What do we make of all this? The responsibility of journals and funding agencies to scrutinize papers and grants and detect fraud is obvious but the issue runs deeper. As with any large-scale endemic scientific problem, this one says more about scientific culture than about the technical details in the paper, although the technical details are important. It's worth noting that all these papers are from biological or medical fields. These fields routinely try to analyze and present complex, multifactorial data sporting error bars as tall as skyscrapers. Even honest, non-retracted work is having a hard time being duplicated in these fields. One only needs to look at the recent controversy about SIRT inhibitors, caloric restriction and lifespan extension to appreciate how messy biological research can get. In a field as messy as biology, it's easy to make stuff up and move a few gel lanes in Western blots here and there in what is typically a mountain of data involving multiple techniques and experiments. Biomedical research is likely going to continue to invite fraud because of its sheer and growing complexity. In addition the monetary benefits, fame and visibility associated with potentially important results in fields like caloric restriction are immense, leading to even more opportunities for fraud and wishful thinking.

However, the emphasis on biomedical research should not blind us to the greater problem the journal points to; the pervasive culture of publish-or-perish and grantsmanship that often views research as a zero-sum game. Science has very much turned into a high-stakes competitive business. More than ever, researchers are competing against each other for funding, resources, grants and publicity. Competition has now extended into the international arena with researchers in certain countries thinking it a matter of patriotic pride to score points against their transcontinental counterparts. There are cases where vast sums of money are allocated by developing countries to their prized scientific fighters to catapult themselves into the big league of top scientific nations. It should go without saying that these kinds of pressures are going to do nothing to dissuade researchers from tweaking a few experimental parameters here, adding a data point to the graph there; all for the sake of getting their papers into high-impact journals, ensuring tenure, awards and fame.

But sadly this culture is here to stay, at least for the foreseeable future. What can we do to curb its worst excesses? The paper points to encouraging solutions like the Office of Research Integrity and regular courses on ethics and data presentation. There are few counterparts to these measures in developing countries and this needs to be taken seriously by their governments. More importantly, researchers, journal editors and referees need to be taught both the value and the methodology of honest research from day one of their careers. They also need to be taught means of detecting fraud and plagiarism, a task which has been rendered somewhat easier by technology and software for detecting duplication and possible manipulation. The authors also suggest the creation of a centralized database of scientific misconduct. It's worth noting that in its own way Retraction Watch has admirably served this purpose. Finally, the study points to a great responsibility on the part of journals and unfortunately one which not all of them are fulfilling. As indicated above, many journals still communicate ambiguous retraction notices that give no hint as to the cause of retraction. Some journals still have retracted papers on their websites. Retractions will continue to be underestimated if journals don't tell us about them. One very valuable service that journals can provide in this regard is to make as many papers open-access as possible, or at least allow comments on most papers that might potentially point to suspicious data.

Ultimately though, the study asks what all of us - scientists, editors, educators, readers and the lay public- can do to stem this growing tide of scientific fraud. Retraction Watch is just one example of what we can collectively do to keep everyone honest. In the past few years, online sources like blogs have made it clear that they can be in the front ranks of detecting and reporting fraud. There have been at least some retractions in the past few years where the retracted paper was first discussed on a blog. It's quite clear that bloggers and journalists can often sniff out suspicious papers even before official sources. The official sources need to pay attention to these first responders and act on their analysis and recommendations. The quick and extensive dissemination of unworthy science will also hopefully provide a disincentive to scientists who are tempted by fraud and manipulation. Let's hope that the existence of a completely non-anonymous jury of thousands of eagle-eyed peers will nip any invitation to fraud in the bud. And in one sense the presence of scientific frauds keeps us all awake; paraphrasing Jefferson, perhaps it's best that "the tree of scientific honesty be refreshed from time to time by the words of wrongdoers and vigilantes of peer review".

Originally posted on the Scientific American Blog Network

The anatomy of peer review: Why airing dirty laundry in public is important

Since everyone is talking about Ron Breslow, I thought I might bring readers' attention to a truly fascinating (from the point of view of the sociology of science) article published in Nature in 1992 regarding two of Breslow's papers. The article was written by Prof. Fred Menger (Emory) and Prof. Albert Haim (Stony Brook) and details what happened when the two tried to publish a rebuttal addressing errors in two Breslow articles from the Journal of the American Chemical Society (JACS).

First I want to emphasize the reason why I am writing about this episode. I write about it not because I want to add to Breslow's troubles. I don't want it to sound like I am kicking someone when they are down, nor do I think that I will ever have the capability to do this to someone like Breslow. As I have reiterated in other places, neither this account nor the recent controversy should blind us to an unambiguous fact; Breslow's contributions to research, service and education have been outstanding by any standards. Most scientists would be lucky if they could accomplish half of what he has done during an unusually long and productive career that has still not slowed down. We can all hope that people will continue to remember him by his superlative accomplishments.

So I am not writing this post because I want to add to further criticism of Breslow. It's because I have always thought that the Nature article is a unique document in the history and sociology of chemistry, a great illustration of the pitfalls and promises of peer review that deserves a wider audience. I think laymen (who don't have journal access) will find its contents very interesting, and there's lessons in there for fellow scientists too. I do want to add a disclaimer: Prof. Menger was on my Ph.D. committee and I have enormous respect for his research, but as someone interested in the way science functions, this paper would have been equally fascinating to me even if I had absolutely no connection to him. There were several other papers related to this incident, but the one in Nature stands out as a unique example of scrutiny.

The article is one of the most remarkable publications I have ever read, for two reasons. Firstly, because it presents a rare glimpse into what we might call the "anatomy" of peer review; and it does this in excruciating detail, with warts and all exposed. Secondly, because it makes you wonder whether a journal like Nature would ever again have the inclination to publish something like this.

The whole episode germinated when Breslow and a pair of graduate students (Eric Anslyn and Deeng-Lih Huang) published two papers in JACS dealing with the kinetics of an imidazole-catalysed nucleotide cleavage reaction. This is standard stuff in physical organic and bioorganic chemistry and is part of a large body of work going back 50 years. What was interesting was that the authors seemed to derive negative rate constants for some of the reactions. Now, even college students will recognize this as odd; a rate constant is supposed to indicate the speed of a reaction. If it's positive, the reaction proceeds. If it's zero, the reaction halts. But what is a negative rate constant supposed to mean? Of course the original authors had their own interpretation of their numbers. Independently, Menger and Haim investigated this question and they found out a few rather significant problems with the two papers. The technical details can only be appreciated by a physical organic chemist, but the main problem seemed to be in treating the background reactions with water in the absence of imidazole. It seemed that at least some of the "negative" rate constants were artifacts of the data fitting.

The publication saga started when Menger and Haim independently submitted papers to JACS criticizing the original paper and offering corrections. The editor of JACS at the time was Alan Bard, an internationally renowned chemist who served a distinguished stint as journal editor for several years. Menger's paper was rejected by the reviewers with some odd and rather self-contradictory commentary. On one hand, the reviewers acknowledged the errors in the Breslow papers, but on the other hand they inexplicably chose to reject the manuscript, "hoping" that Breslow and Huang would publish a more detailed explanation and correction. It wasn't clear why they would not let Menger himself publish a correction in the journal.

Haim's paper was also rejected by the journal with similar comments. At this point both Menger and Haim wrote to Bard and the associate editor in charge of the manuscripts. They appealed to JACS's editorial policy which encouraged the submission of manuscripts detailing major errors in published material. This appeal did not have much effect. Something strange also transpired at this juncture; as Menger details, Breslow sent a rather interesting note to Haim, saying: "I don't know what you are so excited about. Are you being led astray by a notoriously unstable individual?". 

Following their unsuccessful attempt at publication in JACS, both Menger and Haim did what scientists always do; try to publish in other places. Haim first send his manuscript to the Journal of Physical Chemistry (JPC). JPC's response was about as strange as JACS's; while acknowledging the problems with the original papers, they too chose not to publish Haim's manuscript. Menger in turn sent his paper to the Journal of Organic Chemistry (JOC). He seems to have found a much more sympathetic audience in the journal's chief and associate editors, Clayton Heathcock and Andrew Streitwieser, both leading researchers. The paper was accepted without much ado by Streitwieser who wondered how the original paper had made it past the JACS referees.

The story does not end there. Menger's JOC paper stimulated a number of largely favorable responses. An especially noteworthy response was from a Nobel Laureate, who sent a note to Bard saying, 

"It seems to me there is a long-established scientific etiquette which says that papers pointing out errors should be published in the same journal in which the original article appeared". 

But these encouraging exchanges were followed by a few calls for retraction of the JOC paper, one of these being from Breslow. At least some of these raised legitimate scientific questions. This prompted Heathcock to ask for clarification, which Menger duly provided. Streitwieser accepted Menger's responses and the paper was published. To the journal's credit, they even permitted a footnote in which Menger said that his attempts to publish his paper in JACS had been rejected.

Meanwhile, Haim was still trying to publish his correction in JACS. He sent a revised, more thorough manuscript to Bard, asking that the original associate editor who had rejected the manuscript be replaced by a fresh pair of eyes. Bard did not agree to this request, but asked Haim to recommend ten referees for the paper. The article was sent to four of these reviewers. Two of the reviewers again agreed that the original science contained errors but again asked that Breslow and his collaborators be given a chance to publish corrections. One reviewer's response was especially interesting; he or she thought that the matter should simply be dropped, ostensibly to protect the reputation of the author:

"It could lead to a reputation, rightly or wrongly, of the author being a nitpicker and Breslow would certainly fight back loudly. Who needs such things"

I find the part about being a nitpicker especially interesting. Firstly, the criticism was not just about a few minor details; it was about rather fundamental analyses and conclusions in the paper. But more importantly, a lot of science in fact is nitpicking because it's through nitpicking that one often uncovers the really important things. Science especially should provide a welcome refuge for nitpickers. 

In any case, after yet another rejection, Haim submitted another revised manuscript. It's worth noting that most reviewers' comments during the 11 months that Haim had tried to get his manuscript published had been favorable, and nobody had ever called Haim's basic analysis into serious question. Yet the paper kept on getting rejected for various reasons. Finally, Haim appealed to Bard and the paper suddenly and inexplicably got published.

Menger ends his account of the long saga with the following words:

"As the dust settles, it is comforting to reflect that the system ultimately worked. After all, both of us succeeded in getting our papers published. Yet this was accomplished only at the cost of considerable anguish to us. Few people, we presume, would be willing to go through this experience...Two problems are involved here. First is the mishandling of the original publications, which many people have come to regard as substandard. The second is the position taken by the associate editor after flaws were pointed out. That position can only be described as evasive and defensive. Without attributing motivation for his actions, we simply state that we believe them to have been inimical to the best interests of science"

He ends by hoping that the fear of open criticism would encourage scientists to police themselves better (At this point science bloggers should let out a collective hurrah, for reasons that will become apparent below). There is a postscript: Breslow replied to this lengthy report shortly (and gratifyingly, his response was published in the same journal) and described experiments that would clarify his earlier work, but he did not address the many questions about peer review raised in Menger's communication.

This fascinating account raises many important issues. For one thing, it was quite clear that the original paper had problems; even the reviewers consistently agreed with this part of the story. Thus the science seems pretty clear, and the ambiguity in the situation came from the human element. We will never know what went behind the scenes when the manuscripts were rejected even after the reviewers agreed with the rebuttal. Unfortunately motivations are hard to unravel, but one cannot help but suspect that there was prestige and influence at work here which thwarted efficient and open scientific revision. The fact that powerful people from the chemistry community (especially Breslow and Bard) were involved cannot be an inconsequential factor.

Secondly, it does seem important to me (although this is a relatively minor issue in my mind) for journals to publish corrections to papers in their own pages; at the very least, this underscores a culture of responsibility on the part of the journal and sends out a positive message. However this practice involves some interesting operational questions. Should the journal first allow the original authors to publish a correction? If so, how long should it wait before doing this? It seems clear to me that legitimate corrections should immediately be published, irrespective of the source.

The most remarkable fact about this account is that Nature published it, and in writing it Menger performed a unique and valuable public service. Personally I have never seen such a detailed dissection of peer review described in a major journal. Some people would deplore this public airing of dirty laundry. They would say that none of this can undo what happened, and the only effect of such articles is bad blood and destroyed reputations. I happen to disagree. I think journals should occasionally publish such analyses, because it alerts us to the very human aspect of science. It demonstrates to the public what science is truly like, how scientists can make mistakes, and how they can react when they are corrected or challenged. It sheds important light on the limitations of the peer review process, but also reaffirms faith in its ultimately self-correcting nature. Some people might think that this is a great example of how peer review should not be, but I would like to think that this is in fact exactly how the process works in the vast majority of cases; imperfect, ambiguous, influenced by human factors like reputations, biases and beliefs. If we want to understand science, we need to acknowledge its true workings instead of trying to fit it into our idealized worldview of perfect peer review.

In this day and age, blogs are performing the exact same function as Nature did in 1992, and this is clearly apparent from the latest Breslow brouhaha. Menger and Haim in 2012 would not have to test their patience by trying to publish in JACS for 11 months; instead they could upload their correction on a website and let the wonder of instant online dissemination work its magic. Blogs may not yet be as respectable as JACS, but the recent incident shows that they can be perfectly respectable outlets of criticism as long as the criticism is fair and rigorous. The growing ascendancy of blogs and their capacity to inflict instant harm on sloppy or unscrupulous science should hopefully result in much better self-policing, leading authors to be more careful about what they publish in "more respectable" venues. Thus, quite paradoxically, blogs could lead to the publication of better science in the very official sources which have largely neglected them until now. This would be a delightful irony.

Perhaps the greatest message that the public can take home from such incidents is that even great scientists can make mistakes and remain great scientists, and that science continues to progress in one way or another. No matter how bad this kind of stuff sounds, it's actually business as usual for the scientific process, and there's nothing wrong with it.

Aliens, arsenic and alternative peer-review: Has science publishing become too conservative?

In 1959, physicists Philip Morrison and Giuseppe Cocconi advanced a hypothesis about how we could detect signals from extraterrestrial civilizations. The two suggested monitoring microwave signals from outer space at the frequency of 1420 MHz. This frequency is the frequency of neutral hydrogen, the most abundant element in the universe and one which aliens would likely harness for communication. The paper marked the beginning of serious interest in searching for extraterrestrial life. A year later, Freeman Dyson followed up on this suggestion with an even more fanciful idea. He conjectured that a sufficiently advanced civilization might be able to actually disassemble a planet the size of Jupiter and use its parts to create a shell of material that would surround the parent planet’s solar system. This sphere would capture solar energy and allow civilizations to make the most efficient use of all such energy. The most telling signature of such an advanced habitat would be an intense infrared signal coming from the sphere. Thus Dyson recommended looking for infrared signals in addition to radio signals if we were to search for aliens. The sphere came to be known as a ‘Dyson sphere’ and became fodder for a generation of science fiction enthusiasts and Star Trek fans.

These two ideas and especially the second one sound outrageous and highly speculative to say the least. Can you guess where both were published? In the two most prestigious science journals in the world; the Morrison paper was published in Nature while Dyson published his report in Science. This was in 1960. I can say in a heartbeat that I don’t see similar ideas being published in these journals today, and this is a situation which we all should regret.

I bring up this issue because I think it indicates the significant changes in attitude about publishing novel scientific ideas that have occurred from 1960 to the present. In 1960 even serious journals like Nature and Science were open to publishing fanciful speculation, provided it was clearly enumerated. Now the demands for publishing have become more stringent, but also more narrowly defined. While this may have led to the publishing of more ‘concrete’ science, it has also dissuaded researchers from venturing out into novel territory. Most importantly, it has led the scientific community to put an unnecessarily high premium on ideas being right rather than interesting.

Science progresses not by being right or wrong but by being interesting. Most scientific ideas in their infancy are tentative, unsubstantiated and incomplete. Yet modern scientific publishing and peer review largely discourage the presentation of these ideas by insisting on convincing evidence that they are right. In most cases this emphasis on accuracy and complete validation is necessary to save science from itself; we have seen all too many cases of pseudoscience that looked superficially plausible but which turned out to be full of holes. Science usually plays it safe by insisting on unimpeachable evidence. But in my opinion this stringent self-correcting process has gone too far, and in our desire to err on the safer side we have erred on the extreme side. This is having a negative impact on what we can call creative science. The insistence on foolproof data and the public censure that researchers would face if they don’t provide it is deterring many scientists from publishing provocative results that are still in the early stages of gestation. Demands for conservative presentation are also accompanied by conservative peer review since reviewers fear backlash as much as authors. All this is unfortunate and is to the detriment of the very core of scientific progress, since it’s only when provocative ideas are published can other researchers validate, verify and refute them.

The furor about the recent paper on “arsenic-based” life brings these issues into sharp focus. Much of the hailstorm of criticism would have been avoided if the standards and formats of scientific publishing allowed the presentation of ideas that may not be fully substantiated but which are nonetheless interesting. By now we are all familiar with the torrent of criticism about the paper that has come from all quarters, from blog posts to opinions from well-known experts. What is clear is that the experiments done were shoddy and controls were lacking. But the criticism is detracting from the potential value of the paper. Irrespective of whether the claims of arsenic actually being incorporated in the bacterium’s replicative and metabolic machinery are true, the paper is undoubtedly interesting, if only as an example of a hitherto unknown novel extremophile. Yet it is in danger of simply being forgotten as one of the uglier episodes in the history of science publishing.

There is in fact a solution to this problem, one which I have been in favor of for a long time. What if there was a separate section specifically devoted to relatively far-fetched ideas and this paper had been published in that section? The paper would then likely have been taken much less seriously and its tenets would have been accepted simply as thought-provoking observations pointing to further experimentation rather than established facts. So here’s my suggestion; let the top scientific journals have a separate section entitled ‘Speculation’ (or perhaps ‘Imaginings’) which allows the presentation of ideas that are fanciful and speculative. The ideas proposed could range from purely theoretical constructs to the documentation and interpretation of unusual experimental observations. The only requirement is that they should be unorthodox and interesting, backed up by more or less known scientific principles, clearly defined and enumerated and contain testable hypotheses. Let there be a second type of peer-review process for these ideas, one which is as honest as the primary process but more forgiving of the lack of foolproof evidence.

The idea about Dyson spheres would fit in nicely in such a section. Another example that comes to my mind is an idea proposed by the biophysicist Luca Turin. Turin conjectured that we may smell molecules based not on their shape but on the vibrations of their bonds. The history of this idea is interesting since others had already proposed it earlier in respectable journals. Turin actually wrote it up and sent it to Nature. Nature deliberated for an entire year and rejected the paper. In this case Nature should at least be commended for taking so long and presumably giving careful consideration to the idea, but the point is that they wouldn’t have had a problem publishing it in a ‘Speculation’ section right away. Turin’s idea was interesting, novel, highly interdisciplinary, enumerated in great detail and backed up by well-known principles of chemistry and spectroscopy. It satisfied all the criteria of a novel scientific idea that may or may not be right. Turin finally published in a journal which only specialists read, thus precluding the concept from being appreciated by an interdisciplinary cross-section of scientists. There is now at least some evidence that his ideas may be right.

Interestingly, there is at least one entire journal devoted to the publication of interesting hypotheses. This is the journal ‘Medical Hypotheses’. Medical Hypotheses prominently lacks peer review (although they have instituted some peer review recently) and has occasionally come under fire for publishing highly questionable papers, such as those criticizing the link between HIV and AIDS. But it has also served as a playground for the interaction of many interesting ideas. The editorial board of Medical Hypotheses features highly respected scientists like the neurologist V S Ramachandran and the Nobel Prize winning neuroscientist Arvid Carlsson. Ramachandran himself has iterated the need for such a journal. Science and Nature merely have to devote a small section in each issue to the kinds of ideas that are published in Medical Hypotheses, perhaps with a higher standard.

It’s worth reiterating Thomas Kuhn’s notions of paradigm shifts in science here. Scientific paradigms rarely change by playing it safe. Most scientific revolutions have been initiated by bold and heretical ideas from maverick individuals, whether it was Darwin’s ideas about natural selection, Einstein’s thoughts about the constancy of the speed of light, Wegener’s ideas about continental shift or Bohr’s construction of the quantum atom. Not a single one of these ideas was validated by foolproof evidence when it was proposed. Many of them sounded outright bizarre and counter-intuitive. But it was still paramount to bring these ideas to a greater audience. Only time would tell whether they were right or wrong, but they were undoubtedly supremely novel and interesting. And almost all of them were published by leading journals. It was the willingness to entertain interesting ideas that made possible the scientific revolutions of the twentieth century. It seems to be a strange historical anomaly to find journals much more prone to publishing speculative ideas a hundred years ago than today. Today we seem to worship the safety of truth at the expense of the uncertain but bold reaches of novelty.

Of course, the existence of a second-tier of publication and peer review would undoubtedly have to be carefully monitored. There is after all a thin line between reasonable speculation and pseudoscience. The reviewers in this tier would have to pay even more careful attention than they usually do to ensure that they are not pushing baseless fantasies. But as we have seen in the case of the vibrational theory of smell and the case of arsenic-loving bacteria, it’s not that hard to separate legitimate science with uncertain truth value from mere storytelling.

Once the ground rules are established and the initial obstacles are overcome, the second tier of peer review would have many advantages apart from encouraging the publication of speculation. It would also make reviewers more comfortable in recommending publication; since the ideas are speculative anyway, they would not insist on complete verification and would not fear backlash if the ideas they had reviewed turn out to be wrong. Journal editors would similarly find it easier to approve publication. And the scientific community at large perhaps would not be as critical as it has been in the case of the recent paper because it too would accept the proposed ideas not as declarations of truth but as tentative exploration. But the greatest beneficiaries of the improved system would undoubtedly be the publishing scientists. Their minds would be much freer to dream and they would fear much less retaliation from the community for daring to do this. Most importantly, unlike the recent case, they would not be under pressure to make statements whose implications exceed the objective factual implications of their claims, and they would be happy to just present the claims as interesting observations that point the way towards further experiments.

Science progresses by being the ultimate free-market of ideas; this has led to it being a highly social process where scientists build on each other’s work. But for this social process to work the ideas must be liberated from their initial nebulous beginnings. Ideas in the scientific marketplace come in different flavors, from boring and established to interesting and maverick. The current scientific publication and peer-review process imposes a straitjacket that ideas have to fit in in order to be ‘pre-selected’ for entry into this market. This keeps out some of the most interesting ideas and more importantly, dissuades thinkers from even pursuing them in the first place. The straitjacket does serve the valuable purpose of filtering flotsam but it is also filtering out too many other interesting things. Science is too haphazard and full of unexpected twists and turns to be entrusted to rigid rules of review and publication. We need to accept the liability of occasionally having a dubious idea published in order to keep open the possibility of also giving novel beginnings a public platform; the beauty of science is that the bonafide dubious ideas automatically get weeded out through scrutiny and so we should not have to worry about too many of them going on extended rampages. But the potentially good ideas can only be fleshed out by other scientists when they are allowed to be exposed to criticism, appreciation and ridicule. Even if the ideas themselves ultimately sink, they may serve as spores which lead to the germination of other ideas. And it is the germination of these other ideas that gets transformed into trees of scientific discovery.

We are all sheltered, invigorated and inspired by the branches of these trees. Let’s give them an opportunity to grow.

"The most ludicrous system ever devised"

That's Nobel laureate Harry Kroto on the peer-review system. Since he won his Nobel for fullerenes, Kroto has become a tireless promoter of science education and communication. This week's issue of Nature has a series of brief interviews with several Nobel laureates. One of the questions asked was about the peer-review system and whether it is an optimal one. Most laureates gave an analogy and paraphrased Churchill's quip about democracy: it’s a system full of flaws, yet better than the other alternatives. But Kroto went one step further:

Many people consider the peer-review system broken. Do you share their view, and do you have a solution?

The peer-review system is the most ludicrous system ever devised. It is useless and does not make sense in dealing with science funding when history abounds with a plethora of examples that indicate that the most important breakthroughs are impossible to foresee.

The science budget should be split into three (not necessarily equal) parts and downloaded to departments. The local institutions, and not government departments, should disburse funding as they are close to the coalface and can decide what needs support and what is in the long-term interest of the department. There should be no research proposals on which to waste time.

One part should go to young people chosen by their universities as the researchers on which their institution's future will depend — they have done the work, why waste time doing it again when people have no time and are too far away from the coalface and in general do not have the relevant expertise?

The second part should go to a group whose most recent report was excellent. This is the racehorse solution — if a scientist has just done some great work, let her or him run again.

Although I would have probably eschewed such strong words and do sympathize with the other laureates' perspective, my heart is with Sir Kroto. Revolutionary science has often been rejected by the peer-review system; it's worth noting that Enrico Fermi's paper on beta decay was rejected by Nature.

I myself have believed in having a separate section in leading science journals devoted to "improbable" science, speculative and brain-tickling ideas flung out for contemplation by the rest of the community. The section should make it clear that such ideas have not been validated, but then that's true of any scientific idea when it's being conceived. I seriously believe that such sections would provide a lot of food for thought for researchers who are willing to go out on a limb. Maybe the published, incomplete ideas will meet their own ideas to be synthesized into a more coherent whole.

Now of course that does not mean that any crackpot idea deserves to be published. There certainly needs to be a minimum standard for acceptance. For this there could be a second kind of peer-review, where reviewers are more forgiving and more creative in judging the merit of the proposed concept. These reviewers could judge the idea not on the basis of its validation but on the basis of its novelty, novelty that’s nonetheless grounded in sound basic principles of science (thus homeopathy would be instantly excluded). Such a two-tier system would then provide an opportunity for the publication of both “normal” science as well as potentially revolutionary science. An example that comes to my mind is Luca Turin’s novel idea about olfactory molecules being detected by vibration rather than shape. The idea certainly seemed grounded in basic physics and chemistry. Its publication would have pushed at least a couple of researchers to validate or disprove it. As it turns out it was rejected by a leading journal after a long wait.

As Freeman Dyson says, the most important scientists are often "rebels" who speak out against the conventional wisdom. Their far-fetched sounding pronouncements of today have often been transformed into the important discoveries of tomorrow. The least that science journals can do is to give their ideas a worldwide platform.