"The most ludicrous system ever devised"

That's Nobel laureate Harry Kroto on the peer-review system. Since he won his Nobel for fullerenes, Kroto has become a tireless promoter of science education and communication. This week's issue of Nature has a series of brief interviews with several Nobel laureates. One of the questions asked was about the peer-review system and whether it is an optimal one. Most laureates gave an analogy and paraphrased Churchill's quip about democracy: it’s a system full of flaws, yet better than the other alternatives. But Kroto went one step further:
Many people consider the peer-review system broken. Do you share their view, and do you have a solution?

The peer-review system is the most ludicrous system ever devised. It is useless and does not make sense in dealing with science funding when history abounds with a plethora of examples that indicate that the most important breakthroughs are impossible to foresee.

The science budget should be split into three (not necessarily equal) parts and downloaded to departments. The local institutions, and not government departments, should disburse funding as they are close to the coalface and can decide what needs support and what is in the long-term interest of the department. There should be no research proposals on which to waste time.

One part should go to young people chosen by their universities as the researchers on which their institution's future will depend — they have done the work, why waste time doing it again when people have no time and are too far away from the coalface and in general do not have the relevant expertise?

The second part should go to a group whose most recent report was excellent. This is the racehorse solution — if a scientist has just done some great work, let her or him run again.
Although I would have probably eschewed such strong words and do sympathize with the other laureates' perspective, my heart is with Sir Kroto. Revolutionary science has often been rejected by the peer-review system; it's worth noting that Enrico Fermi's paper on beta decay was rejected by Nature.

I myself have believed in having a separate section in leading science journals devoted to "improbable" science, speculative and brain-tickling ideas flung out for contemplation by the rest of the community. The section should make it clear that such ideas have not been validated, but then that's true of any scientific idea when it's being conceived. I seriously believe that such sections would provide a lot of food for thought for researchers who are willing to go out on a limb. Maybe the published, incomplete ideas will meet their own ideas to be synthesized into a more coherent whole.

Now of course that does not mean that any crackpot idea deserves to be published. There certainly needs to be a minimum standard for acceptance. For this there could be a second kind of peer-review, where reviewers are more forgiving and more creative in judging the merit of the proposed concept. These reviewers could judge the idea not on the basis of its validation but on the basis of its novelty, novelty that’s nonetheless grounded in sound basic principles of science (thus homeopathy would be instantly excluded). Such a two-tier system would then provide an opportunity for the publication of both “normal” science as well as potentially revolutionary science. An example that comes to my mind is Luca Turin’s novel idea about olfactory molecules being detected by vibration rather than shape. The idea certainly seemed grounded in basic physics and chemistry. Its publication would have pushed at least a couple of researchers to validate or disprove it. As it turns out it was rejected by a leading journal after a long wait.

As Freeman Dyson says, the most important scientists are often "rebels" who speak out against the conventional wisdom. Their far-fetched sounding pronouncements of today have often been transformed into the important discoveries of tomorrow. The least that science journals can do is to give their ideas a worldwide platform.

9 comments:

  1. Sir Harry, not Sir Kroto. /nitpick

    ReplyDelete
  2. for the publication of both “normal” science as well as potentially revolutionary science


    In that case, there should be a provision asking all those ppl publishing "normal" research should label it as such. Because if you look at the proposals most research is not "normal" but revolutionary.

    ReplyDelete
  3. You're talking exclusively about peer-review for publication, while S'Harry is talking exclusively about peer-review for funding.

    They're two quite different beasts...

    ReplyDelete
  4. Neil: True, but I see similar problems. Revolutionary research can be rejected by both funding agencies and journal editors and reviewers.

    milieu, thanks. I forgot that unusual and endearing usage!

    ReplyDelete
  5. I had to do it because you did not. :) http://bit.ly/bFR9kx

    ReplyDelete
  6. Look a glamour science journal like Nature, Science, Cell, etc. rejecting a paper means exactly zero. You, of course, are aware that there are other journals. By using the example of an ultimately important work being rejected from a glamour mag, you play right into the aristocratic idea that the work being published in these journals is somehow superior to work published in other journals.

    While peer review may be imperfect, the fact is that scientists (at least those who don't suck at it) read articles from a plethora of journals, at least those articles related to their field. I know The Scientist recently published some crap about the failures of peer review because important studies were published in such weak journals as Molecular and Cellular Biology, Journal of Immunology.

    ReplyDelete
  7. Lorax, I think you make a very good point. Most of the really useful and interesting papers I have read from my field are indeed not from Nature, Science etc. but from lower tier journals. The problem still is that Nature and Science are interdisciplinary journals, so it's much more likely for a scientist from a different but related field who has good ideas to read my article if it's published in one of these journals.

    Akshat: good points as usual! I do agree that writing proposals is an extremely valuable activity, even if sometimes a necessary evil.

    ReplyDelete
  8. Just a quick note to let you know that your post is included in the latest Scientia Pro Publica. Thank you for sharing it.

    ReplyDelete

Markup Key:
- <b>bold</b> = bold
- <i>italic</i> = italic
- <a href="http://www.fieldofscience.com/">FoS</a> = FoS