Field of Science

Aliens, arsenic and alternative peer-review: Has science publishing become too conservative?

In 1959, physicists Philip Morrison and Giuseppe Cocconi advanced a hypothesis about how we could detect signals from extraterrestrial civilizations. The two suggested monitoring microwave signals from outer space at the frequency of 1420 MHz. This frequency is the frequency of neutral hydrogen, the most abundant element in the universe and one which aliens would likely harness for communication. The paper marked the beginning of serious interest in searching for extraterrestrial life. A year later, Freeman Dyson followed up on this suggestion with an even more fanciful idea. He conjectured that a sufficiently advanced civilization might be able to actually disassemble a planet the size of Jupiter and use its parts to create a shell of material that would surround the parent planet’s solar system. This sphere would capture solar energy and allow civilizations to make the most efficient use of all such energy. The most telling signature of such an advanced habitat would be an intense infrared signal coming from the sphere. Thus Dyson recommended looking for infrared signals in addition to radio signals if we were to search for aliens. The sphere came to be known as a ‘Dyson sphere’ and became fodder for a generation of science fiction enthusiasts and Star Trek fans.

These two ideas and especially the second one sound outrageous and highly speculative to say the least. Can you guess where both were published? In the two most prestigious science journals in the world; the Morrison paper was published in Nature while Dyson published his report in Science. This was in 1960. I can say in a heartbeat that I don’t see similar ideas being published in these journals today, and this is a situation which we all should regret.

I bring up this issue because I think it indicates the significant changes in attitude about publishing novel scientific ideas that have occurred from 1960 to the present. In 1960 even serious journals like Nature and Science were open to publishing fanciful speculation, provided it was clearly enumerated. Now the demands for publishing have become more stringent, but also more narrowly defined. While this may have led to the publishing of more ‘concrete’ science, it has also dissuaded researchers from venturing out into novel territory. Most importantly, it has led the scientific community to put an unnecessarily high premium on ideas being right rather than interesting.

Science progresses not by being right or wrong but by being interesting. Most scientific ideas in their infancy are tentative, unsubstantiated and incomplete. Yet modern scientific publishing and peer review largely discourage the presentation of these ideas by insisting on convincing evidence that they are right. In most cases this emphasis on accuracy and complete validation is necessary to save science from itself; we have seen all too many cases of pseudoscience that looked superficially plausible but which turned out to be full of holes. Science usually plays it safe by insisting on unimpeachable evidence. But in my opinion this stringent self-correcting process has gone too far, and in our desire to err on the safer side we have erred on the extreme side. This is having a negative impact on what we can call creative science. The insistence on foolproof data and the public censure that researchers would face if they don’t provide it is deterring many scientists from publishing provocative results that are still in the early stages of gestation. Demands for conservative presentation are also accompanied by conservative peer review since reviewers fear backlash as much as authors. All this is unfortunate and is to the detriment of the very core of scientific progress, since it’s only when provocative ideas are published can other researchers validate, verify and refute them.

The furor about the recent paper on “arsenic-based” life brings these issues into sharp focus. Much of the hailstorm of criticism would have been avoided if the standards and formats of scientific publishing allowed the presentation of ideas that may not be fully substantiated but which are nonetheless interesting. By now we are all familiar with the torrent of criticism about the paper that has come from all quarters, from blog posts to opinions from well-known experts. What is clear is that the experiments done were shoddy and controls were lacking. But the criticism is detracting from the potential value of the paper. Irrespective of whether the claims of arsenic actually being incorporated in the bacterium’s replicative and metabolic machinery are true, the paper is undoubtedly interesting, if only as an example of a hitherto unknown novel extremophile. Yet it is in danger of simply being forgotten as one of the uglier episodes in the history of science publishing.

There is in fact a solution to this problem, one which I have been in favor of for a long time. What if there was a separate section specifically devoted to relatively far-fetched ideas and this paper had been published in that section? The paper would then likely have been taken much less seriously and its tenets would have been accepted simply as thought-provoking observations pointing to further experimentation rather than established facts. So here’s my suggestion; let the top scientific journals have a separate section entitled ‘Speculation’ (or perhaps ‘Imaginings’) which allows the presentation of ideas that are fanciful and speculative. The ideas proposed could range from purely theoretical constructs to the documentation and interpretation of unusual experimental observations. The only requirement is that they should be unorthodox and interesting, backed up by more or less known scientific principles, clearly defined and enumerated and contain testable hypotheses. Let there be a second type of peer-review process for these ideas, one which is as honest as the primary process but more forgiving of the lack of foolproof evidence.

The idea about Dyson spheres would fit in nicely in such a section. Another example that comes to my mind is an idea proposed by the biophysicist Luca Turin. Turin conjectured that we may smell molecules based not on their shape but on the vibrations of their bonds. The history of this idea is interesting since others had already proposed it earlier in respectable journals. Turin actually wrote it up and sent it to Nature. Nature deliberated for an entire year and rejected the paper. In this case Nature should at least be commended for taking so long and presumably giving careful consideration to the idea, but the point is that they wouldn’t have had a problem publishing it in a ‘Speculation’ section right away. Turin’s idea was interesting, novel, highly interdisciplinary, enumerated in great detail and backed up by well-known principles of chemistry and spectroscopy. It satisfied all the criteria of a novel scientific idea that may or may not be right. Turin finally published in a journal which only specialists read, thus precluding the concept from being appreciated by an interdisciplinary cross-section of scientists. There is now at least some evidence that his ideas may be right.

Interestingly, there is at least one entire journal devoted to the publication of interesting hypotheses. This is the journal ‘Medical Hypotheses’. Medical Hypotheses prominently lacks peer review (although they have instituted some peer review recently) and has occasionally come under fire for publishing highly questionable papers, such as those criticizing the link between HIV and AIDS. But it has also served as a playground for the interaction of many interesting ideas. The editorial board of Medical Hypotheses features highly respected scientists like the neurologist V S Ramachandran and the Nobel Prize winning neuroscientist Arvid Carlsson. Ramachandran himself has iterated the need for such a journal. Science and Nature merely have to devote a small section in each issue to the kinds of ideas that are published in Medical Hypotheses, perhaps with a higher standard.

It’s worth reiterating Thomas Kuhn’s notions of paradigm shifts in science here. Scientific paradigms rarely change by playing it safe. Most scientific revolutions have been initiated by bold and heretical ideas from maverick individuals, whether it was Darwin’s ideas about natural selection, Einstein’s thoughts about the constancy of the speed of light, Wegener’s ideas about continental shift or Bohr’s construction of the quantum atom. Not a single one of these ideas was validated by foolproof evidence when it was proposed. Many of them sounded outright bizarre and counter-intuitive. But it was still paramount to bring these ideas to a greater audience. Only time would tell whether they were right or wrong, but they were undoubtedly supremely novel and interesting. And almost all of them were published by leading journals. It was the willingness to entertain interesting ideas that made possible the scientific revolutions of the twentieth century. It seems to be a strange historical anomaly to find journals much more prone to publishing speculative ideas a hundred years ago than today. Today we seem to worship the safety of truth at the expense of the uncertain but bold reaches of novelty.

Of course, the existence of a second-tier of publication and peer review would undoubtedly have to be carefully monitored. There is after all a thin line between reasonable speculation and pseudoscience. The reviewers in this tier would have to pay even more careful attention than they usually do to ensure that they are not pushing baseless fantasies. But as we have seen in the case of the vibrational theory of smell and the case of arsenic-loving bacteria, it’s not that hard to separate legitimate science with uncertain truth value from mere storytelling.

Once the ground rules are established and the initial obstacles are overcome, the second tier of peer review would have many advantages apart from encouraging the publication of speculation. It would also make reviewers more comfortable in recommending publication; since the ideas are speculative anyway, they would not insist on complete verification and would not fear backlash if the ideas they had reviewed turn out to be wrong. Journal editors would similarly find it easier to approve publication. And the scientific community at large perhaps would not be as critical as it has been in the case of the recent paper because it too would accept the proposed ideas not as declarations of truth but as tentative exploration. But the greatest beneficiaries of the improved system would undoubtedly be the publishing scientists. Their minds would be much freer to dream and they would fear much less retaliation from the community for daring to do this. Most importantly, unlike the recent case, they would not be under pressure to make statements whose implications exceed the objective factual implications of their claims, and they would be happy to just present the claims as interesting observations that point the way towards further experiments.

Science progresses by being the ultimate free-market of ideas; this has led to it being a highly social process where scientists build on each other’s work. But for this social process to work the ideas must be liberated from their initial nebulous beginnings. Ideas in the scientific marketplace come in different flavors, from boring and established to interesting and maverick. The current scientific publication and peer-review process imposes a straitjacket that ideas have to fit in in order to be ‘pre-selected’ for entry into this market. This keeps out some of the most interesting ideas and more importantly, dissuades thinkers from even pursuing them in the first place. The straitjacket does serve the valuable purpose of filtering flotsam but it is also filtering out too many other interesting things. Science is too haphazard and full of unexpected twists and turns to be entrusted to rigid rules of review and publication. We need to accept the liability of occasionally having a dubious idea published in order to keep open the possibility of also giving novel beginnings a public platform; the beauty of science is that the bonafide dubious ideas automatically get weeded out through scrutiny and so we should not have to worry about too many of them going on extended rampages. But the potentially good ideas can only be fleshed out by other scientists when they are allowed to be exposed to criticism, appreciation and ridicule. Even if the ideas themselves ultimately sink, they may serve as spores which lead to the germination of other ideas. And it is the germination of these other ideas that gets transformed into trees of scientific discovery.

We are all sheltered, invigorated and inspired by the branches of these trees. Let’s give them an opportunity to grow.

15 comments:

  1. I don't know that it's the speculation that bothers most people (or at least me), but the fact that it's speculation which is said to be truth. Most of the poorly received papers were not very speculative in nature - they were simply poorly conceived and couldn't back up their claims (in La Clair's case of hexacyclinol, couldn't back up anything). Informed speculation is useful as such, just as nongroundbreaking papers are useful as such. If people can't tell which is which, though, then 1) the journal system becomes useless, and 2) it's hard to trust that they can make the distinction between informed speculation and the output of uncontrolled drug use.

    I don't think the complaints over the arsenic-based life paper are the reason we don't have more speculative idea papers, and I'm not really sure why not - if it were clearly delineated it would probably be interesting. I just don't like being sold filet mignon and finding a McDonald's hamburger instead.

    ReplyDelete
  2. While I agree with you that there is certainly a need for speculative papers, I would argue that there is a big difference between a paper stating "It may be possible for some organism, as yet unspecified, to incorporate arsenic into its DNA" and "This particular organism incorporates arsenic into its DNA."

    ReplyDelete
  3. What? Nature and Science conservative? Since when. You know there's journals for real science, like Biochemistry, or Cell, or Organic Syntheses. And, for the batshit insane stuff you know, there's always Arxiv.

    As for Luca Turin. Hey, he got a pretty sweet TED talk out of his amazing hypothesis (which for the record I believe accounts for some of our sense of smell)...

    And einstein wasn't the first person to notice the constancy of the speed of light. Maxwell was (it drops out pretty readily from his equations). And for almost a decade before einstein's theory of special relativity, the Lorentz Transformations were known... Hell, Michelson and Morely fricking did the experiment almost TWO decades before einstein. Just saying I think the notion of maverickness is very much overrated, and a result of ignorance of scientific history.

    ReplyDelete
  4. 'journal of theoretical biology' and 'origins and evolution of the biosphere' both deal explicitly with the subject and are full of entirely speculative (and mixed, here's a little evidence-but we didn't quite work it out) work.
    If science and nature are selective, considering they publish work in EVERY field they better not just publish everything that comes into them. Note that Turin's paper was considered for a year, obviously they considered it.

    ReplyDelete
  5. Thanks for your comments. The authors are definitely at fault here for pushing a speculative hypothesis (the incorporation of As) as the most likely explanation. What I am suggesting is that they may not have been tempted to do this if the paper had been published in an exclusive speculations section. In a speculation section all you do is present the observations and then present your interpretation simply as a hypothesis. But the current standards of publication more or less prevent you from speculating. The La Clair paper was of course different; that was just completely bad science which was devoid of real novelty. Interestingly Hap has a good point about the journal system becoming useless because of the inability to detect flotsam from substance; a speculation section will make the distinction clear at least in one specific context.

    As for Einstein, to my knowledge Lorentz, Michelson, Poincare and all others somehow tried to implicitly account for the existence of the aether or at least thought their results to be in contradiction with its presence. The speed of light did indeed drop out from Maxwell's equations but the speed was still thought to be relative to the aether. Einstein was the first one to directly deduce the constancy by considering the previous results as sufficient without any ad hoc assumptions and an explicit rejection of the aether.

    It's interesting that Nature rejected Fermi's paper on beta decay with the opinion that it contained results "too remote from reality". Apparently they later admitted this as one of their biggest mistakes. The Journal of Theoretical Biology might indeed be a good forum for results like the As bacteria paper but it would prevent them from reaching a greater audience. The advantage of including such a section in Nature and Science would be that speculation would again become respectable.

    ReplyDelete
  6. "What if there was a separate section specifically devoted to relatively far-fetched ideas(?)"

    If the goal is to get speculative ideas out there, such a section serves no purpose. There are a huge number of options for people who want to promote speculative ideas. We don't live in an age where only a small number of people have access to printing presses and distribution channels any more.

    The issue isn't about whether people can promote unusual ideas. They can, and do.

    The issue is about getting "paid" for coming up with wild notions in the currency of academic science: peer-reviewed publications, preferably in journals with a high impact factor.

    A "speculations" section would be highly attractive to authors, because they could claim "I got a paper in a Glamour Mag!" But there would be nothing in it for the Glamour Mags to start such a section. They are still operating on the model of scarcity, and their identity is built around taking only the best and most influential.

    ReplyDelete
  7. This was a fantastic post.

    I wonder whether the world of research will ever break the cultural view that Science and Nature tower over everything else. You'd hope that the person who originated an idea would get credited for it regardless of whether it is peer-reviewed for Nature or published on his blog.

    Lit and Google searches should help this effort. But the person above raised another point, the academic world still demands that you publish in high impact journals to stay employed. To get any sort of worthwhile credit for self-publishing, you pretty much have to blow everyone else out of the water (e.g., Grigori Perelman).

    ReplyDelete
  8. Given the response from the authors of the As paper, I don't think they would have submitted the paper to a hypothetical "speculative" section of Science, nor do I think they would have been okay with publishing it there if upon review they were told that was the only option. It's pure speculation on my part of course, but I suspect they would have then submitted it to Nature (and then to others), since they are seemingly convinced that their hypothesis is correct.

    ReplyDelete
  9. Ed Yong sent me over here. I agree, this post is great. I especially like: “Science progresses not by being right or wrong but by being interesting. Most scientific ideas in their infancy are tentative, unsubstantiated and incomplete.”
    I believe that public support for science can be improved if they can understand that scientific investigation is a process. If they can see, observe and even participate science in their daily lives, I believe that they will be more supportive of scientists working at a higher level of expertise. Too often, science is presented as a dry set of facts in some text. Which they then may compare with some other book by an Authority and decide to believe or not believe. This is entirely different than understanding that science is about evaluating observable evidence.
    For scientists, I don’t think that this needs to always be driven by speculation, or “maverickyness”. In my version of history, most of the articles published in journals of what chemDroid calls “real science”, were read by only a limited audience and the ideas therein thus had a highly restricted reach. The thought I had that caused Ed to send me here was: For scientists, it needs to be possible to publish preliminary results, and, even if not fun, risk being wrong. Historically, papers were frequently reviewed by and then read by a fairly incestuous group of interlinked scientists. Turning this into a world wide free for all is a fruitful way to get to the bottom of things. But if it causes scientists to be too hesitant to come forward with ideas, and wait for perfection, then science loses.

    ReplyDelete
  10. The simple explanation for why modern journals are conservative is that we know more now, a lot more. When you don't know anything, many ideas are equally possible, but when you have data it constrains the possibilities. What seems fanciful now is only so because we've acquired enough data to know that it is. There's so much data out there now that models are significantly constrained so the challenge is consistency.

    ReplyDelete
  11. There are potential rewards for this kind of work — work that's still not easily labeled either valuable or worthless. These rewards are the Ig Nobel Prizes. The criterion is simply "it makes people laugh, then makes them think".

    Oliver Selfridge long ago suggested a journal for ideas of the sort you're talking about — "half baked ideas", ideas for which no one had yet bothered to assemble much evidence at all. We sometimes publish such things in the Annals of Improbable Research, though we tend to like having at least some twisted shreds evidence....

    Marc Abrahams
    Editor, Annals of Improbable Research

    ReplyDelete
  12. @ Gaythia: Uhm, leave speculative stuff to Popular Science or Scientific American, or hell, TED talks. Do you honestly think that they only don't think the science writers on the staff there bother to read Biochemistry or Cell?

    There are better ways to break the culture of incestuousness in science than opening the floodgates for poor science to be done. The "increasing conservatism" of science journals is exactly the correct response to the deluge of scientific fraud/misconduct that we're mired in. Hellinga, Sames, Cordova, Schultz via Zhang, not to mention pecaddilos that have self-amplified, such as Geoffrey Chang. And those are the ones we know about!

    No, the problem is not that journals are too conservative, the problem is that we are on a science bubble -- there are too many scientists and the market signals have confused unqualified people into joining our ranks and there are not enough people *truly qualified* to push forward the knowledge in anything but a superficial, shoddy, or fraudulent way.

    Don't believe me? What's going on in science has all the hallmarks of a bubble. PhDs applying for jobs that were set aside as BA/BS/MS positions. Fraud becoming an acceptable way to jockey for scarce academic positions. A veritable red tide of shoddy journals and shoddy science to fill the shoddy journals.

    Diminishing science productivity:
    http://www.scientificamerican.com/article.cfm?id=science-productivity

    The Decline Effect:
    http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer

    ReplyDelete
  13. You should check out some old issues of Nature - David Jones used to publish a column based on speculative research in an amusing column called Deadalus, where his fictional DREADCO company proposed many hypothetical ideas, like space elevators. Nothing wrong with that. Nothing to say Nature or Science couldn't continue to do so.
    There's a huge difference between that and publishing shoddy or sloppy research where you weren't your own worst critic, didn't make enough of an effort to falsify hypotheses and, in short, leave it to others to clean up the mess you made.
    Great post by the way.

    ReplyDelete
  14. This sounds like some of the discussion we've been having about brainstorming. In brainstorming, you want to develop as many ideas as possible, even the ones that are way out there. The next step is to ruthlessly cut the ideas down to the few that you have time and ability to do.

    ReplyDelete
  15. I would love to see a post where you discuss turin's ideas. As far as I can tell, there has only been speculative conjecture that his ideas might not be as wrong as people think...but there has been no experimental validation at all of his theories.

    ReplyDelete

Markup Key:
- <b>bold</b> = bold
- <i>italic</i> = italic
- <a href="http://www.fieldofscience.com/">FoS</a> = FoS