Making speculation official: More on the conservatism of leading science journals

I want to thank everyone for the interesting comments on the last post; I thought it would be best to address them in a new one since my response got too long and spawned too many thoughts for the comments section.

I want to enumerate what I think are the benefits of having a separate 'Speculations' section in journals like Science and Nature because that point perhaps did not come across very clearly. In the context of the present controversial paper on arsenic-associated life, here's what would happen in a world which reveled in speculation. The authors submit the paper to Science. The Science reviewers and editors say that the paper is interesting but that the extraordinary claims are not supported by extraordinary evidence. Nonetheless, they would be quite happy to publish it in their brand new 'Imaginings' section as food for thought for other researchers. They ask the authors to tone down their conclusions and present the paper as a set of observations with some possible interpretations; either in a preliminary findings section or a speculation section. Now someone in the comments section suggested that the authors would have rejected Science's offer in such a case. But let's give them the benefit of doubt. While by no means ecstatic, the authors grudgingly accept Science's offer. The paper now looks much more tentative and its conclusions are much more modest. It proudly features as one of the first inaugural articles in 'Imaginings'. But here's the other good thing that happens: NASA and the authors now resist the temptation to present and sensationalize the work in a press conference before publication because of course it's a little embarrassing to hype a paper explicitly marked as speculative. Everyone is happy; the reviewers, the authors, Science and the public. The media will of course still hype the paper but that's pretty much a constant anyway. Generally speaking, the evil stepmother disintegrates in a blinding flash of light, the princess marries the prince in a glade surrounded by furry creatures and everybody lives happily ever after. The End, for now at least.

I agree that this is an ideal scenario. But it has a much higher probability of being played out if Science sported an explicit section on speculation. When work being presented is speculative, both the public and the reviewers are more forgiving of its incompleteness and the authors and sponsoring agencies don't (or at least should not) feel as tempted to hype it. It appears in print exactly the way it should; as a very intriguing set of observations and experiments that deserves closer scrutiny and nothing more. Any possible earth-shattering implications can wait.

There was a thought that journals should actually become more conservative because of the increasing instances of fraud that have been reported during the last few years. The general direction of this kind of thinking is sound, but I don't think it will help scientific progress at all. Fraud in scientific publishing will continue at a minimum ambient level irrespective of whether journals are conservative or not; it's just human nature. The only way journals could significantly crack down on fraud is if they become ultra-conservative. But this would be a disaster since along with fraud it would lead to the filtering out of too many promising novel ideas. The occasional admission of fraud is a burden we have to bear for publishing the boldest flights of imagination. The best thing however is that we don't have to worry too much about the problem at all; as I mentioned earlier, the beauty of science is that it is usually an incredibly efficient self-correcting process. Unlike the rogue agent from The Matrix, fraud does not stick around for too long to cause havoc. If anything, the universal presence of blogs and online information sharing now ensures that fraud is much more swiftly recognized and dealt with than before; many recent cases can attest to this fact. If the world earlier depended on one Neo to save itself, we now have several who are up to the task.

This brings us to another point in the comments section. Some people pointed out that the proliferation of blogs and other online avenues have now actually provided more opportunities than ever to speculate, and we need not depend on elite journals for doing this. While this is undoubtedly true, I wish it solved the problem. I wish that speculation on blogs was as respected as speculation in Nature. But we don't live in that ideal world yet. For whatever reason, journals like Science and Nature are now worshipped even more than what they were before. In my own field of organic chemistry for instance, you would find many pathbreaking papers published in relatively low-impact journals in the sixties and seventies, but hardly any more. Sadly, the obsession of impact factors and the constant pressure to publish and perish have put the premier journals on a pedestal. There are unfortunately many who think that only papers in these journals are worth taking seriously. This is extremely regretful (and is definitely a topic for a separate post) but sadly it's reality. Unless this reality changes, speculation would become respectable only if it's published by Nature and Science. As was pointed out, the Annals of Improbable Research has published ideas that first make us laugh and then make us think. If you look at some of the papers in the journal which have bagged the notorious IgNobel prize, they are actually quite well-supported by data and statistical analysis. Yet regrettably, we will have to wait for at least a few generations before anyone takes the Annals as seriously as Cell or PNAS. One of my main points in the last post was that we need to make speculation not just easier but more respectable and official again. And for better or worse, for now it's going to become respectable and official only if the top journals give it a public platform.

Ultimately, what purpose will all this serve? Many of the benefits have been described; as a commentator succinctly mentioned in the earlier post, it would give the publication of preliminary ideas an official sounding board. The way the present system is set up- and the recent example makes it clear- scientists are just going to be dissuaded from publishing tentative, bold observations and ideas because of the impending public backlash. But the commentator also pointed out another important dividend; the process would perhaps make the true nature of science clear to a public which is too often fed information in black and white sound bytes.

There are rules for doing, interpreting and publishing science, just like there are rules for how to raise children. And just as the rules for raising children wonderfully break down in the face of reality, so do the rules of actual scientific research. Real science is as messy as real child rearing. It's only fair that the public knows about this process.

The beauty of it is that it all comes together in the end. The baby turns into a fine young man or woman, and science continues to flourish.

Note: The comments section makes it clear that we need to distinguish between two kinds of articles, those suitable as "Preliminary Results" and those suitable as "Speculation". The two kinds may certainly overlap; the arsenic paper would thus be primarily in a "Preliminary Results" section but the hypothesis about arsenated DNA backbones would put it into a "Speculations" section and in this case the speculation would not be toned down but kept in.

8 comments:

  1. The problem is how can you know whether a paper is speculative or not. Reviewers are not necessarily experts is all the areas covered in a paper. Unless they try to reproduce what is reported (which is obviously impractical), they cannot know whether the claims are true or not. So as long as what it is written makes sense, it is assumed to be true, if it wasnt assumed to be true, basically every paper would be speculative.

    The reviewers of the arsenic paper were unable to spot the problems with the methodology, if they were, they would have already asked to tone down the conclusions or, possibly, reject the paper.

    ReplyDelete
  2. I think part of the confusion is that you're been talking about two different types of papers in your post.

    Your suggestions for the arsenic paper is basically to tone down the broad, sweeping statements, and present their findings as "Hmm, that's strange", rather than "We believe X is Y". However, all of your other examples weren't "Hmm, the evidence we currently have is strange", they were "We have no evidence, but theory indicates that X may be possible". As I mentioned for the last post, these are different types of papers.

    With respect to the arsenic paper, you're not really arguing for a "Speculations" section, you're arguing for a "Preliminary Findings" section. Which I would also support. There certainly are a number of exciting, potentially revolutionary findings that merit broader discussion prior to be nailed down conclusively enough to be a regular research paper. (I'll note the protein crystallography field has done something similar, where people have published a paper when they got a crystal & preliminary diffraction data, but well before they solved the structure.)

    Calling it "Speculations" is just contributing to the confusion. I would maintain that the broad, excited statements made in and regarding the arsenic paper (which you advocate removing) would rightly be called "speculations" rather than "conclusions", given the lack of evidence. In my mind, moving the paper to a "Speculations" section would be more likely to have them kept in, rather than removed/toned down.

    ReplyDelete
  3. RM: Thank you for your articulate comment. You are right that the arsenic paper was more of a "Preliminary Findings" paper. Hinting at the possibility of arsenic-backboned DNA would have turned it into a "Speculations" paper. Thus an article can of course belong to both categories at once. But you are right that having that paper in the "Speculations" section would have meant keeping the implications for novel DNA in and not toning them down; I think the paper would still have to be re-worded a little though.

    Anon: Speculation papers would mainly be of the "Theory says that X might be observable" kind noted by RM; both the Dyson sphere paper and the Turin paper I noted in the previous post would squarely belong to this category. I have no idea what happened with the review of the arsenic paper though. I guess that will probably always be an unknown.

    ReplyDelete
  4. [Disclaimer: previously published, not recently.]

    I don't think there is a problem to get speculation out there, but to make reasonable speculation heard. (Reasonably heard, literary.) And there is a well traveled route for that, it is up to the authors to make themselves clear. For example, and to hook on to the theme, the arsenic paper could have been divided into two accompanying ones, a "preliminary findings" and one "possible hypotheses".

    Now I don't think they would have gotten that into Science and Nature, which may be recognizing the problem you claim. But at least it shows that those journals messed up and that it isn't a problem of the current system but of authors pushing it.

    "If you look at some of the papers in the journal which have bagged the notorious IgNobel prize, they are actually quite well-supported by data and statistical analysis."

    One of my favorites because it is locally important is the paper on fish farts, not that I have read it. As opposed to the usual waters of submarines, the closed and island populated waters of nordic post-glacial coastal waters makes soundings, um, ... noisy. That the swedish military told themselves that fish schools were russian submarines is of course important to elucidate. (And the military fishing for signals resulted in both actual (red) herrings and ignoble results.)

    ReplyDelete
  5. Um, since when have organic chemists think "that only papers in [nauture, science] are worth taking seriously"?

    If I recall correctly, organic chemistry held much contempt for those two journals, and preferred JCAS, or Angenuie Chemie.

    ReplyDelete
  6. I feel like this whole concept stinks of overengineering. This whole FWS thing would have been fine if 1) NASA didn't hold a press conference, and if 2) they had presented their data, as is, and said, "we think it is possible that these bugs are putting arsenic in their DNA and further experiments will be conducted to assess the validity of this postulate". As a last sentence in the conclusions section. Why do we need a new section for that?

    I'm not even sure that people would want to publish in a "speculations" section of the journal; let the cat out of the bag and someone may scoop you. On the other hand, people may abuse it to throw in tons of shitty ideas in the hopes that one of them catches the fish. Perhaps it's my bias - chalk it up to a general disdain for theory over experiment - but my rough feeling is that if you can't do the leg work to prove your theory, you don't deserve as much credit for it.

    Ideas are cheap. Hard work and strong critical thinking are diluted traits in science now.

    ReplyDelete
  7. I don't believe conservatism will help keep fraud out particularly well, and I wouldn't mind the idea of speculation being given a forum. As a previous commenter said, though, the journals have to understand what is speculative and what is not, and what a reasonable level of evidence is to support a given conclusion.

    The embarrassments that have happened in scientific publishing haven't mostly occurred because the journals got punked by a researcher with more competence than conscience (well, not entirely - Sames-Sezen). In many of the cases (the papers Derek Lowe criticized in BMCL, hexacyclinol, the reactome paper, antiaromatic diaza-12-cyclediiums, and NaH-mediated (catalyzed?) oxidations), there was nowhere near enough data to support their points, or stupid mistakes or obvious problems that should have been caught by journals doing their job. Sames-Sezen is mostly a clear case of fraud, though the manner in which the contested papers were retracted entangles the journals in that ugly web at least a lttle bit. In most of these cases, the embarrassment is because journals didn't do what they people paying them perceived them to be doing. I wasn't paying much attention, but with the NMR-induced enantioselective addition fraud, the correction was relatively rapid and I didn't think ACIEE took heat for it, because it wasn't reasonably within their power to do so. If journals review a manuscript reasonably and are beaten by fraud, I don't think people's contempt will fall on the journal, but on the authors, on whom the fault should devolve.

    The intellectual grappling that people expect journals and their editors to do doesn't seem unreasonable - they are what researchers ought to expect from themselves. [At least in some of the cases, authors should not be getting off free of blame - it's their work, and if bloggers who have just read it have more cogent criticisms of your work than they do, well, they didn't do their job (one of Lowe's Laws - nobody should know more about your research than you]. If journals can't do these things well, while readers are expecting them to, this is a problem. Giving them more jobs (the need to distinguish speculation from preliminary results from completed work that needs a few more experiments) isn't going to make that better.

    ReplyDelete
  8. Above, Chem Droid makes the comment: “Hard work and strong critical thinking are diluted traits in science now.” In a comment partially directed toward me in the previous thread ChemDroid states: “What's going on in science has all the hallmarks of a bubble. PhDs applying for jobs that were set aside as BA/BS/MS positions. Fraud becoming an acceptable way to jockey for scarce academic positions. A veritable red tide of shoddy journals and shoddy science to fill the shoddy journals.”
    While somewhat off topic, I think it might be interesting to pursue these points. I know Curious Wavefunction to be well read, and if willing, I think input here would be enlightening.
    I think that ChemDroid is confusing several issues. In the first place, cited in the previous comment thread, a Scientific American article on diminishing publication volume (which SA uses as a measure of productivity) hardly supports a stance of “A veritable red tide of shoddy journals and shoddy science to fill the shoddy journals.” Personally, I find the direct correlation between numbers of papers published and productivity to be a questionable one.
    I didn’t get beyond the paywall on the Jon Lerher NewYorker article but I did read this post on Lehrer’s blog: http://www.wired.com/wiredscience/2010/12/the-mysterious-decline-effect/ I don’t want to go into Lehrer’s ideas in detail here, except to say that this is about science as an intrinsically human and messy endeavor. I believe, in fact, that Lehrer might like Curious Wavefunction’s proposal to make speculation official. What Lehrer seems to be saying seems to be analogous to what Curious Wavefunction is saying, that is: because human biases exist we cannot always initially be sure of our conclusions and so replications (and alternative investigations) are necessary. At any rate, I see no reason to cite this article as backing for a decline in current science as opposed to some previous golden age.
    Elsewhere online there has been a blog discussion of jobs in chemistry (See: http://chemjobber.blogspot.com/2010/12/future-of-chemistry-jobs-recap-and.html) ChemDroid cites a PhD bubble. But somehow seems to correlate this with shoddiness. I see this problem as an intersection of economics and politics with population demographics. Scientists born ahead of the leading cusp of the baby boom wisked into professorships, industrial and research positions, as Universities and the economy at large expanded; those graduating now are struggling as a contraction takes place. Although I do believe that political remediation methods are possible, the underlying phenomena is not due to generational differences in hard work or critical thinking skills. I see this as just another example of human analytical fallibility, it is much easier to think that all those unemployed people are bums, than to worry that one’s own position is or ought to be vulnerable to their competition, or work to make overall changes in the system.

    ReplyDelete

Markup Key:
- <b>bold</b> = bold
- <i>italic</i> = italic
- <a href="http://www.fieldofscience.com/">FoS</a> = FoS