ENCODE, Apple Maps and function: Why definitions matter

ENCODE (Image: Discover Blogs)

Remember that news-making ENCODE study with its claims that “80% of the genome is functional”? Remember how those claims were the starting point for a public relations disaster which pronounced (for the umpteenth time) the "death of junk DNA"? Even mainstream journalists bought into this misleading claim. I wrote a post on ENCODE where I expressed surprise at why anyone would be surprised by junk DNA to begin with.

Now Dan Graur and his co-workers from the University of Houston have published a meticulous critique of the entire set of interpretations from ENCODE. Actually let me rephrase that. Dan Graur and his co-workers have published a devastating takedown of ENCODE in which they pick apart ENCODE’s claims with the tenacity and aplomb of a vulture picking apart a wildebeest carcass. Anyone who is interested in ENCODE should read this paper, and it’s thankfully free.

First let me comment a bit on the style of the paper which is slightly different from that in your garden variety sleep-inducing technical article. The title – On the Immortality of Television Sets: Function in the Human Genome According to the Evolution-Free Gospel of ENCODE – makes it clear that the authors are pulling no punches, and this impression carries over into the rest of the article. The language in the paper is peppered with targeted sarcasm, digs at Apple (the ENCODE results are compared to AppleMaps), a paean to Robert Ludlum and an appeal to an ENCODE scientist to play the protagonist in a movie named "The Encode Incongruity". And we are just getting warmed up here. The authors spare little expense in telling us what they think about ENCODE, often using colorful language. Let me just say that if half of all papers were this entertainingly written, the scientific literature would be so much more accessible to the general public.

On to the content now. The gist of the article is to pick apart the extremely liberal, misleading and scarcely useful definition of “functional” that the ENCODE group has used. The paper starts by pointing out the distinction between function that’s selected for and function that’s merely causal. The former definition is evolutionary (in terms of conferring a useful survival advantage) while the latter is not. As a useful illustration, the function of the human heart that is selected for is to pump blood while the function that’s causal is an additional weight of 300 grams and a capacity for producing thumping sounds.

The problem with the ENCODE data is that it features causal functions, not selected ones. Thus for instance, ENCODE assigns function to any DNA sequence that displays a reproducible signature like binding to a transcription factor protein. As this paper points out, this definition is just too liberal and often flawed. For instance a DNA sequence may bind to a transcription factor without inducing transcription. In fact the paper asks why the study singled out transcription as a function: “But, what about DNA polymerase and DNA replication? Why make a big fuss about 74.7% of the genome that is transcribed, and yet ignore the fact that 100% of the genome takes part in a strikingly “reproducible biochemical signature” – it replicates!”

Indeed, one of the major problems with the ENCODE study seems to be its emphasis on transcription as a central determinant of “function”. This is problematic, since as the authors note, there's lots of sequences that are transcribed which are known to have no function. But before we move on to this, it’s worth highlighting what the authors call “The Encode Incongruity” in homage to Robert Ludlum. The Encode Incongruity points to an important assumption in the study; the implication that a biological function can be maintained without selection and that the sequences with “causal function” identified by ENCODE will not accumulate deleterious mutations. This assumption is unjustified.

The paper then revisits the five central criteria used by ENCODE to define “function” and carefully takes them apart:

1. “Function” as transcription.

This is perhaps the biggest bee in the bonnet. First of all, it seems that ENCODE used pluripotent stem cells and cancer cells for its core studies. The problem with these cells is that they display a much higher level of transcription than other cells, so any deduction of function from transcription in these cells would be exaggerated to begin with. But more importantly as the article explains, we already know that there are three classes of sequences that are transcribed without function; introns, pseudogenes and mobile elements (“jumping genes”). Pseudogenes are an especially interesting example since they are known to be inactive copies of protein-coding genes that have been rendered dead by mutation. Over the past few years as experiments and computational algorithms have annotated more and more genes, the number of pseudogenes has gone up even as the number of protein-coding genes has gone down. We also know that pseudogenes can be transcribed and even translated in some cells, especially of the kind used in ENCODE, just as we know that they are non-functional by definition. Similar arguments apply to introns and mobile elements, and the article cites papers which demonstrate that knocking these genes out doesn't impair function. So why would any study label these three classes of sequences as functional just because they are transcribed? This seems to be a central flaw in ENCODE.

A related point made by the authors is statistical in which they say that the ENCODE project has sacrificed selectivity for sensitivity. There are some simple numerical arguments that point to the large number of false positives inherent in sacrificing selectivity for sensitivity. In fact this is a criticism that goes to the heart of the whole purpose of the ENCODE study:

“At this point, we must ask ourselves, what is the aim of ENCODE: Is it to identify every possible functional element at the expense of increasing the number of elements that are falsely identified as functional? Or is it to create a list of functional elements that is as free of false positives as possible. If the former, then sensitivity should be favored over selectivity; if the latter then selectivity should be favored over sensitivity. ENCODE chose to bias its results by excessively favoring sensitivity over specificity. In fact, they could have saved millions of dollars and many thousands of research hours by ignoring selectivity altogether, and proclaiming a priori that 100% of the genome is functional. Not one functional element would have been missed by using this procedure.”

2. “Function” as histone modification

Histones are proteins that pack DNA into chromatin. The histones then undergo certain chemical modifications called post-translational modifications that cause the DNA to unpack and be expressed. ENCODE used the presence of 12 histone modifications as evidence of “function”. This paper cites a study that found a very small proportion of possible histone modifications associated with function. Personally I think this is an evolving area of research but I too question the assumption of having a function associated with most histone modifications.

3. “Function” as proximity to regions of open chromatin

In contrast to histone-packaged DNA, open chromatin regions are not bound by histones. ENCODE found that 80% of transcription sites were within open chromatin regions. But then they seem to have committed the classic logical fallacy of inferring the opposite, that most open chromatin regions are functional transcription sites (there’s that association between transcription and function again). As the authors note, only 30% or so of open chromatin sites are even in the neighborhood of transcription sites, so associating most open chromatin sites with transcription seems to be a big leap to say the least.

4. “Function” as transcription-factor binding.

This to me is another huge assumption inherent in the ENCODE study, especially as a chemist. As I mentioned in my earlier post, there are regions of DNA that might bind transcription factors (TFs) just by chance through a few weak chemical interactions. The binding might be extremely weak and may be a quick association-dissociation event. To me it seemed that in associating any kind of transcription-factor binding with function, the ENCODE team had inferred biology from chemistry. The current analysis gives voice to my suspicions. As the authors say, transcription sites are usually very short which means that TF-binding “look-alikes” may arise in a large genome purely by chance. Any binding to these sites may be confused with real TF-binding sites. The authors also cite a study in which only 86% of TF-binding sites in a small sample of 14 sites showed experimental binding to a TF. Extrapolating to the entire genome, it could mean that a fraction of the conjectured TF-binding sites may actually bind TFs.

5. “Function” as DNA methylation.

This is another instance in which it seems to me that biology is being inferred from chemistry. DNA methylation is one of the dominant mechanisms of epigenetics. But by itself DNA methylation is only a chemical reaction. The ENCODE team built on a finding that negatively correlated gene expression with methylation in CpG (cytosine-guanine) sites.  Based on this they concluded that 96% of all CpGs in the genome are methylated, and therefore functional. But again, in the absence of explicit experimental verification, CpG methylation cannot be equated with gene expression. At the very least this indicates follow-up work which will need to confirm the relationship. Until then the hypothesis that CpG methylation implies function will have to remain a hypothesis.

So what do we make of all this? It’s clear that many of the conclusions from ENCODE have been extrapolations devoid of hard evidence. 

But the real fly in the ointment is the idea of “junk DNA” which seems to have evoked rather extreme opinions that have ranged from proclaiming junk DNA as extinct to proclaiming it as God. Both these opinions perform a great disservice to the true nature of the genome. The former reaction virtually rolls the red carpet for “designer” creationists who can now enthusiastically remind us of how each and every base pair in the genome has been lovingly designed. At the same time, asserting that junk DNA must be God is tantamount to declaring that every piece of currently designated junk DNA must forever be non-functional. While the former transgression is much worse, it’s important to amend the latter belief. To do this the authors remind us of a distinction made by Sydney Brenner between “junk DNA” and “garbage DNA”. There’s the rubbish we keep and the rubbish we discard, but some rubbish may potentially turn useful in the future. At the same time, rubbish that may be useful in the future is not rubbish that’s useful in the present. Just because some “junk DNA” may turn out to have a function in the future does not mean most junk DNA will be functional. In fact as I mentioned in my post, the presence of large swathes of non-functional DNA in our genomes is perfectly consistent with standard evolutionary arguments.

The paper ends with an interesting discussion about “small” and “big” science that may explain some of the errors in the ENCODE study. The authors point out that big science has generally been in the business of generating and delivering data in an easy-to-access format. Small science has been much more competent in then interpreting the data. This does not mean that scientists working on big science are incapable of data interpretation; what it means is that the very nature of big data (and the time and resource allocation inherent in it) may make it very difficult for these scientists to launch the kinds of targeted projects that would do the job of careful data interpretation. Perhaps, the paper suggests, ENCODE’s mistake was in trying to act as both the deliverer and the interpreter of data. In the authors’ considered opinion, ENCODE “tried to perform a kind of textual hermeneutics on the 3.5 billion base-pair genome, disregarded the rules of scientific interpretation and adopted a position of theological hermeneutics, whereby every letter in a text is assumed a priori to have a meaning”. In other words, ENCODE seems to have succumbed to an unfortunate case of ubiquitous pattern seeking from which humans often suffer.

In any case, there are valuable lessons in this whole episode. The mountains of misleading publicity it generated, even in journals like Science and Nature, were a textbook study in media hype. As the authors say:

“The ENCODE results were predicted by one of its lead authors to necessitate the rewriting of textbooks (Pennisi 2012). We agree, many textbooks dealing with marketing, mass-media hype, and public relations may well have to be rewritten.”

From a scientific viewpoint, the biggest lesson here may be to always keep fundamental evolutionary principles in mind when interpreting large amounts of noisy biological data under controlled laboratory conditions. It’s worth remembering the last line of the paper:

“Evolutionary conservation may be frustratingly silent on the nature of the functions it highlights, but progress in understanding the functional significance of DNA sequences can only be achieved by not ignoring evolutionary principles…Those involved in Big Science will do well to remember the depressingly true popular maxim: “If it is too good to be true, it is too good to be true.”

The authors compare ENCODE to AppleMaps, the direction-finding app in the iPhone that notoriously bombed when it came out. Yet AppleMaps also provides a useful metaphor. Software can evolve into a useful state. Hopefully, so will our understanding of the genome.

First published on the Scientific American Blog Network.

Uncle Syd's idea for funding new assistant professors

I was leafing (virtually of course) through old issues of "Current Biology" when I came across a thought-provoking, slightly tongue-in-cheek essay by Sydney Brenner in an issue from 1994. Brenner used to write a regular column for the magazine and his thoughts ranged from improving lab conditions to junk DNA; as is characteristic of Brenner's incisive mind, almost all the columns give you something new to think about. 

It was Brenner's ("Uncle Syd") fictitious letter to an assistant professor ("Dear Willie") just starting out in his new career that caught my eye. Perhaps the column can offer some ideas to assistant professors floundering in this gloomy age of funding crunches and declining job prospects. After acknowledging the fundamental and paradoxical difficulty that new professors who need funding the most don't have any experience in getting it, Brenner comes up with a framework - the BISCUIT:

"I have for long entertained an elegant solution to this difficulty, and that is to found a bank, BISCUIT (Bank of International Scientific Capital and Unpublished Information and Techniques), that will lend scientific capital to first-time grant applicants and others in need. It will not only lend ideas for research but also loan experiments that have been carried out but have not been published. We have to be careful with the latter, because although such holdings are of high value they could undergo instant depreciation if someone else does the experiment and publishes the result. Where, you ask, does the bank get its capital? No problem. I know a number ofscientists who have a surplus of scientific ideas and lots of experiments that they find too boring to write up and these 'wealthy' individuals would be the first investors. The bank would also continue to receive deposits. Once we got going, everything would be fine, because the borrowers would not only have to pay back capital but we would charge interest so that our holdings grew. And, of course, if any depositor were to suffer a catastrophic career collapse, he could withdraw all of his capital and start again. The beauty of it is that he would get new, up-to-date ideas and experiments and, in this way, his original deposit, although used a long time ago, will have retained its value and will not have been corroded by time. I am amazed that in these times ofhigh-powered service industries nobody has thought of doing this before, but perhaps that's because it is only scientists who will profit from the BISCUIT bank."

Reference: Current Biology, 1994, 4, 10, 956

On toxic couches and carcinogens: Chemophobia, deconstructed.

Last week I attended a great session on chemophobia at ScienceOnline 2013 headed by Carmen Drahl and Dr. Rubidium. The session emphasized how "trigger words" - alarmist phrases judiciously placed in the middle of otherwise well-intentioned paragraphs - can make people believe that something is more serious than it is. The session also reinforced the all-important point that context makes all the difference when it comes to chemistry.

Sadly I could not read a recent post about flame retardants in couches on the Scientific American Guest Blog without remembering some of these caveats. The post unfortunately seems to me to present a first-rate example of how well-intentioned opinion and advice can nonetheless be couched in alarmism and assertions drawn out of context. It evidences lapses that are common in chemophobic reporting. Let me state upfront that my argument is as much about the tone and message of the post as it is with the pros and cons of the scientific evidence (although there's some highly questionable scientific conclusions in there). Some of my analysis might look like nitpicking, but the devil is often in the details.

The article is written by Sarah Janssen, an M.D. Ph.D. who is worried about supposedly "toxic chemicals" in her couch. In this case the chemical turns out to be something called Chlorinated Tris. Dr. Janssen is apparently so worried that she has already decided that her family should sit on the floor/carpet or eat at the table than be exposed to the couch. At the end of the post she says that she will look forward to the time when she can buy a "toxic-free couch".

The trigger words start coming at you pretty much right away:

"A nationwide study of 102 couches revealed that my couch, among others tested, contains OVER A POUND of chlorinated Tris, a cancer-causing chemical removed from children’s pajamas in the 1970s and now listed on California’s Proposition 65 list of carcinogens"

Observe how ONE POUND is capitalized, as if the capitalization makes any additional arguments in favor of the compound's toxicity superfluous. But we all know that the dose depends on the context; there's more than one pound of lots of chemical substances in almost every piece of furniture that I use, but the weight by itself hardly makes the material harmful. In fact since the weight of a typical couch is at least 20 pounds, I wouldn't expect to find any less than one pound of a flame-retardant substance in it. The point is that simple manipulations like capitalization enhance the public's perception of impact, and doing this without a good reason sends the wrong message.

Now let's look at the chemical itself, Chlorinated Tris, or TRIS(1,3-DICHLORO-2-PROPYL) PHOSPHATE (TDCPP) in chemical parlance (there, did the capitalization make it sound more sinister?). Googling this chemical turns up a bunch of newspaper articles without primary references. How about a more formal source, in this case a June 2011 report by the California EPA? Scientifically inclined readers will find lots of interesting data in there and it's clear that chlorinated tris has a variety of observable and potentially concerning effects on cells. But for me the most important part of the report talked about a study on the effects of TDCPP on cancer risk in a group of 289 workers at a TDCPP plant between 1956 to 1980. The operative line in that paragraph is the following:

"The authors concluded that although the SMR (standard mortality rate) from lung cancer was higher than expected, overall there was no evidence linking the lung cancers to TDCPP exposure because all three cases with lung cancer were heavy to moderate cigarette smokers. Small sample size and the inability to account for confounding factors make it difficult to draw conclusions from this study."

In addition the paragraph states that p-values (a measure of statistical significance) could not be calculated because of small sample size. Now this study was done with people who have literally lived and breathed in a TDCPP-rich environment for almost thirty years. If anyone should suffer the ill-effects of TDCPP it should probably be this group. And still the conclusions were dubious at best, so one wonders if merely sitting on a couch would do anything at all.

As is usually the case, the report has much more information about the effects of TDCPP in mice and here you do see evidence of tumor formation. But the sample sizes are again small. More importantly, what's the dosage of TDCPP that causes statistically significant cancers to appear in mice? It's 80 mg per kilogram per day. This would translate to 5.6 grams per day for a 70-kg human being. And although I haven't read all the original studies with mice, I am assuming that this amount would have to be ingested, inhaled or injected. So no, unless you are out of supplies in a nuclear holocaust and are forced to survive by actually eating the foam from your couch, you would most likely not get cancer from simply sitting on a couch with TDCPP in it. And even this tenuous conclusion comes from studies with mice; as indicated above, the data is far from clear for humans. In fact I would guess that the probability of suffering an obesity-induced heart attack from sitting for long periods on a couch exceeds the probability of getting cancer from TDCPP.

Now that doesn't mean that I am claiming that TDCPP has no harmful effects in humans. But it's clear that at the very least we need to get much more rigorous data to establish a causal relationship with any kind of confidence. For now the evidence just doesn't seem to be there. Claiming that simply sitting on a TDCPP-filled couch could cause cancer, with any kind of probability, is really no more than a theory disconnected from data.

It gets worse. The post later talks about the smoke from fire retardant-containing furniture "putting firefighters' health at greater risk of cancer". When you click on that link it takes you to an article in the San Francisco Chronicle documenting the story of a firefighter named Stefani who is a cancer survivor. The firefighter had expressed concerns about his cancer being linked to smoke inhalation from household items like furniture. But here's what the article itself says at one point:

"The relationship between Stefani's job and transitional cell carcinoma is less clear...there's no hard evidence yet that chemicals contribute to this condition, said Dr. Kirsten Greene, a UCSF assistant urology professor who helped run the study."

When an expert who ran the study questions the link between cancer and flame retardants, it should give you pause for thought (on a related note, kudos to the SFC for reporting the skepticism).

Finally, there's no better way to drive home the pernicious influence of "chemicals" than to demonstrate their existence in the bodies of every species on the planet:

"But flame retardants aren’t just polluting our homes—they are polluting the world, literally. During manufacturing, use and disposal, these chemicals are released into the environment where they can be found in air, water, and wildlife. Birds, fish, mammals including whales and dolphins and animals living far from sources of exposure, such as polar bears in the Arctic, have been found to have flame retardants in their bodies."

But take a look at the polar bears paper. Notice that we have now switched from TDCPP to brominated flame retardants, a different category of compound. If there's anything chemists know, it's the fact that function follows structure; no chemist would assume that TDCPP and brominated ethers would have the same effects without explicit evidence. The subsequent paragraph describing a variety of other non-carcinogenic effects also talks about brominated compounds. The continuity in the article would have you believe that we are still talking about TDCPP and couches. More importantly though, chemical substances are not all toxic just because they show up in multiple species, and bioaccumulation does not automatically translate into carcinogenicity (as is clear from the polar bear study). Since the advent of human civilization there have been thousands of synthetic molecules that have been dispersed in the environment, and the vast majority of them co-exist in peace with other species. But the most important point here is that it's extremely hard to extrapolate these studies to the conclusion that sitting on your couch may expose you to a carcinogen; that kind of extrapolation pretty much ignores dose, context, statistical significance and species-specific differences and lumps all "flame retardants" into the same category without allowing for compound-specific effects.

Trigger words proliferate the rest of the post: "dangerous chemicals", "harmful chemical substances", "toxic-free couch"...the list goes on. I don't want it to sound like I am picking on this particular post or author; sadly this kind of context-free alarmism is all too common in our chemophobic culture. But articles like this keep on making one thing clear: the details matter. You really cannot write a report like this without looking into details like statistics, nature of test organisms, dosage, method of administration, controls, sample size and species-specific differences. Lack of attention to these details is often a common hallmark of articles propagating chemophobia. If you ignore these details you are not really reporting science, you are simply reporting a gut feeling. And gut feelings are not exactly good metrics for making policy decisions.

Chemistry Nobel Prizes and second acts

While having a discussion about Barry Sharpless with a fellow chemist the following random question occurred to me: How many chemists have contributed at least one significant piece of work to science after winning the Nobel Prize? Sharpless himself invented click chemistry after being recognized for asymmetric synthesis.

After winning the prize many laureates bask in the sunlight and cut down significantly on research. Some use their newfound celebrity status to advance educational or social causes. Others can disappear from the world of science altogether. But a select few plough on as if they had never won the prize and the names of these persistent souls should be more widely known. 

A couple of examples spring to mind. At the top of the list should be the man who continued to do research that turned out to be so important that he shared another Nobel Prize for it: Fred Sanger, who got his first Nobel Prize for protein sequencing and the second for DNA sequencing.

R. B. Woodward must also join Fred Sanger at the top. Although technically he won just one Nobel Prize he would have undoubtedly won another (for the Woodward-Hoffmann rules) had he been alive (and cut down on the scotch and cigarettes) and was probably a candidate for a third (ferrocene and organometallic chemistry).

But we can find examples much further back. Ernest Rutherford received an anomalous 1908 prize in chemistry instead of physics, but after winning it he continued to churn out Nobel Prize-winning students and pathbreaking discoveries, including the demonstration of artificial or induced radioactivity. Harold Urey received a prize in 1934 for his discovery of deuterium, but he contributed at least partially to the founding of origins of life chemistry with Stanley Kubrick (ok it's Miller, huge typo, but I am going to let this stay; I can only surmise that I was fantasizing about a Kubrick movie titled "2001 A Space Odyssey: RNA World Edition") in the 50s. In this context Manfred Eigen is another interesting example; after being recognized for methods to study fast reactions in 1967, Eigen embarked on an origins of life career and among other things came up with the idea of "hypercycles".

Interestingly, Linus Pauling who is widely considered the greatest chemist of the twentieth century had exhausted his quota of important chemical discoveries by the time he won the Nobel Prize in 1954. Until then he had already revolutionized the theory of chemical bonding, proposed the alpha helix and beta sheet structures in proteins and had contributed a host of other important ideas ranging from rules for predicting the structure of minerals to proposals about the mechanism of action of antibodies and enzymes. Pauling continued to do interesting work scientific after this, but most of his attention was focused on arms disarmament for which he won the peace prize in 1962. For the rest of his life Pauling was occupied partly with interesting scientific projects and partly with controversial medical projects. Nothing that he did came close to his accomplishments before winning the prize, although given the magnitude of those accomplishments I guess we can cut him some slack.

I don't have an exhaustive list here but I doubt if there's many more names to add here. Doing one piece of significant science in your life seems hard enough so perhaps it's unrealistic to expect two. And yet there are a few hardy souls who achieve this.

Never a simple staple

Stapled peptide. Image: Nature

Carmen Drahl of C&EN has a pretty nice overview of what's happening in the world of stapled peptides. Those who have followed this development are aware of both the promise and the caveats in the area. Interest in constrained peptides was sparked by the observation that helices are involved in a lot of important protein-protein interactions and therapeutics areas. Cancer seems to be an especially prominent target.

The simple thinking is that if you constrain a helix to its helical shape using various chemical strategies, you could minimize the entropic cost of helix formation. In addition, by modulating the properties of whatever chemical group you are using to constrain the structure, you could possibly get better affinity, cell penetration and stability. The strategies used have been varied, from olefin linkers to hydrocarbon linkers, and they have a long history. But stapled peptides have been especially noteworthy; the staple in this case is a short hydrocarbon linker connecting one residue to another a few angstroms away. Based on this idea, groups from Harvard launched a biotech startup named Aileron. Stapled peptides, it seemed were on a promising upward trajectory.

Except that nothing in biology is as simple as it seems. A recent report from groups in Australia and Genentech demonstrated that a previously promising stapled helix which inhibits the antiapoptotic protein Bim is not as potent as it seemed. The problems seem to lie with the fundamental helical structure of the molecule; the paper found that in adding the hydrocarbon staple, you are perturbing some productive intramolecular hydrogen bonding interactions between the residues, and this results in an unfavorable enthalpy of binding.

It's worth reading Carmen's piece not just for the science but for the potential conflict of personalities; science is a human game after all. In this case the conflict arises not only from the Australia/Genentech group's doubts about the original Harvard peptide but the Harvard group's contention that the Australia/Genentech team was in fact looking at the wrong peptide.

Hopefully the misunderstanding will be cleared soon but one thing is clear; this debate seems to be good for science. Constraining helices and other peptides is a completely legitimate way of thinking about improving drug potency and properties, but the devil is in the details. Whenever you put on a constraint you are going to subtly perturb the structure of the molecule, and what seems important here is to investigate how these subtleties will affect the desired functions and properties.

Ultimately stapled peptides are no different from a dozen other drug design strategies. There are cases where they will work and cases where they won't, and any efforts to tease these differences apart will advance the science. Neither stapled peptides nor any other kind of specific drug modification is going to give us the silver bullet. But we already knew this, didn't we?

#Arseniclife reviews: Missing the forest for the trees

In this year's ScienceOnline conference I co-moderated a productive session on peer review in which I pointed out how overly conservative or agenda-driven peer reviews can prevent the publication of legitimate science. Now here's a case where the opposite seems to have occurred; highly questionable science making it through the filter of peer review as easily as particles of dust would make it through a sieve with penny-sized holes.

Thanks to the Freedom of Information Act, USA Today and a couple of other scientists got their hands on the reviews of the infamous #arseniclife paper. There were three reviewers of the study, and all of them approved the paper for publication.

What's interesting is how effortlessly the reviewers miss the forest for the trees. We of course have the benefit of hindsight here, but it's still striking how all three reviews simply swallow the flawed paper's basic and potentially textbook-changing paradigm - the substitution of arsenic for phosphorus - right off the bat. Once they accept this basic premise, all their other objections can simply be seen as nitpicking and window dressing. Only one reviewer asks questions that come close to questioning the absence of phosphorus in the medium, but even he or she quickly veers off course. Another calls the paper a "rare pleasure" to read, seemingly unaware that the pleasure which the paper has provided comes from an extraordinarily ambitious claim that needs to be vetted as closely as possible.

In fact the reviewers ask good questions about vacuoles seen in the bacterium, about better standards for some of the experiments, about better methods to quantify arsenic in its various forms. They even ask a few very chemical questions regarding bond distances. But all these questions are somewhat beside the point since they flow from a fundamentally flawed belief.

When I was in graduate school, the most important thing that my advisor taught me was to always question the assumptions behind a study. If you don't do this, it's easy to be seduced by the technical details of the experiment and to let these details convince you that the basic premise is validated. That's what seems to me to have happened here. All the reviewers seem to have been sucked into legitimate and interesting questions about minutiae. But all the time they forget that what really needs to be questioned is the giant assumption from which all the minutiae have been derived, an assumption that we now know does not stand up to scrutiny. There's an important lesson here.