The 100 most highly cited papers of all time: Tools, not ideas

A rather obscure paper by biochemist
Oliver Lowry is the most highly cited
scientific paper of all time
(Image: beckerarchives)

Nature has published a comprehensive list of the top 100 most highly cited papers of all time and the list is well worth a look for what it reveals about what's really important in science, what people perceive as being important and in fact how science progresses.

I like the list because it confirms something that I have written about on the blog a few times: science is as much a tool-driven revolution as an idea-driven one. This is evidenced by the fact that almost all of the most highly cited papers in that list are about useful techniques rather than great ideas and discoveries. Thus the expansion of the universe, the structure of DNA and the theory of relativity don't rank on the list but Sanger's gene sequencing method, an algorithm for comparing protein and DNA sequences (ClustalW) and a popular statistical technique do. And the dominance of many of these came about because of computers, so the selection also speaks very highly to the rise of computer technology.

Now one can argue that the reason the structure of DNA does not rank on the list is because its importance is so obvious that it has become a textbook fact and does not need to be cited. This is true, and the list does nothing to denigrate the significance of such household ideas (and it also does not elevate the importance of tools above ideas). But what it does convey is that the really visible papers are those which gave researchers practical tools rather than profound ideas. It also tells us that what's in the textbook is not always what's used the most by scientists in their day to day work, even if it may be an important component of their background knowledge.

The selection is dominated by papers from protein biochemistry, bioinformatics and statistics. The most highly cited paper documents the use of the Folin phenol reagent which even researchers consider a "tad outdated". This big enchilada leads the pack with a whopping 305,000 citations; the next best paper with 213,000 citations is a dot on the horizon by comparison. The genomics revolution meant a huge unmet need both for methods to quantify, isolate and sequence biomolecules and to process, compare and analyze these sequences. Not surprisingly, specific lab protocols for isolating and studying biomolecules and computer algorithms for analyzing their information content rank at the top of the list. The latter phenomenon is a great example of two technological revolutions - cheap software and hardware and the applications they engender - piggybacking on each other.

Statistics and crystallography - two other disciplines with huge practical ramifications - also feature prominently on the list. Computers were again tremendously important in the practical realization of these disciplines. For instance the program SHELX made it possible to analyze complex diffraction data from x-ray diffraction. Similarly the original paper describing the Kaplan-Meier test is highly cited: today the reason the Kaplan-Meier test is so popular is partly because it has found critical use in fields like clinical trials, but more importantly because it has been incorporated in popular software tools like Matlab and R which even non-statisticians can use efficiently. As statistics becomes more user friendly it is likely that these papers will be even more highly cited, but the people who cite them might just be using their products the way a gardner uses a lawnmower without really understanding how it works.

Computational chemistry also makes an appearance and density functional theory (DFT) which has caused a revolution in the accurate and fast calculation of molecular properties is what raises the profile of the field. The two most highly cited papers in this area include one by by Lee, Yang and Parr and another by Becke. The interesting thing about the amalgamation of these two methods is that they have now become ensconced in an abbreviation (B3LYP) which is used as part of a recipe by thousands of graduate students, postdocs and professional researchers to do all kinds of quantum chemical calculations, from simple energy determinations to complex reaction studies. But the prominence of B3YLP also goes to the heart of the reason why a paper can be highly cited: it becomes so firmly enmeshed into a standard arsenal of tools that most people start blindly citing it and using it as a black box. I have used the technique myself dozens of times including in a few papers, but I don't remember the last time I took a look at the original paper.

One of the most puzzling facts about the list is the almost complete absence of papers from physics and astronomy. Maybe I am missing something here but the absence tempts me to reiterate something that I have noted before: that chemistry and biology, much more so than physics, are about tools rather than ideas. Nonetheless I find the paucity of physics papers puzzling since there is no reason a priori why physics should be devoid of practical tools nor why ideas in physics should not be highly cited. One intriguing point noted in the article is that physicists might be less prone to citing each other's papers than biologists, and if true this is really a cultural phenomenon responsible for the absence of physics papers on that list.

Nevertheless the list is quite readable. It tells us that science is ultimately as much about the mundane use of tools and techniques as it is about the genesis and distribution of profound ideas. Sadly the former view is not half as much appreciated by the general public as the latter. The list provides a great counterpoint to the idea of science as a series of paradigm shifts. It tells us that in science, as in many other fields, what matters ultimately is what we can use.

A citation against citations

In the latest issue of Angewandte Chemie, Stanford chemistry professor Richard Zare has some cogent words of advice for assessing young faculty members when they are up for tenure. Zare has written the article partly as a response to what he sees as an excessive use and abuse of citation indices throughout the world in judging tenure-worthy achievements.

As Zare notes, the h-index which has been adopted in part because it seems to measure both quality and quantity is particularly ill-equipped to measure early success in research. This is mostly because (and this is one of the biggest arguments against any citation metric) the significance of most research does not become obvious until much later. Zare cites the example of physicist Steven Weinberg's paper unifying the electroweak force; while it was cited only a few times in the first few years, it is now one of the most highly cited papers in the history of physics. Another obvious example is Watson and Crick's DNA paper which gathered very few citations in its fledgling years. Thus citations, if they make sense at all, make much more sense in one's later career.

Yet there is a disturbing trend of universities worldwide adopting citation statistics to drive tenure decisions. In some countries like China, researchers are even awarded cash prizes for trying to publish large numbers of papers in both leading and minor science journals. As a recent article bemoaned, this has led to journals like Nature and Science being flooded by papers of dubious quality from certain countries. Not only can such practices create bias against papers from these countries, but they harm the global enterprise of science as a whole by emphasizing publication at the expense of genuinely interesting work.

Far better is to try to objectively judge the promise of young investigators by evaluating their impact on specific subfields. To this end Zare describes the system adopted by the Stanford chemistry department which puts the highest emphasis on 10 to 15 letters of recommendation from around the world. The letter writers are essentially asked to answer the simple question, "Has the investigators' work in his/her chosen area led to new understanding and directions in the field?". Everything else comes second, including the number and authorship rank of papers along with various indices. As Zare says,

We do not look into how much funding the candidate has brought to the university in the form of grants. We do not count the number of published papers ; we also do not rank publications according to authorship order. We do not use some elaborate algorithm that weighs publications in journals according to the impact factor of the journal. We seldom discuss h-index metrics, which aim to measure the impact of a researcher's publications. We simply ask outside experts, as well as our tenured faculty members, whether a candidate has significantly changed how we understand chemistry.

I find it very surprising - and encouraging - that the amount of money brought in does not play a major role in determining tenure. If true, this seems to go against disturbing current trends that are geared toward evaluating professors similar to sales managers at Macy's or hedge fund managers on Wall Street.

There is one caveat to what seems to be an otherwise cogent and role model-worthy tenure policy adopted by the Stanford chemistry department. Just as the true impact of research does not become clear until years later, the true value of ideas also does not become clear by asking 10, 15 or even 100 referees. Although this helps, it can mask the fact that most original scientific contributions at least partly challenge conventional wisdom, and keepers of the faith are almost always reluctant in endorsing such contributions. What should really matter is not whether a young scientist's work has led to new truths, but whether it has been interesting enough to spark a flurry of research activity that in turn may lead to minor or major truths. In science, being interesting is more important than being right, and tenure committees should take this fact into account.

Are chemists much more secretive and obsessive than physicists?

Derek said it on his blog today and I have been saying it for some time. The physics community did itself and others a great service by floating ArXiv, which has become the standard venue for publication of premium physics papers focused on theory and computation. As Derek asks, why isn't there such a free service started by chemists for their community?

I concur, and a related question I have concerns being able to look at citations. The APS website (which hosts the JACS-equivalent physics journal Physical Review among others) allows readers to view the number of citations for all its papers and therefore allows us to sort papers by citations. No such feature exists for ACS or Wiley chemistry journals; as far as I know one has to log in to a paid site like Web of Science to be able to view citations.

This leads me to a question of psychology. Are chemists much more secretive and obsessive about their data and results compared to physicists? Are they much more self-conscious about revealing the impact (or lack thereof) of their publications to the public? Does some vestigial culture of secrecy going back to the alchemists' creed still linger in our minds?

It could not have to do with the status of the field since many branches of physics are as cutting edge as fields of chemistry. The first explanation that comes to my mind has to do with the color green. It's pretty clear that compared to physics, many fields of chemistry such as medicinal chemistry and materials science have money-making written on them. Unlike most physicists, chemists can patent their molecules and make money from them to a much greater extent. If this is the case, then a control group might be that of engineers. Is a reluctance to make publications or citations easily available also prevalent among the engineering community?

Yet such money-making results constitute only one part of the chemical literature. What about the several academic chemistry papers that have no tangible commercial potential? Why not make them available for free and make their citations available? I don't know the right explanation for this habit, but simple inertia and lack of vigorous discussion and initiative seem to clearly play a role. I do hope it's not because of a statistically significant difference between chemists and physicists that causes the former to hoard results.

The way I see it, the chemistry community clearly should borrow a page from the physicists. Firstly, certain kinds of chemistry papers (just like theoretical papers on ArXiv) need to be made available for free. Secondly, we all deserve a look at citation data which should not be so inaccessible and expensive. If science is supposed to be a community enterprise, this seems to be the least we can do.