A citation against citations

In the latest issue of Angewandte Chemie, Stanford chemistry professor Richard Zare has some cogent words of advice for assessing young faculty members when they are up for tenure. Zare has written the article partly as a response to what he sees as an excessive use and abuse of citation indices throughout the world in judging tenure-worthy achievements.

As Zare notes, the h-index which has been adopted in part because it seems to measure both quality and quantity is particularly ill-equipped to measure early success in research. This is mostly because (and this is one of the biggest arguments against any citation metric) the significance of most research does not become obvious until much later. Zare cites the example of physicist Steven Weinberg's paper unifying the electroweak force; while it was cited only a few times in the first few years, it is now one of the most highly cited papers in the history of physics. Another obvious example is Watson and Crick's DNA paper which gathered very few citations in its fledgling years. Thus citations, if they make sense at all, make much more sense in one's later career.

Yet there is a disturbing trend of universities worldwide adopting citation statistics to drive tenure decisions. In some countries like China, researchers are even awarded cash prizes for trying to publish large numbers of papers in both leading and minor science journals. As a recent article bemoaned, this has led to journals like Nature and Science being flooded by papers of dubious quality from certain countries. Not only can such practices create bias against papers from these countries, but they harm the global enterprise of science as a whole by emphasizing publication at the expense of genuinely interesting work.

Far better is to try to objectively judge the promise of young investigators by evaluating their impact on specific subfields. To this end Zare describes the system adopted by the Stanford chemistry department which puts the highest emphasis on 10 to 15 letters of recommendation from around the world. The letter writers are essentially asked to answer the simple question, "Has the investigators' work in his/her chosen area led to new understanding and directions in the field?". Everything else comes second, including the number and authorship rank of papers along with various indices. As Zare says,

We do not look into how much funding the candidate has brought to the university in the form of grants. We do not count the number of published papers ; we also do not rank publications according to authorship order. We do not use some elaborate algorithm that weighs publications in journals according to the impact factor of the journal. We seldom discuss h-index metrics, which aim to measure the impact of a researcher's publications. We simply ask outside experts, as well as our tenured faculty members, whether a candidate has significantly changed how we understand chemistry.

I find it very surprising - and encouraging - that the amount of money brought in does not play a major role in determining tenure. If true, this seems to go against disturbing current trends that are geared toward evaluating professors similar to sales managers at Macy's or hedge fund managers on Wall Street.

There is one caveat to what seems to be an otherwise cogent and role model-worthy tenure policy adopted by the Stanford chemistry department. Just as the true impact of research does not become clear until years later, the true value of ideas also does not become clear by asking 10, 15 or even 100 referees. Although this helps, it can mask the fact that most original scientific contributions at least partly challenge conventional wisdom, and keepers of the faith are almost always reluctant in endorsing such contributions. What should really matter is not whether a young scientist's work has led to new truths, but whether it has been interesting enough to spark a flurry of research activity that in turn may lead to minor or major truths. In science, being interesting is more important than being right, and tenure committees should take this fact into account.