Field of Science

Modular complexity and the problem of reverse engineering the brain

Bell's number calculates the number of connections between
various components of a system and scales exponentially
with those components (Image: Science Magazine).
I have been reading an excellent collection of essays on the brain titled "The Future of the Brain" which contains ruminations on current and future brain research from leading neuroscientists and other researchers like Gary Marcus, George Church and the Moser husband and wife pair who won last year's Nobel prize. Quite a few of the authors are from the Allen Institute for Brain Science in Seattle. In starting this institute, Microsoft co-founder Paul Allen has placed his bets on mapping the brain…or at least the mouse visual cortex for starters. His institute is engaged in charting the sum total of neurons and other working parts of the visual cortex and then mapping their connections. Allen is not alone in doing this; there’s projects like the Connectome at MIT which are trying to do the same thing (and the project’s leader Sebastian Seung has written a readable book about it).

Now we have heard prognostications about mapping and reverse engineering brains from more eccentric sources before, but fortunately Allen is one of those who does not believe that the singularity is around the corner. He also seems to have entrusted his vision to sane minds. His institute’s chief science officer is Christof Koch, former professor at Caltech, longtime collaborator of the late Francis Crick and self-proclaimed “romantic reductionist” who started at the institute earlier this year. Koch has written one of the articles in the essay collection. His article and the book in general reminded me of a very interesting perspective that he penned in Science last year which points out the staggering challenge of understanding the connections between all the components of the brain; the “neural interactome” if you will. The article is worth reading if you want to get an idea of how even simple numerical arguments illuminate the sheer magnitude of mapping out the neurons, cells, proteins and connections that make up the wonder that’s the human brain.

Koch starts by pointing out that calculating the interactions between all the components in the brain is not the same as computing the interactions between, say, all atoms of an ideal gas since unlike a gas, the interactions are between different kinds of entities and are therefore not identical. Instead, he proposes, we have to use something called Bell’s number Bn which reminds me of the partitions that I learnt about when I was sleepwalking through set theory in college. Briefly for n objects, Bn refers to the number of combinations (doubles, triples, quadruples etc.) that can be formed. Thus, when n=3 Bn is 5. Not surprisingly, Bn scales exponentially with n and Koch points out that B10 is already 115,975. If we think of a typical presynaptic terminal with its 1000 proteins or so, Bn starts giving us serious heartburn. For something like the visual cortex where n= 2 million Bn would be inconceivable, and it's futile to even start thinking about what the number would be for the entire brain. Koch then uses a simple calculation based on Moore’s Law in trying to estimate the time needed for “sequencing” these interactions. For n = 2 million the time needed would be of the order of 10 million years. And as the graph on top demonstrates, for more than 105 components or so the amount of time spirals out of hand at warp speed.

This considers only the 2 million neurons in the visual cortex; it doesn’t even consider the proteins and cells which might interact with the neurons on an individual basis. In addition, at this point we are not even really aware of how neuronal types there are in the brain: neurons are not all identical like indistinguishable electrons. What makes the picture even more complicated that these types may be malleable so that sometimes a single neuron can be of one type while at other types it can team up with other neurons to form a unit that is of a different type. This multilayered, fluid hierarchy rapidly reveals the outlines of what Paul Allen has called the “complexity brake”: he described this in the same article that was cogently critical of Ray Kurzweil's singularity. And the neural complexity brake that Koch is talking about seems poised to make an asteroid-sized impact on our dreams.

So are we doomed in trying to understand the brain, consciousness and the whole works? Not necessarily, argues Koch. He gives the example of electronic circuits where individual components are grouped separately into modules. If you bunch a number of interacting entities together and form a separate module, then the complexity of the problem reduces since you now have to only calculate interactions between modules. The key question then is, is the brain modular, and how many modules does it present? Commonsense would have us think it is modular, but it is far from clear how we can exactly define the modules. We would also need a sense of the minimal number of modules to calculate interactions between them. This work is going to need a long time (hopefully not as long as that for B2 million) and I don’t think we are going to have an exhaustive list any time soon, especially since these are going to be composed of different kinds of components and not just one kind. But it's quite clear that whataver the nature of these modules, delineating their particulars would go a long way in making the problem more manageable.

Any attempt to define these modules are going to run into problems of emergent complexity that I have occasionally written about. Two neurons plus one protein might be different from two neurons plus two proteins in unanticipated ways. Also if we are thinking about forward and reverse neural pathways, I would hazard a guess that one neuron plus one neuron in one direction may even be different from the same interaction in the reverse direction. Then there’s the more obvious problem of dynamics. The brain is not a static entity and its interactions would reasonably be expected to change over time. This might interpose a formidable new barrier in brain mapping, since it may mean that whatever modules are defined may not even be the same during every time slice. A fluid landscape of complex modules whose very identity changes every single moment could well be a neuroscientist’s nightmare. In addition, the amount of data that captures such neural dynamics would be staggering since even a millimeter sized volume of rat visual tissue requires a few terabytes of data to store all its intricacies. However, the data storage problem pales in comparison to the data interpretation problem.

Nevertheless this goal of mapping modules seems far more attainable in principle than calculating every individual interaction, and that’s probably the reason Koch left Caltech to join the Allen Institute in spite of the pessimistic calculation above. The value of modular approaches goes beyond neuroscience though; similar thinking may provide insights into other areas of biology, such as the interaction of genes with proteins and of proteins with drugs. As an amusing analogy, this kind of analysis reminds me of trying to understand the interactions between different components in a stew; we have to appreciate how the salt interacts with the pepper and how the pepper interacts with the broth and how the three of them combined interact with the chicken. Could the salt and broth be considered a single module?

If we can ever get a sense of the modular structure of the brain, we may have at least a fighting chance to map out the whole neural interactome. I am not holding my breath too hard, but my ears will be wide open since this is definitely going to be one of the most exciting areas of science around.

Adapted from a previous post on Scientific American Blogs.

The breadth of Carl Djerassi's scientific work

One measure of the diversity of Carl Djerassi's work can be gained from a simple diversity analysis of the top 50 of his most cited papers - he wrote more than 1200. The articles range from pure synthetic methodology to novel natural product isolation and structure determination to marine steroid biosynthesis to circular dichroism to artificial intelligence to mechanistic organic chemistry.

Here's a sampling based on a listing of those top 50 in Scifinder. And this of course does not even consider his very substantial second career in the arts and humanities. A towering intellect and a role model for the rest of us to emulate. 


  • By Budzikiewicz, H.; Wilson, J. M.; Djerassi, Carl
  • From Journal of the American Chemical Society (1963), 85(22), 3688-99.  |  Language: Unavailable, Database: CAPLUS
  • By Gunatilaka, A. A. Leslie; Gopichand, Yalamanchili; Schmitz, Francis J.; Djerassi, Carl
  • From Journal of Organic Chemistry (1981), 46(19), 3860-6.  |  Language: English, Database: CAPLUS
  • By Moffitt, William; Woodward, R. B.; Moscowitz, A.; Klyne, W.; Djerassi, Carl
  • From Journal of the American Chemical Society (1961), 83, 4013-18.  |  Language: Unavailable, Database: CAPLUS
  • By Djerassi, Carl; Gorman, Marvin
  • From Journal of the American Chemical Society (1953), 75, 3704-8.  |  Language: Unavailable, Database: CAPLUS
  • By Djerassi, Carl; Engle, Robert R.
  • From Journal of the American Chemical Society (1953), 75, 3838-40.  |  Language: Unavailable, Database: CAPLUS
  • By Sheikh, Younus M.; Djerassi, Carl
  • From Tetrahedron (1974), 30(23/24), 4095-103.  |  Language: English, Database: CAPLUS
  • By Djerassi, Carl; Silva, Christopher J.
  • From Accounts of Chemical Research (1991), 24(12), 371-8.  |  Language: English, Database: CAPLUS
  • By Djerassi, Carl; Scholz, Caesar R.
  • From Journal of the American Chemical Society (1948), 70, 417-18.  |  Language: Unavailable, Database: CAPLUS
  • By Grant, Barbara; Djerassi, Carl
  • From Journal of Organic Chemistry (1974), 39(7), 968-70.  |  Language: English, Database: CAPLUS
  • By Lederberg, Joshua; Sutherland, Georgia L.; Buchanan, Bruce G.; Feigenbaum, Edward A.; Robertson, Alexander V.; Duffield, Alan M.; Djerassi, Carl
  • From Journal of the American Chemical Society (1969), 91(11), 2973-6.  |  Language: English, Database: CAPLUS
  • By Djerassi, Carl; Zderic, John A.
  • From Journal of the American Chemical Society (1956), 78, 6390-5.  |  Language: Unavailable, Database: CAPLUS
  • By Eggert, Hanne; Djerassi, Carl
  • From Journal of the American Chemical Society (1973), 95(11), 3710-18.  |  Language: English, Database: CAPLUS
  • By Walser, A.; Djerassi, Carl
  • From Helvetica Chimica Acta (1965), 48(2), 391-404.  |  Language: German, Database: CAPLUS

Pathogen resistance can track with drug penetration, not just with drug number or variety

Here's an interesting paper from a multi-institution group looking at a perpetual problem in antibacterial and antiviral drug therapy - the emergence of mutations and resistance. As one of the authors describes it on her blog, the project started when the team realized that while the conventional wisdom about multiple drugs working better because of increased barriers to multiple mutations in the pathogen is correct, the mechanism for this seemed to be rather fuzzy, especially because the data seemed to show that multiple mutations arise not in tuples but as singles, one at a time.

The authors essentially divided the body into different compartments with viruses and bacteria and then assumed differential penetration of drugs in multidrug therapy into these compartments. They built a mathematical model to track the emergence of mutations as a function of exposure to various drugs. What they found is that imperfect drug penetration into a compartment can allow mutations against that drug to arise one at a time, and quite speedily at that. Thus, even if multiple drugs are being administered, the pathogen may not be exposed to these drugs simultaneously at the relevant concentrations in every compartment: as indicated in the title of the paper, even if the treatment may therefore consist of multitherapy in principle, it leads to "spatial monotherapy" in practice leading to the focused exposure of certain compartments only to single drugs at a time.

The study is interesting because it shows that mutations and resistance may not only be a function of drug number and variety but could also result from the drugs not making it to their site of action effectively - it's something that's not surprising in retrospect but it's nice to firm up this belief with a model. It also means that simply administering multiple drugs may not curb resistance if those drugs have vastly different penetration profiles.

If this hypothesis is true however, then you would see some correlation between drug penetration profiles and resistance mutations and not just with the diversity and number of drugs - in part it could explain why certain combinations of drugs even from similar classes work better than others (for instance, it could potentially explain why protease inhibitor X + RT inhibitor Y work better than protease inhibitor A + RT inhibitor B). I am assuming that this kind of cell and compartment penetration data does exist for certain drugs so it would be interesting to model it and track such potential correlations.

Carl Djerassi (1923-2015): Chemist, writer, polymath, cultural icon

R B Woodward, Vladimir Prelog and Carl Djerassi on the beach at a conference in Riga, Latvia. Photo signed by Djerassi and generously gifted to the author by Prof. Jeffrey Seeman, University of Richmond.
Very few scientists of the 20th century have had as much of both a scientific as well as a cultural impact on the world as Carl Djerassi. It is a measure of how many things Djerassi excelled at that even his amazing purely scientific career seems like a distant horizon. While most scientists are quite happy to become world-renowned in their narrow subfield of science, Djerassi polished off multiple fields of chemistry and then reinvented himself as a notable playwright and writer. And he did all this after significantly contributing to one of the greatest social revolutions of the 20th century, if not of all time - safe, affordable and easily accessible contraception for women. Very few technical inventions in history contributed to giving women control over their lives the way the pill did. Scientists are usually not cultural icons, and the wrong people often are, but Djerassi definitely deserves to be one.

On his 90th birthday, my friend, the noted historian of chemistry Jeff Seeman summarized a few of Djerassi's astonishing contributions and honors:


He has published more than 1,200 scientific papers, 350 in the Journal of the American Chemical Society.
He is a chemical “father of the Pill.”
He has published three autobiographies, one memoir, five novels, two nonfiction books, 11 plays, two collections of poetry, three collections of essays and short stories, and one art book.
He has made significant contributions, in cash and in kind, to charitable causes and artistic endeavors.
He has received the National Medal of Science and the National Medal of Technology.
He has been awarded the Priestley Medal, the first Wolf Prize in Chemistry, and many other awards, as well as 32 honorary doctorates.
He has been recognized by Austria with a postage stamp issued in his honor.
He is in constant demand as a lecturer around the world.
He will turn 90 on Oct. 29.
Yet, Carl Djerassi has never been fully satisfied.

I first encountered Djerassi's work when as an undergraduate I studied the octant rule that he, R B Woodward, Bill Moffit and others pioneered to study the configuration of steroids. Nobody really uses it anymore since more sophisticated methods like NMR spectroscopy have superseded it, but as I came to know more about Djerassi's contributions, it amazed me that the same man who published the octant rule also pioneered the use of mass spectrometry in natural products chemistry, unraveled the biosynthesis of several key steroids and alkaloids and also published some of the first papers on the applications of artificial intelligence in organic chemistry. His 1200 papers span the breadth of the discipline, and among younger chemists only Clark Still comes to my mind as someone who had the same diversity of contributions.

Djerassi's autobiography ("Steroids Made It Possible") which is edited by Jeff is wonderful and a real treat. In it he talks about a variety of scientific and private topics, ranging from the letter to Eleanor Roosevelt that got him a college scholarship to his experiments with mescaline to his accidental grin in the photo showing Richard Nixon awarding him the National Medal of Science (he hated Nixon and in fact was on Nixon's silly "enemies" list, but Nixon said something funny right at the moment the photo was taken). The book also contains painful ruminations such as the one about his daughter's struggle with addiction and her suicide. Djerassi was nothing but upfront about his life, both in this memoir as well as his subsequent two books, the latest of which just came out and which I haven't read yet. 


His writings are also studded with sketches of great chemists like R B Woodward, Gilbert Stork, Bill Johnson and E J Corey, most of whom Djerassi counted among his close friends and colleagues. In some sense, a journey through his science is a journey through the development of postwar organic chemistry in its golden age. And speaking of R B Woodward, Djerassi managed to become the highest cited chemist of the 60s - at a time when his friend had already staked his claim as the the greatest organic chemist of the century and continued to publish seminal papers. That's no small feat.
Djerassi was of course also a noted playwright, reinventing himself during the second half of his life and crafting the play "Oxygen" with his fellow chemist Roald Hoffmann for instance. He was a great example of someone who bridged C P Snow's two cultures, inculcating and displaying a wide storehouse of knowledge ranging from philosophy and art to literature and science. His shares in Syntex Corporation where he researched steroids also made him a wealthy man and allowed him to do this. It also enabled him to retire early, amass an enviable private art collection and spend part of every year in London and other parts of Europe writing and giving talks. His fiction is well worth reading, and his characters are as interesting and honest as the science he pioneered.

Most laymen will of course always know Djerassi as one of the fathers of the contraceptive pill, although those of us who are aware of his chemical contributions appreciate that it was but one part of his prolific career. As many of us also know, Djerassi was a favorite on Nobel Prize lists for a long time, and as they did with many other scientists, the Nobel Prize committee did themselves a disservice by not awarding him one. But Djerassi's career more than that of most others indicates the irrelevance of prizes, as honorable as they may be. In that sense Djerassi is like Gandhi. His work was beyond prizes, and considering the social revolution that The Pill brought about, he will always stand not only as one of the scientific greats of the 20th century but as one of its most important human beings. RIP.

Precision medicine is not precision engineering

From the NYT, a plea for not getting carried away with the vision of 'moonshot medicine' as precision engineering. The op-ed is written by Michael Joyner, a doctor at the Mayo Clinic and takes issue with Obama's precision medicine initiative which he apparently underlined in his state of the union address earlier this month. Obama's proposal is nothing new and has been echoed by proponents since the beginnings of the Human Genome Project. The idea of identifying specific genetic variants in patients and then targeting them with specific drugs seems logical enough. But as the article indicates, we have clearly spoken too soon:

The basic idea behind it is that we each have genetic variants that put us at increased or decreased risk of getting various diseases, or that make us more or less responsive to specific treatments. If we can read someone’s genetic code, then we should be able to provide him or her with more effective therapeutic and preventive strategies. 
But for most common diseases, hundreds of genetic risk variants with small effects have been identified, and it is hard to develop a clear picture of who is really at risk for what. This was actually one of the major and unexpected findings of the Human Genome Project. In the 1990s and early 2000s, it was thought that a few genetic variants would be found to account for a lot of disease risk. But for widespread diseases like diabetes, heart disease and most cancers, no clear genetic story has emerged for a vast majority of cases.

Then there's the psychological aspect:

The push toward precision medicine could also lead to unintended consequences based on how humans respond to perceptions of risk. There is evidence that if people believe they are less at risk for a given disease, they feel excessively protected and their behavior gets worse, putting them at increased risk. Likewise, those who feel they are at greater risk, even if the increased risk is small, might become fatalistic, making their behavior worse as well. Then there are the worriers, who might embark on a course of excessive tests and biopsies “just in case.” In a medical system already marked by the overuse of diagnostic tests and procedures, this could lead to even more wasteful spending.

And finally, the whole idea that biology can be thought of as a linear engineering systems is fraught with flaws and uncertainty (it's also not everyday that you hear the word 'omertà'):

Given the general omertà about researchers’ criticizing funding initiatives, you probably won’t hear too many objections from the research community about President Obama’s plan for precision medicine. But I am deeply skeptical. Like most “moonshot” medical research initiatives, precision medicine is likely to fall short of expectations. Medical problems and their underlying biology are not linear engineering exercises, and solving them is more than a matter of vision, money and will.

This is not the first time that complex medical endeavors have been made to sound much simpler than what they are. The breathless initial promise of the ability of gene sequencing to directly lead to new drugs has not been somewhat dampened, but other efforts still echo this promise, most notably the Brain Initiative announced with much fanfare a few years ago. The goals of the initiative are laudable and some of the technologies that are being envisaged are fascinating, but we are still light years away in any incarnation whatever to "mapping the brain", let alone using that information for direct therapeutic intervention.

Unlike precision engineering where we are dealing with moving parts whose behavior is a matter of well-understood physical principles, the emergent chaos of biological systems is a very different matter. We should certainly not stop trying to conquer this frontier but we should also make sure everyone knows how far the horizons are. Biology is not physics.

Boundary value conditions, domain applicability and "American Sniper"

Actor Bradley Cooper in "American Sniper"
General relativity is a generalization of Newtonian mechanics which applies to large objects moving at high speeds that curve spacetime. Similarity, quantum mechanics is a generalization of classical mechanics which applies to very small objects like electrons and photons. Newtonian mechanics thus has a domain of applicability within which it works perfectly well even if it fails to work under the larger rubric of Einsteinian general relativity. Similarly classical mechanics has a domain of applicability within which it is golden. Both are perfectly comfortable within their own boundary value conditions - medium-sized objects and slow speeds for instance.

If you think about it, morality has similar domains of applicability. I was reminded of this comparison when I watched the Clint Eastwood-directed movie "American Sniper" over the weekend. The movie is about Chris Kyle, a celebrated US Navy sniper who over his four tours of duty in Iraq racked up 160 confirmed kills. The film has garnered some controversy for its supposed glorification of war and mindless patriotism. Kyle faced some gut-wrenching decisions in his sniper career in gunning down women and children who were enemy combatants, but by most accounts the decisions were quite simple for him and his conscience seemed clear. Personally, irrespective of where my political sympathies lie, I thought the movie was very gripping and well-made and enjoyed it, and Bradley Cooper was a revelation. After it ended I witnessed a kind of sustained and erie silence as the audience shuffled out of the theater, a silence that I haven't encountered very frequently before.

Since the audience was based in Boston, MA, it is unlikely that the silence was the result of them being enamored and stunned by Chris Kyle's sacrifice and patriotism. Instead I would like to think that the contemplative pause was provoked by a classic and highly problematic dilemma that mankind has faced since it acquired the ability to wage war: how can you vigorously oppose what you think is a highly unnecessary and immoral war and yet support the actions and decisions of individual soldiers like Chris Kyle who are putting their lives on the line with courage? It's a question Americans of every stripe face: when a soldier appears at the airport after returning from a tour, do you condemn him or her or laud their courage?

It's a very difficult question to answer, one whose answer is certainly not black and white and which we will almost certainly never definitively figure out in the foreseeable future. But a comparison with theories in science and especially in physics at least provides an interesting analysis in my opinion.

Here's the point: When Chris Kyle was looking down the barrel of his sniper rifle and had to pick between the life of a ten year-old Iraqi kid carrying a grenade and the life of five American marines, he picked the marines. The choice still wasn't easy but it was the best one to make under the limited parameters and boundary value conditions that he was operating under. The parameters were related to choosing between culpable Iraqi lives and American lives. The boundary value conditions had to do with the narrowly defined parameters of his mission or assignment. You could convince yourself that he had the right solution under those conditions even if you vehemently protested the war at large.

The fact of the matter is that moral decisions, as hard and endlessly unanswerable as they may be, still often operate within a limited domain of applicability. These domains of applicability have announced themselves in every war from the Mexican war to Vietnam. You can be for or against the moral decisions with some degree of confidence within those particular domains, even if you may be entirely opposed or in favor of them when the domains expand. It's like being in favor of Newtonian mechanics when you want to send rockets to the moon while being vehemently opposed to it when you want to send rockets to a neighboring galaxy containing black holes. Or like being in favor of classical mechanics when you want to deal with large blocks of gold while denying the use of that domain of applicability while analyzing individual gold atoms and the energy levels of their electrons. You can therefore potentially consider the political leaders who orchestrated the Iraq war to be war criminals even if you agree with the difficult decisions that Kyle made in those few seconds when he was looking down that gun barrel in Fallujah. 

There may be no answer to moral questions like those surrounding the Iraq war, and it is very likely that we may be forever divided into black and white bins when it comes to answering those questions. But I would like to believe that by dividing the conundrums into domains of applicability and at least agreeing that one's answer may depend on the exact boundary value conditions that one is dealing with, we may possibly reach a consensus and reach out into that gray area that always bedevils the gut-wrenching implications of our morality in both war and peace. Perhaps then we can meet halfway across and reach some kind of consensus, even if that consensus may be occupying the kind of twilight netherworld that entangled electrons do.

Surprises in physics: From black bodies to the accelerating universe

Max Planck's revolutionary 

1900 discovery that energy in 
the subatomic world exists as discrete 
packets marked the beginning of a century of spectacular 
surprises in physics 
Surprises rank high on the list of things that make science a source of everlasting delight. When it comes to being surprised scientists are no different from the general public. Just like children on their birthdays being surprised by unexpected gifts, scientists revel in the surprises that nature whips up in front of them. Surprises in science point to something deeper: the mystery, excitement and things unseen and unknown that always keep the scientific enterprise interesting, amusing and profound.
Scientific surprises are not always tantamount to important findings. For instance the experimental detection of the Higgs boson was a great achievement but it was not exactly surprising since the theoretical prediction had been made much earlier. The theoretical prediction itself emerged as a logical extension of ideas which were then in the air, and no fewer than six individuals contributed to its genesis. The fact is that some discoveries in science are important, some are surprising and some are both. It is this third category that is the most memorable, and rare is the scientist who finds himself or herself the beneficiary of a discovery that is both surprising and important.
Thus it’s worth taking a look back and charting some of the most important surprises in science, those that either forced us to rethink a lot of our assumptions or, in rare cases, those that truly changed our view of the world. Here I present a list of surprises in physics drawn from roughly the last one hundred years. Physics is in equal parts a theoretical and an experimental science, so its surprises come from both arenas. Feel free to note other surprises which I might have missed, and remember that surprising is not the same as important and important is not the same as surprising (and therefore this list excludes many of the most important discoveries in physics).
1900 – Max Planck: The beginning of quantum mechanics – Max Planck’s solution to the thorny theoretical problem of blackbody radiation started what was surely the greatest revolution in physics since Isaac Newton’s time. Planck was famously trying to explain the dependence of the intensity of radiation emitted by a black body on its frequency and temperature. The conservative German physicist was essentially struggling with what we today call “curve fitting”, finding the right equation for a graph of experimental data. He realized that the only way he could do this was to imagine a formula with a new constant called h and the assumption that energy was emitted by the blackbody only in certain discrete units. Planck intended his solution as a mathematical fix and not a representation of reality, and it was only when Albert Einstein appeared on the scene that he realized the nature of the radical transformation in our view of reality that he had inaugurated. The age of the quantum had dawned.
1905 – Albert Einstein: Special relativity – In 1905 Einstein wrote five papers that changed the face of physics, but perhaps only one of these can be called truly surprising. This was his famous paper setting out the principles of the special theory of relativity. There were of course many profound surprises in the theory – including time dilation and length contraction – but the biggest fundamental surprise was probably the iron rule that required the speed of light to be constant. The breakthrough was counterintuitive, bold and clearly revolutionary.
1909 – Ernest Rutherford: The atomic nucleus – Until Rutherford’s time the structure of the atom was nebulous, the best guess coming from J. J. Thomson who imagined it as a “plum pudding” with negatively charged electrons uniformly embedded in a positively charged sphere. It was in Manchester that Rutherford, along with two associates and a 70 pound grant from the Royal Institution, performed his famous gold foil experiment in which he shot alpha particles at a thin gold foil. If Thomson’s uniform and homogeneous atomic model had been correct the particles should have been scattered equally in all directions. Instead a very few of them came right back at the experimenters. The novelty and surprise of the result is best captured by Rutherford himself: “It was almost as incredible as if you had fired a 15 inch shell at a piece of tissue paper and it came right back at you”. Rutherford’s discovery signaled the beginning of nuclear physics.
1911 – Heike Kamerlingh Onnes: Superconductivity – Until 1911 there were divergent views regarding the behavior of matter at very low temperature. Kamerlingh Onnes from Leiden settled the debate in 1911 by finding, to his utter surprise, that mercury at 4.2 degrees Kelvin lost all resistance to the flow of current. The discovery was so surprising and significant that Onnes received a Nobel Prize only two years later. Superconductivity continues to be a frontier of physics research, both pure and applied.
1913 – Niels Bohr: The quantum theory of the atom – Bohr’s surprise was the postulation that electrons can occupy only certain energy levels in atoms and emit photons with energies defined by Planck’s formula when they make transitions. This really marked the beginning of the weird, probabilistic world of quantum mechanics, encapsulated by the question, “How does the electron know when to make a transition”?
1917 – Albert Einstein: General relativity – Einstein followed up his annus mirabilis with his crowning achievement – a theory of gravitation – in 1917. General relativity sealed the seamless meld of space and time and postulated gravity not as Newton’s force but as a property of spacetime itself, arising from the curvature of spacetime by the presence of mass.
1919 – Arthur Eddington: The bending of starlight – Leading an expedition to the island of Principe lying off the West coast of Africa, Eddington confirmed one of the most startling predictions of general relativity, the bending of starlight by the gravitational field of a massive object. This was not strictly a surprise since Einstein had predicted it, but in science any theory is only as good as the data that supports it. The discovery not only splashed Einstein’s face all over the world’s papers but also signified a rare bond of scientific friendship between two nations which had just ended a devastating war with each other.
1926 – Werner Heisenberg: The Uncertainty Principle – Heisenberg’s Uncertainty Principle – which has since then been hijacked and affixed to remote and bizarre notions in popular culture – drew a wedge between our assumptions of determinism and the nature of reality. Along with the phenomenon of quantum entanglement it essentially captures all the weirdness about the quantum world. Since then we have been struggling with the legacy of the true nature of quantum mechanics.
1928 – Paul Dirac: The Dirac equation – The Dirac equation is one of the true glories of theoretical physics. In a single line that can be neatly stated on a cocktail napkin, it combines special relativity with quantum mechanics. But most surprisingly, it proposes a new form of matter – antimatter. Dirac found this implication of the equation so unsettling that for some time he assumed that the missing twin of the electron predicted by the theory must be a proton. But in 1932 Carl Anderson found the elusive positron in cosmic rays. Both Dirac and Anderson got well-deserved Nobel Prizes for prediction and discovery.
1929 – Edwin Hubble: The Expanding Universe – Just as the Dirac equation is one of the true glories of theoretical physics, so is Hubble’s discovery of an expanding universe in the subsequent year one of the true glories of astronomy. Again, the possibility of an expanding universe has been conjectured by Russian Alexander Friedmann and Belgian priest Georges Lemaitre. But this possibility was so bizarre that even Einstein could not take it seriously at first. It took Hubble and his painstaking studies of nebulae at the Mount Wilson observatory in California to etch the truth in stone.
1938 – Otto Hahn and Fritz Strassman: Nuclear fission – Both Enrico Fermi’s group and the husband-wife duo of Irene and Frederic Joliot-Curie narrowly missed discovering fission in the mid-thirties. Curiously, the chemist Ira Noddack had hypothesized the phenomenon in 1934 but was ignored; one wonders if everyone would have taken her more seriously had she been a male scientist, preferably German or English. Yet it was a female scientist, Lisa Meitner, who interpreted Hahn and Strassman’s novel breaking up of uranium into barium and calculated the startling magnitude of energy release, paving the way toward both the peaceful and the destructive uses of atomic energy. Fission was thought to be so improbable that great theorists like Oppenheimer and Bethe had done calculations arguing against its existence. And yet here it was, an elegant theory demolished by an ugly, consequential fact.
1939 – Robert Oppenheimer and Hartland Snyder: Black holes – In the same issue of the journal Physical Review in which Niels Bohr and John Wheeler published their famous liquid drop model of fission, Oppenheimer and his student Hartland Snyder published the first description of what we now call black holes. The paper’s conclusions were surprising and bizarre and followed on the heels of a decade’s worth of groundbreaking research on the implications of general relativity for stellar evolution by Subrahmanyan Chandrasekhar, Lev Landau, Fritz Zwicky and others. Curiously, Oppenheimer never followed up on this paper and in fact displayed a remarkable indifference to general relativity for the rest of his life.
1947 – Willis Lamb: The Lamb Shift – The Lamb Shift was one of those subtle experimental measurements that lay the foundation for a whole new field. In 1947 Lamb and Retherford measured a tiny, surprising difference in energy levels of the electron in the hydrogen atom; the Dirac equation predicted identical energies for both these levels. This tiny difference concealed a wealth of important aspects of quantum field theory, and feverish and pathbreaking theoretical work explaining the shift culminated in the creation of the strikingly accurate theory of quantum electrodynamics (QED) by Schwinger, Feynman, Dyson and Tomonaga by the end of the decade.
1956 – Yang and Lee: Parity violation – Parity is one of the most basic properties of the atomic world. In physics a parity transformation is an operation that changes the sign of the spatial coordinates of an object. For instance a parity transformation on a right-handed coordinate system will turn it into a left-handed one. Until 1956 it was rightly and reasonably believed that nature should not care whether we use right or left handed coordinate systems and that the laws of physics should be the same for both. Yet in 1956, Chen Ning Yang and Tsung Dao Lee showed that parity was not conserved for certain particles which decayed through the weak nuclear force, so that the products of their decay depended on whether their spins were aligned parallel or antiparallel to a magnetic field. This result was confirmed by C. S. Wu in an elegant experiment at Columbia University. The discovery that at a very fundamental level nature actually cares about parity was so radical that Yang and Lee received a Nobel Prize only one year later. Wu was left out.
1964 – Penzias and Wilson: The cosmic microwave background – The discovery of an expanding universe had given at least some scientists the inkling of a purported “big bang” origin for the universe. But until 1964 this was just a hypothesis, so much so that the Cambridge astrophysicist Fred Hoyle had coined the phrase “Big Bang” as a derogatory dig at the theory. 1964 changed all that. Arno Penzias and Robert Wilson’s discovery – made using a small Bell Labs antenna in Holmdel, New Jersey – of the remnant of the big bang in the form of a steady background radiation hum was one of the most significant and surprising discoveries in twentieth century science, and one which has led to at least two Nobel Prizes. With Penzias and Wilson our current view of the universe acquired a firm foundation.
1964 – John Bell: Bell’s Theorem – Einstein had never been happy with quantum mechanics, all the way until his death, partly because his 1933 EPR paradox paper seemed to imply non-local, seemingly faster-than-light communication of information between particles. In spite of quantum mechanics’s spectacular predictive power, Einstein was convinced that there must be some kind of “hidden variables” which could explain what seem like contradictory and bizarre properties of the quantum world and bring locality into the picture. In 1964 an obscure Irish physicist named John Bell laid these questions to rest with a remarkable and surprising theorem that can be derived and stated using high school algebra. Because of Bell’s obscurity it seemed to come out of left field. In short, Bell’s Theorem said that no local, hidden variable theory of quantum mechanics could explain its basic postulates. The theorem thus established quantum weirdness – and especially entanglement – as a fundamental part of the physical universe. An equally surprising development was the set of experiments by Alain Aspect, Stuart Freedman, John Clauser and others in the 70s and 80s confirming Bell’s Theorem.
1973 – Gross, Politzer and Wilczek: Asymptotic freedom – Asymptotic freedom refers to the strange weakening of the force between quarks as they come near. This is a surprising and totally counterintuitive mechanism of particle interaction since in case of every other force (gravitation, electromagnetism with opposite charges and the weak force), the interactions become stronger with decreasing distance. The discovery of asymptotic freedom put the quantum theory of the strong force (called quantum chromodynamics) on a sure footing.
1986 – Bednorz and Mueller: High temperature superconductivity – In 1986 the world of physics and materials science was shaken by the discovery of ceramic materials like yttrium copper barium oxide which become superconducting at “high” temperatures, in some cases above the boiling point of liquid nitrogen (77 degrees kelvin) and in others as high as 138 degrees kelvin. This not only caused a revolution in our understanding of superconductivity (these materials were ceramics, not metals) but also led to dreams of practically harnessing the phenomenon. The dream has not quite come true yet, but ceramic superconductivity continues to be a thriving field of physics research. IBM’s Alex Muller and George Bednorz received a Nobel Prize in the very next year; so groundbreaking and surprising was the discovery.
1998 – Perlmutter, Reiss, Schmidt and others: The accelerating universe – Even after Hubble discovered the expanding universe, for the longest time scientists believed that the rate of expansion was slowing: since gravity was an attractive force the belief seemed entirely logical. Observations of distant supernovae in the early 90s completely turned this picture on its head. It was found that the universe was not only expanding but the expansion was speeding up. Einstein’s cosmological constant was resurrected and a new entity, dark energy, was postulated as a placeholder to account for the process. This was by any definition a surprising and counterintuitive discovery wherein experiment drove our understanding of reality without any major theoretical input.
A few observations emerge from this partial list of surprising discoveries in physics. The most important is that while lone individuals made discoveries in the early period, most findings in the later period were the work of groups of researchers or at least research duos. Many times more than one team made the same discovery. The discoveries were also a mix of theoretical and experimental work; in many cases theory and experiment were equally important since it’s one thing to postulate a surprising result but quite another to actually observe it in front of your eyes. I also find it interesting that no truly surprising fundamental discovery has emerged in the twenty-first century yet, although we are just getting started.
One thing is clear from the list, that physics will continue to make surprising discoveries and that our understanding of the universe will never be complete. We can all look forward to that.

Added: Chad Orzel of "Uncertain Principles" has his own interesting picks.

Adapted from a previous post on Scientific American Blogs.