Surprises in physics: From black bodies to the accelerating universe

Max Planck's revolutionary 

1900 discovery that energy in 

the subatomic world exists as discrete 

packets marked the beginning of a century of spectacular 

surprises in physics 

Surprises rank high on the list of things that make science a source of everlasting delight. When it comes to being surprised scientists are no different from the general public. Just like children on their birthdays being surprised by unexpected gifts, scientists revel in the surprises that nature whips up in front of them. Surprises in science point to something deeper: the mystery, excitement and things unseen and unknown that always keep the scientific enterprise interesting, amusing and profound.

Scientific surprises are not always tantamount to important findings. For instance the experimental detection of the Higgs boson was a great achievement but it was not exactly surprising since the theoretical prediction had been made much earlier. The theoretical prediction itself emerged as a logical extension of ideas which were then in the air, and no fewer than six individuals contributed to its genesis. The fact is that some discoveries in science are important, some are surprising and some are both. It is this third category that is the most memorable, and rare is the scientist who finds himself or herself the beneficiary of a discovery that is both surprising and important.

Thus it’s worth taking a look back and charting some of the most important surprises in science, those that either forced us to rethink a lot of our assumptions or, in rare cases, those that truly changed our view of the world. Here I present a list of surprises in physics drawn from roughly the last one hundred years. Physics is in equal parts a theoretical and an experimental science, so its surprises come from both arenas. Feel free to note other surprises which I might have missed, and remember that surprising is not the same as important and important is not the same as surprising (and therefore this list excludes many of the most important discoveries in physics).

1900 – Max Planck: The beginning of quantum mechanics – Max Planck’s solution to the thorny theoretical problem of blackbody radiation started what was surely the greatest revolution in physics since Isaac Newton’s time. Planck was famously trying to explain the dependence of the intensity of radiation emitted by a black body on its frequency and temperature. The conservative German physicist was essentially struggling with what we today call “curve fitting”, finding the right equation for a graph of experimental data. He realized that the only way he could do this was to imagine a formula with a new constant called h and the assumption that energy was emitted by the blackbody only in certain discrete units. Planck intended his solution as a mathematical fix and not a representation of reality, and it was only when Albert Einstein appeared on the scene that he realized the nature of the radical transformation in our view of reality that he had inaugurated. The age of the quantum had dawned.

1905 – Albert Einstein: Special relativity – In 1905 Einstein wrote five papers that changed the face of physics, but perhaps only one of these can be called truly surprising. This was his famous paper setting out the principles of the special theory of relativity. There were of course many profound surprises in the theory – including time dilation and length contraction – but the biggest fundamental surprise was probably the iron rule that required the speed of light to be constant. The breakthrough was counterintuitive, bold and clearly revolutionary.

1909 – Ernest Rutherford: The atomic nucleus – Until Rutherford’s time the structure of the atom was nebulous, the best guess coming from J. J. Thomson who imagined it as a “plum pudding” with negatively charged electrons uniformly embedded in a positively charged sphere. It was in Manchester that Rutherford, along with two associates and a 70 pound grant from the Royal Institution, performed his famous gold foil experiment in which he shot alpha particles at a thin gold foil. If Thomson’s uniform and homogeneous atomic model had been correct the particles should have been scattered equally in all directions. Instead a very few of them came right back at the experimenters. The novelty and surprise of the result is best captured by Rutherford himself: “It was almost as incredible as if you had fired a 15 inch shell at a piece of tissue paper and it came right back at you”. Rutherford’s discovery signaled the beginning of nuclear physics.

1911 – Heike Kamerlingh Onnes: Superconductivity – Until 1911 there were divergent views regarding the behavior of matter at very low temperature. Kamerlingh Onnes from Leiden settled the debate in 1911 by finding, to his utter surprise, that mercury at 4.2 degrees Kelvin lost all resistance to the flow of current. The discovery was so surprising and significant that Onnes received a Nobel Prize only two years later. Superconductivity continues to be a frontier of physics research, both pure and applied.

1913 – Niels Bohr: The quantum theory of the atom – Bohr’s surprise was the postulation that electrons can occupy only certain energy levels in atoms and emit photons with energies defined by Planck’s formula when they make transitions. This really marked the beginning of the weird, probabilistic world of quantum mechanics, encapsulated by the question, “How does the electron know when to make a transition”?

1917 – Albert Einstein: General relativity – Einstein followed up his annus mirabilis with his crowning achievement – a theory of gravitation – in 1917. General relativity sealed the seamless meld of space and time and postulated gravity not as Newton’s force but as a property of spacetime itself, arising from the curvature of spacetime by the presence of mass.

1919 – Arthur Eddington: The bending of starlight – Leading an expedition to the island of Principe lying off the West coast of Africa, Eddington confirmed one of the most startling predictions of general relativity, the bending of starlight by the gravitational field of a massive object. This was not strictly a surprise since Einstein had predicted it, but in science any theory is only as good as the data that supports it. The discovery not only splashed Einstein’s face all over the world’s papers but also signified a rare bond of scientific friendship between two nations which had just ended a devastating war with each other.

1926 – Werner Heisenberg: The Uncertainty Principle – Heisenberg’s Uncertainty Principle – which has since then been hijacked and affixed to remote and bizarre notions in popular culture – drew a wedge between our assumptions of determinism and the nature of reality. Along with the phenomenon of quantum entanglement it essentially captures all the weirdness about the quantum world. Since then we have been struggling with the legacy of the true nature of quantum mechanics.

1928 – Paul Dirac: The Dirac equation – The Dirac equation is one of the true glories of theoretical physics. In a single line that can be neatly stated on a cocktail napkin, it combines special relativity with quantum mechanics. But most surprisingly, it proposes a new form of matter – antimatter. Dirac found this implication of the equation so unsettling that for some time he assumed that the missing twin of the electron predicted by the theory must be a proton. But in 1932 Carl Anderson found the elusive positron in cosmic rays. Both Dirac and Anderson got well-deserved Nobel Prizes for prediction and discovery.

1929 – Edwin Hubble: The Expanding Universe – Just as the Dirac equation is one of the true glories of theoretical physics, so is Hubble’s discovery of an expanding universe in the subsequent year one of the true glories of astronomy. Again, the possibility of an expanding universe has been conjectured by Russian Alexander Friedmann and Belgian priest Georges Lemaitre. But this possibility was so bizarre that even Einstein could not take it seriously at first. It took Hubble and his painstaking studies of nebulae at the Mount Wilson observatory in California to etch the truth in stone.

1938 – Otto Hahn and Fritz Strassman: Nuclear fission – Both Enrico Fermi’s group and the husband-wife duo of Irene and Frederic Joliot-Curie narrowly missed discovering fission in the mid-thirties. Curiously, the chemist Ira Noddack had hypothesized the phenomenon in 1934 but was ignored; one wonders if everyone would have taken her more seriously had she been a male scientist, preferably German or English. Yet it was a female scientist, Lisa Meitner, who interpreted Hahn and Strassman’s novel breaking up of uranium into barium and calculated the startling magnitude of energy release, paving the way toward both the peaceful and the destructive uses of atomic energy. Fission was thought to be so improbable that great theorists like Oppenheimer and Bethe had done calculations arguing against its existence. And yet here it was, an elegant theory demolished by an ugly, consequential fact.

1939 – Robert Oppenheimer and Hartland Snyder: Black holes – In the same issue of the journal Physical Review in which Niels Bohr and John Wheeler published their famous liquid drop model of fission, Oppenheimer and his student Hartland Snyder published the first description of what we now call black holes. The paper’s conclusions were surprising and bizarre and followed on the heels of a decade’s worth of groundbreaking research on the implications of general relativity for stellar evolution by Subrahmanyan Chandrasekhar, Lev Landau, Fritz Zwicky and others. Curiously, Oppenheimer never followed up on this paper and in fact displayed a remarkable indifference to general relativity for the rest of his life.

1947 – Willis Lamb: The Lamb Shift – The Lamb Shift was one of those subtle experimental measurements that lay the foundation for a whole new field. In 1947 Lamb and Retherford measured a tiny, surprising difference in energy levels of the electron in the hydrogen atom; the Dirac equation predicted identical energies for both these levels. This tiny difference concealed a wealth of important aspects of quantum field theory, and feverish and pathbreaking theoretical work explaining the shift culminated in the creation of the strikingly accurate theory of quantum electrodynamics (QED) by Schwinger, Feynman, Dyson and Tomonaga by the end of the decade.

1956 – Yang and Lee: Parity violation – Parity is one of the most basic properties of the atomic world. In physics a parity transformation is an operation that changes the sign of the spatial coordinates of an object. For instance a parity transformation on a right-handed coordinate system will turn it into a left-handed one. Until 1956 it was rightly and reasonably believed that nature should not care whether we use right or left handed coordinate systems and that the laws of physics should be the same for both. Yet in 1956, Chen Ning Yang and Tsung Dao Lee showed that parity was not conserved for certain particles which decayed through the weak nuclear force, so that the products of their decay depended on whether their spins were aligned parallel or antiparallel to a magnetic field. This result was confirmed by C. S. Wu in an elegant experiment at Columbia University. The discovery that at a very fundamental level nature actually cares about parity was so radical that Yang and Lee received a Nobel Prize only one year later. Wu was left out.

1964 – Penzias and Wilson: The cosmic microwave background – The discovery of an expanding universe had given at least some scientists the inkling of a purported “big bang” origin for the universe. But until 1964 this was just a hypothesis, so much so that the Cambridge astrophysicist Fred Hoyle had coined the phrase “Big Bang” as a derogatory dig at the theory. 1964 changed all that. Arno Penzias and Robert Wilson’s discovery – made using a small Bell Labs antenna in Holmdel, New Jersey – of the remnant of the big bang in the form of a steady background radiation hum was one of the most significant and surprising discoveries in twentieth century science, and one which has led to at least two Nobel Prizes. With Penzias and Wilson our current view of the universe acquired a firm foundation.

1964 – John Bell: Bell’s Theorem – Einstein had never been happy with quantum mechanics, all the way until his death, partly because his 1933 EPR paradox paper seemed to imply non-local, seemingly faster-than-light communication of information between particles. In spite of quantum mechanics’s spectacular predictive power, Einstein was convinced that there must be some kind of “hidden variables” which could explain what seem like contradictory and bizarre properties of the quantum world and bring locality into the picture. In 1964 an obscure Irish physicist named John Bell laid these questions to rest with a remarkable and surprising theorem that can be derived and stated using high school algebra. Because of Bell’s obscurity it seemed to come out of left field. In short, Bell’s Theorem said that no local, hidden variable theory of quantum mechanics could explain its basic postulates. The theorem thus established quantum weirdness – and especially entanglement – as a fundamental part of the physical universe. An equally surprising development was the set of experiments by Alain Aspect, Stuart Freedman, John Clauser and others in the 70s and 80s confirming Bell’s Theorem.

1973 – Gross, Politzer and Wilczek: Asymptotic freedom – Asymptotic freedom refers to the strange weakening of the force between quarks as they come near. This is a surprising and totally counterintuitive mechanism of particle interaction since in case of every other force (gravitation, electromagnetism with opposite charges and the weak force), the interactions become stronger with decreasing distance. The discovery of asymptotic freedom put the quantum theory of the strong force (called quantum chromodynamics) on a sure footing.

1986 – Bednorz and Mueller: High temperature superconductivity – In 1986 the world of physics and materials science was shaken by the discovery of ceramic materials like yttrium copper barium oxide which become superconducting at “high” temperatures, in some cases above the boiling point of liquid nitrogen (77 degrees kelvin) and in others as high as 138 degrees kelvin. This not only caused a revolution in our understanding of superconductivity (these materials were ceramics, not metals) but also led to dreams of practically harnessing the phenomenon. The dream has not quite come true yet, but ceramic superconductivity continues to be a thriving field of physics research. IBM’s Alex Muller and George Bednorz received a Nobel Prize in the very next year; so groundbreaking and surprising was the discovery.

1998 – Perlmutter, Reiss, Schmidt and others: The accelerating universe – Even after Hubble discovered the expanding universe, for the longest time scientists believed that the rate of expansion was slowing: since gravity was an attractive force the belief seemed entirely logical. Observations of distant supernovae in the early 90s completely turned this picture on its head. It was found that the universe was not only expanding but the expansion was speeding up. Einstein’s cosmological constant was resurrected and a new entity, dark energy, was postulated as a placeholder to account for the process. This was by any definition a surprising and counterintuitive discovery wherein experiment drove our understanding of reality without any major theoretical input.

A few observations emerge from this partial list of surprising discoveries in physics. The most important is that while lone individuals made discoveries in the early period, most findings in the later period were the work of groups of researchers or at least research duos. Many times more than one team made the same discovery. The discoveries were also a mix of theoretical and experimental work; in many cases theory and experiment were equally important since it’s one thing to postulate a surprising result but quite another to actually observe it in front of your eyes. I also find it interesting that no truly surprising fundamental discovery has emerged in the twenty-first century yet, although we are just getting started.

One thing is clear from the list, that physics will continue to make surprising discoveries and that our understanding of the universe will never be complete. We can all look forward to that.

Added: Chad Orzel of "Uncertain Principles" has his own interesting picks.

Adapted from a previous post on Scientific American Blogs.