How Niels Bohr predicted Rydberg atoms

 

In Niels Bohr's original 1913 formulation of the quantum atom, the Bohr radius r was proportional to n^2, n being the principal quantum number. Highly excited states would correspond to very large values of n and Bohr predicted these "giant" atoms would exist. Since the volume scales as r^3 or n^6, for n=33 you should see a "hydrogenic" atom a billion times larger than a ground state hydrogen atom. However, no spectral lines corresponding to such atoms were observed. So was Bohr's theory wrong?

No! Bohr pointed out that unlike physicists, *astronomers* had observed faint spectral lines in the spectra or stars and nebulae, consistent with his theory. Because of the large proportion of gas and low density, he predicted such highly excited states would exist.

Because of the extremely low densities, these excited states could live for as long as 1 second - a lifetime for an atom. In 1957, astronomers looking for electron-proton recombination in the interstellar medium serendipitously observed spectra from hydrogen atoms for n=110! In the 1970s, after Bohr's death, the advent of tunable dye lasers finally made it possible to observe these excited states in the lab. Because of their long lifetimes and huge electric dipole moments, these atoms have potential applications in quantum computing.

These "atoms" are called Rydberg atoms because Johannes Rydberg had hypothesized about these large-quantum-number states in the 19th century. But Bohr provided a physical basis and an explanation, so they should really be called Rydberg-Bohr atoms at the least. Today, Rydberg atoms have diverse applications ranging from lasers to quantum computing to plasma physics to radio receivers for military applications. But it all goes back - almost as an afterthought - to Bohr's original pioneering 1913 paper and should be recognized as such.

Complementarity And The World: Niels Bohr’s Message In A Bottle

Werner Heisenberg was on a boat with Niels Bohr and a few friends, shortly after he discovered his famous uncertainty principle in 1927. A bedrock of quantum theory, the principle states that one cannot determine both the velocity and the position of particles like electrons with arbitrary accuracy. Heisenberg’s discovery foretold of an intrinsic opposition between these quantities; better knowledge of one necessarily meant worse knowledge of the other. Talk turned to physics, and after Bohr had described Heisenberg’s seminal insight, one of his friends quipped, “But Niels, this is not really new, you said exactly the same thing ten years ago.”

In fact, Bohr had already convinced Heisenberg that his uncertainty principle was a special case of a more general idea that Bohr had been expounding for some time – a thread of Ariadne that would guide travelers lost through the quantum world; a principle of great and general import named the principle of complementarity.

Complementarity arose naturally for Bohr after the strange discoveries of subatomic particles revealed a world that was fundamentally probabilistic. The positions of subatomic particles could not be assigned with definite certainty but only with statistical odds. This was a complete break with Newtonian classical physics where particles had a definite trajectory, a place in the world order that could be predicted with complete certainty if one had the right measurements and mathematics at hand. In 1925, working at Bohr’s theoretical physics institute in Copenhagen, Heisenberg was Bohr’s most important protégé had invented quantum theory when he was only twenty-four. Two years later came uncertainty; Heisenberg grasped that foundational truth about the physical world when Bohr was away on a skiing trip in Norway and Heisenberg was taking a walk at night in the park behind the institute.

When Bohr came back he was unhappy with the paper Heisenberg had written, partly because he thought the younger man seemed to echo his own ideas, but more understandably because Bohr – a man who was exasperatingly famous for going through a dozen drafts of a scientific paper and several drafts of even private letters – thought Heisenberg had not expressed himself clearly enough. The 42-year-old kept working on the 26-year-old until the latter admitted that “the uncertainty relations were just a special case of the more general complementarity principle.”

So what was this complementarity principle? Simply put, it was the observation that there are many truths about the world and many ways of seeing it. These truths might appear divergent or contradictory, but they are all equally essential in representing the true nature of reality; they are complementary. As Bohr famously put it, “The opposite of a big truth is also a big truth”. Complementarity provided a way to reconcile the paradoxes that seemed to bedevil quantum theory’s interpretation of reality.

The central scientific paradox was what is called wave-particle duality. In 1803, the British polymath Thomas Young had proposed that light, contrary to Isaac Newton’s view of it, consists of waves; an experiment like diffraction makes this wave nature clear. A hundred years later, in 1905, Einstein proposed that light in fact consists of particles, an idea he invoked in order to explain the photoelectric effect and which won him a Nobel Prize; these particles were later called photons. Soon it was found through other experiments that all subatomic particles and not just photons could display wave and particle behavior. In 1924, the French physicist and aristocrat Louis de Broglie saw a way through the impasse when he came up with a simple equation that related the momentum of a particle – a particle property – inversely to its wavelength – a wave property.

In spite of de Broglie’s insight, particles clearly don’t look like waves and waves don’t look like particles in real life. In fact the very names seem to put them at odds with one another. It was Bohr who saw both the problem and the solution. Particles and waves both exist and are equally valid and essential ways of interpreting the quantum world. Depending on what experiment you do you might see one or the other and never both, but they are not contradictory, they are complementary. Most crucially, you simply cannot make sense of reality without having both in hand. It was a powerful insight that cut through the complexities of intuition and language; it was not too different in principle from other counterintuitive truths that science has uncovered, for instance the truth that both lighter and heavier bodies fall at the same rate. Complementarity rationalized opposing tendencies of the physical world and indicated that they were one. It was what had made Bohr subsume the opposing quantities in Heisenberg’s uncertainty principle under the same rubric.

Complementarity was also pregnant with far more general interpretation. The most effective application of it to human affairs in Bohr’s hands was the problem posed by nuclear weapons. Even before the bomb had been used on Hiroshima, Bohr saw deeper and further than anyone else that the very fact that nuclear weapons are so enormously destructive might make them the most potent force for peace that the world has ever seen, simply because statesmen will realize that nobody can truly “win” a nuclear war if everyone has them. “We are in a completely new situation that cannot be resolved by war”, Bohr said. The complementarity of the bomb continues to keep the peace through deterrence.

Another noteworthy example was a speech delivered by Bohr in 1938 to the International Congress of Anthropological and Ethnological Sciences at Kronberg Castle in Denmark. Apologizing at the outset for presuming to speak about a topic on which he was not an expert, Bohr proceeded to provide a succinct summary of complementarity in the context of atomic physics. Turning to biology, he then made the perspicacious observation – still the subject of considerable debate – that reason and instinct which might appear to be opposed to each other are complementary, both providing a complete picture of a sentient being. Bohr then came to the crux of the matter, pointing out that complementarity is of the essence when people are judging other cultures which might seem divergent from their own but which turn out to be different and equally productive ways of looking at the world. As Bohr eloquently put it: “Each culture represents a harmonious balance of traditional conventions by means of which latent potentialities of human life can unfold themselves in a way which reveals to us new aspects of its unlimited richness and variety.” Bohr believed that contact between cultures can go a long way in not just dispelling biases but in mutually enriching both parties: “A more or less intimate contact between different human societies can lead to a gradual fusion of traditions, giving birth to a quite new culture”. This is as clear an appeal for internationalism and mutual understanding that one can think of; if everyone had understood complementarity, maybe we might have had less fascism, imperialism and genocide. The final goal of complementary views of societies, as Bohr pointed out powerfully in the same lecture, isn’t different from the goal of science as a whole – it is “the gradual removal of prejudices”.

As we approach what seem to be novel problems in the 21st century, Bohr’s complementarity is a message in a bottle from one fraught world to another, telling us that seeing these new problems through the lens of an old principle can be most rewarding. We seem to live in a time when many see social and political problems through a binary, black-or-white, zero sum lens. Either my viewpoint is right or yours, but not both. Complementarity bridges that division. For instance consider the problem of individualism vs communalism, a divide that also hints at the cultural divide Bohr spoke about, in this case largely an Eastern vs Western divide. Western society is fiercely individualistic; self-interests guide people’s lives and most people don’t want others to tell them that they should live their lives for others. Meanwhile, Eastern and some European societies are much more communal; community interests often override self-interest and individuals are told that their self-development should take a backseat to the development of their community and society. Bohr’s complementarity tells us that this divergent view should not exist. Communal and individualistic views are both essential for looking at the world and building a more productive society; in fact one can gain self-knowledge and wisdom by working for a community, and likewise a community can be improved when people engage in individualistic self-improvement that helps everyone.

There are other problems for which complementarity provides a potential solution. I will speak mostly for the United States since that’s where I live, but these problems are in fact global. Consider the problem of immigration, one to which Bohr’s 1938 address is directly applicable. People criticized as “globalists” think that unfettered immigration is a net good. The opposing camp thinks that preserving a nation’s culture is important, and too much or too rapid immigration will weaken this culture. But complementarity tells us that nationalism is in fact strengthened when immigrants work together for the common good of the country. At the same time, immigrants should put their country first and prioritize work that will strengthen their nation’s economy, military and social institutions. We are global citizens, but we are also shaped by evolution and culture to take care of our immediate own. The opposite of a big truth is also a big truth.

Even scientific debates like the nature (genes) vs nurture (environment) conundrum can benefit from complementary views. People criticized as biological “essentialists” believe that genes dictate a lot of an individual’s physical and psychological makeup while the opposing “nurture” camp believes that much of the effect of genes can be changed by the environment. But complementarity says that just like the joint wave-particle view of reality, individuals are whole and complete, and this wholeness arises from a combination of genes and environment. In that sense, how much of a person’s mental and physical constitution we can control by either manipulating their genes or their environment is almost irrelevant. What’s relevant is the basic understanding in the first place that both matter; even if both camps agree with this baseline, they would already be talking a lot more with each other.

A third application of complementarity to international affairs, one which stems directly from Bohr’s view of the complementarity of the bomb, is to the relationship between United States and China, a relationship which will likely be the single-most important geopolitical determinant of the 21st century. China clearly has an autocratic regime that is not likely to yield to demands for more democratic behavior, both internally and externally, anytime soon. This has led to many in the United States to regard China as an implacable foe, almost a second Soviet Union. An important consequence of this view has been to see almost every technological development in the two countries, from gene editing to artificial intelligence to new weapons, as a contest.

But irrespective of the moral wisdom of engaging in this contest, complementarity tells us that such contests are likely to lead to the mutual ruin of both China and the United States, and by extension the rest of the world; for the same reason that any arms race would hollow out countries’ coffers and ramp up the specter of mutual annihilation. The reason is simple: both computer code and the physics of nuclear weapons are products of the fundamental laws of science and technology discovered or invented by human minds. Both can be divined and implemented by any country with smart scientists and engineers, which basically means any developing or developed country. An arms race in AI between China and the United States, for instance, would be as futile and dangerous as was the nuclear arms race between the United States and the former Soviet Union. Both countries would be fooling themselves if they think they can write better computer code and keep it secret for a long time. In that sense the fact that computer code, just like nuclear fission, is essentially a discovery of the human mind poses inconvenient truths for both countries. Whether we like it or not, we need to realize the complementarity of artificial intelligence akin to how we grudgingly realized the complementarity of the bomb: we need to realize that AI is powerful, that it is dangerous, that its secrets cannot stay secret for very long, that there are no real defenses against it, that the very dangers of AI cry for peaceful solutions to AI, and therefore that mutual cooperation between China and the United States under an umbrella of an international organization like the United Nations would be the only solution to avoid mutual cyber-destruction. China might be autocratic, but it has self-interests and wouldn’t want to see its own ruin. In the end, harmony between the United States and China might not be forced by bridging the moral divide between the two countries’ social and political systems: it would be forced by the very laws of science and technology.

Without oversimplifying the issue, it’s clear to me that Bohr’s complementarity provides a mediating middle ground for almost any other social or political issue I can think of; it’s not so much that it offers a solution but that it will compel each side to see the importance of the other side’s argument in providing a complete view of reality that cannot be provided by either viewpoint by itself. Pro-life or pro-choice? One can respect both the life of an unborn child and the life of the mother; the two are complementary. Socialism or capitalism? One can certainly have a mixed market economy – of the kind found in Niels Bohr’s home country for instance – that would give us the benefits of both. Climate legislation or rapid economic growth? One can create jobs related to new climate technologies that will result in economic growth. Science or religion? They address complementary aspects of the world, Stephen Jay Gould’s “non-overlapping magisteria”.

If we accept the idea of complementarity, we are in essence accepting the validity of all ways of looking at the world, and not just one. This does not mean that all ways are equally right – we can’t accept the germ theory of disease and the “theory” of diseases as a punishment from God on equal terms – but it is precisely through placing them on a level playing field and letting them play out their logical flow that we can even know how much of which view is right. In addition, Bohr realized that the world is indeed gray, that even flawed visions may contain snatches of truth that should be acknowledged as potential building blocks in our view of reality. But ultimately, Bohr’s plea for complementarity was a plea for what he called an “open world”, an ideal that for him was the highest that the peoples of the world could aspire to, an ideal that arose naturally from the democratic republic of science. If we accept complementarity, we automatically become open to examining every single approach to a problem, every way of parsing reality. Most importantly, we become open to true, unfettered communication with our fellow human beings, a tentative but lasting step toward Bohr’s – and science’s – “gradual removal of prejudices”. That seems like an important message for today.

First published on 3 Quarks Daily.

Black holes and the curse of beauty: When revolutionary physicists turn conservative

This is my latest monthly column for 3 Quarks Daily.

On September 1, 1939, the leading journal of physics in the United States, Physical Review, carried two remarkable papers. One was by a young professor of physics at Princeton University named John Wheeler and his mentor Niels Bohr. The other was by a young postdoctoral fellow at the University of California, Berkeley, Hartland Snyder, and his mentor, a slightly older professor of physics named J. Robert Oppenheimer.

The first paper described the mechanism of nuclear fission. Fission had been discovered nine months earlier by a team of physicists and chemists working in Berlin and Stockholm who found that bombarding uranium with neutrons could lead to a chain reaction with a startling release of energy. The basic reasons for the large release of energy in the process came from Einstein's famous equation, E = mc², and were understood well. But a lot of questions remained: What was the general theory behind the process? Why did uranium split into two and not more fragments? Under what conditions would a uranium atom split? Would other elements also undergo fission?

Bohr and Wheeler answered many of these questions in their paper. Bohr had already come up with an enduring analogy for understanding the nucleus: that of a liquid drop that wobbles in all directions and is held together by surface tension until an external force that is violent enough tears it apart. But this is a classical view of the uranium nucleus. Niels Bohr had been a pioneer of quantum mechanics. From a quantum mechanical standpoint the uranium nucleus is both a particle and a wave represented as a wavefunction, a mathematical object whose manipulation allows us to calculate properties of the element. In their paper Wheeler and Bohr found that the uranium nucleus is almost perfectly poised on the cusp of classical and quantum mechanics, being described partly as a liquid drop and partly by a wavefunction. At twenty five pages the paper is a tour de force, and it paved the way for understanding many other features of fission that were critical to both peaceful and military uses of atomic energy.

The second paper, by Oppenheimer and Snyder, was not as long; only four pages. But these four pages were monumental in their importance because they described, for the first time in history, what we call black holes. The road to black holes had begun about ten years earlier when a young Indian physicist pondered the fate of white dwarfs on a long voyage by sea to England. At the ripe old age of nineteen, Subrahmanyan Chandrasekhar worked out that white dwarfs wouldn't be able to support themselves against gravity if their mass increased beyond a certain limit. A few years later in 1935, Chandrasekhar had a showdown with Arthur Eddington, one of the most famous astronomers in the world, who could not believe that nature could be so pathological as to permit gravitational collapse. Eddington was a previous revolutionary who had famously tested Einstein's theory of relativity and its prediction of starlight bending in 1919. By 1935 he had turned conservative.

Four years after the Chandrasekhar-Eddington confrontation, Oppenheimer became an instant revolutionary when he worked out the details of gravitational collapse all the way to their logical conclusion. In their short paper he and Snyder demonstrated that a star that has exhausted all its thermonuclear fuel cannot hold itself against its own gravity. When it undergoes gravitational collapse, it would present to the outside world a surface beyond which any falling object will appear to be in perpetual free fall. This surface is what we now call the event horizon; beyond the event horizon even light cannot escape, and time essentially stops flowing for an outside observer.

Curiously enough, the black hole paper by Oppenheimer and Snyder sank like a stone while the Wheeler-Bohr paper on fission gained wide publicity. In retrospect the reason seems clear. On the same day that both papers came out, Germany attacked Poland and started World War 2. The potential importance of fission as a source of violent and destructive energy had not gone unnoticed, and so the Wheeler-Bohr paper was of critical and ominous portent. In addition, the paper was in the field of nuclear physics which had been for a long time the most exciting field of physics. Oppenheimer's paper on the other hand was in general relativity. Einstein had invented general relativity more than twenty years earlier, but it was considered more mathematics than physics in the 1930s. Quantum mechanics and nuclear physics were considered the most promising fields for young physicists to make their mark in; relativity was a backwater.

What is more interesting than the fate of the papers themselves though is the fate of the three principal characters associated with them. In their fate as well as that of others, we can see the differences between revolutionaries and conservatives in physics.

Niels Bohr had pioneered quantum mechanics with his paper on atomic structure in 1913 and since then had been a founding father of the field. He had run an intellectual salon at his institute at Copenhagen which had attracted some of the most original physicists of the century; men like Werner Heisenberg, Wolfgang Pauli and George Gamow. By any definition Bohr had been a true revolutionary. But in his later life he turned conservative, at least in two respects. Firstly, he stubbornly clung to a philosophical interpretation of quantum mechanics called the Copenhagen Interpretation which placed the observer front and center. Bohr and his disciples rejected other approaches to quantum interpretation, including one named the Many Worlds Interpretation pioneered by John Wheeler's student Hugh Everett. Secondly, Bohr could not grasp the revolutionary take on quantum mechanics invented by Richard Feynman called the sum-over-histories approach. In this approach, instead of considering a single trajectory for a quantum particle, you consider all possible trajectories. In 1948, during a talk in front of other famous physicists in which Feynman tried to explain his theory, Bohr essentially hijacked the stage and scolded Feynman for ignoring basic physics principles while Feynman had to humiliatingly stand next to him. In both these cases Bohr was wrong, although the verdict is still out on the philosophical interpretation of quantum mechanics. It seems however that Bohr forgot one of his own maxims: "The opposite of a big truth is also a big truth". For some reason Bohr was unable to accept the opposites of his own big truths. The quantum revolutionary had become an old-fashioned conservative.

John Wheeler, meanwhile, went on to make not just one but two revolutionary contributions to physics. After pioneering nuclear fission theory with Bohr, Wheeler immersed himself in the backwater of general relativity and brought it into the limelight, becoming one of the world's foremost relativists. In the public consciousness, he will probably be most famous for coining the term "black hole". But Wheeler's contributions as an educator were even more important. Just like his own mentor Bohr, he established a school of physics at Princeton that produced some of the foremost physicists in the world; among them Richard Feynman, Kip Thorne and Jakob Bekenstein. Today Wheeler's scientific children and grandchildren occupy many of the major centers of relativity research around the world, and until the end of his long life that remained his proudest accomplishment. Wheeler was a perfect example of a scientist who stayed a revolutionary all his life, coming up with wild ideas and challenging the conventional wisdom.

What about the man who may not have coined the term "black holes" but who actually invented them in that troubled year of 1939? In many ways Oppenheimer's case is the most interesting one, because after publishing that paper he became completely disinterested in relativity and black holes, a conservative who did not think the field had anything new to offer. What is ironic about Oppenheimer is that his paper on black holes is his only contribution to relativity – he was always known for his work in nuclear physics and quantum mechanics after all – and yet today this very minor part of his career is considered to be his most important contribution to science. There are good reasons to believe that had he lived long enough to see the existence of black holes experimentally validated, he would have won a Nobel Prize.

And yet he was utterly oblivious to his creations. Several reasons may have accounted for Oppenheimer's lack of interest. Perhaps the most obvious reason is his leadership of the Manhattan Project and his fame as the father of the atomic bomb and a critical government advisor after the war. He also became the director of the rarefied Institute for Advanced Study and got saddled with administrative duties. It's worth noting that after the war, Oppenheimer co-authored only a single paper on physics, so his lack of research in relativity really reflects his lack of research in general. It's also true that particle physics became the most fashionable field of physics research after the war, and stayed that way for at least two decades. Oppenheimer himself served as a kind of spiritual guide to that field, leading three key postwar conferences that brought together the foremost physicists in the field and inaugurated a new era of research. But it's not that Oppenheimer simply didn't have the time to explore relativity; it's that he was utterly indifferent to developments in the field, including ones that Wheeler was pioneering at the time. The physicist Freeman Dyson recalls how he tried to draw out Oppenheimer and discuss black holes many times after the war, but Oppenheimer always changed the subject. He just did not think black holes or anything to do with them mattered.

In fact the real reason for Oppenheimer's abandonment of black holes is more profound. In his later years, he was afflicted by a disease which I call "fundamentalitis". As described by Dyson, fundamentalitis leads to a belief that only the most basic, fundamental research in physics matters. Only fundamental research should occupy the attention of the best scientists; other work is reserved for second-rate physicists and their graduate students. For Oppenheimer, quantum electrodynamics was fundamental, beta decay was fundamental, mesons were fundamental; black holes were applied physics, worthy of second-rate minds.

Oppenheimer was not the only physicist to be stricken by fundamentalitis. The malady was contagious and in fact had already infected the occupant of the office of the floor below Oppenheimer's – Albert Einstein. Einstein had become disillusioned with quantum mechanics ever since his famous debates with Bohr in the 1920s and his belief that God did not play dice. He continued to be a holdout against quantum mechanics; a sad, isolated, often mocked figure ignoring the field and working on his own misguided unification of relativity and electromagnetism. Oppenheimer himself said with no little degree of scorn that Einstein had turned into a lighthouse, not a beacon. But what is less appreciated is Einstein's complete lack of interest in black holes, which in some sense is even more puzzling considering that black holes are the culmination of his own theory. Einstein thought that black holes were a pathological example of his relativity, rather than a general phenomenon which might showcase deep mysteries of the universe. He also wrongly thought that the angular momentum of the particles in a purported black hole would stabilize its structure at some point; this thinking was very similar to Eddington's rejection of gravitational collapse, essentially based on faith that some law of physics would prevent it from happening.

Unfortunately Einstein was obsessed with the same fundamentalitis that Oppenheimer was, thinking that black holes were too applied while unified field theory was the only thing worth pursuing. Between them, Einstein and Oppenheimer managed to ignore the two most exciting developments in physics – black holes and quantum mechanics – of their lives until the end. Perhaps the biggest irony is that the same black holes that both of them scorned are now yielding some of the most exciting, and yes – fundamental – findings in cosmology, thermodynamics, information theory and computer science. The children are coming back to haunt the ghosts of their parents.

Einstein and Oppenheimer's fundamentalitis points to an even deeper quality of physics that has guided the work of physicists since time immemorial. That quality is beauty, especially mathematical beauty. Perhaps the foremost proponent of mathematical beauty in twentieth century physics was the austere Englishman Paul Dirac. Dirac said that an equation could not be true until it was beautiful, and he had a point. Some of the most important and universal equations in physics are beautiful by way of their concision and universal applicability. Think about E= mc², or Ludwig Boltzmann's equation relating entropy to disorder, S=klnW. Einstein's field equations of general relativity and Dirac's equation of the electron that marries special relativity with quantum mechanics are both prime examples of elegance and deep beauty. Keats famously said that "Beauty is truth and truth is beauty", and Dirac and Einstein seem to have taken his adage to heart.

And yet stories of Dirac and Einstein's quest for beauty are misleading. To begin with, both of them and particularly their disciples seem to have exaggerated the physicists' reliance on beauty as a measure of reality. Einstein may have become enamored of beauty in his later life, but when he developed relativity, he was heavily guided by experiment and stayed very close to the data. He was after all the pioneer of the thought experiment. As a patent clerk in the Swiss patent office at Bern, Einstein gained a deep appreciation for mechanical instrumentation and its power to reveal the secrets of nature. He worked with his friend Leo Szilard on that most practical of gadgets – a refrigerator. His later debates with Bohr on quantum mechanics often featured ingenious thought experiments with devices that he had mentally constructed. In fact Einstein's most profoundly emotional experience came not with a mathematical breakthrough but when he realized that his theory could explain deviations in the perihelion of Mercury, an unsolved problem for a century; this realization left him feeling that "something had snapped" inside him. Einstein's success thus did not arise as much from beauty as from good old-fashioned compliance with experiment. Beauty was a sort of secondary effect, serving as a post-facto rationalization for the correctness of the theory.

Unfortunately Einstein adopted a very different attitude in later years, trying to find a unified field theory that was beautiful rather than true. He started ignoring the experimental data that was being collected by particle physicists around him. We now know that Einstein's goal was fundamentally flawed since it did not include the theory of the strong nuclear force, a theory which took another thirty years to evolve and which could not have progressed without copious experimental data. You cannot come up with a complete theory, beautiful or otherwise, if you simply lack one of the key pieces. Einstein seems to have forgotten a central maxim of doing science, laid down by the sixteenth century natural philosopher Francis Bacon, one of the fathers of the scientific method: "All depends on keeping the eye steadily fixed upon the facts of nature and so receiving their images simply as they are. For God forbid that we should give out a dream of our own imagination for a pattern of the world". In his zeal to make physics beautiful, Einstein ignored the facts of nature and pursued the dreams of his once-awesome imagination.

Perhaps the biggest irony in the story of Einstein and black holes comes from the words of the man who started it all. In 1983, Subrahmanyan Chandrasekhar published a dense and authoritative tome called "The Mathematical Theory of Black Holes" which laid out the complete theory of this fascinating object in all its mathematical glory. In it Chandra (as he was called by his friends) had the following to say:

"In my entire scientific life, extending over forty-five years, the most shattering experience has been the realization that an exact solution of Einstein's equations of general relativity, discovered by the New Zealand mathematician, Roy Kerr, provides the absolutely exact representation of untold numbers of massive black holes that populate the universe. This shuddering before the beautiful, this incredible fact that a discovery motivated by a search after the beautiful in mathematics should find its exact replica in Nature, persuades me to say that beauty is that to which the human mind responds at its deepest and most profound."

Black holes and beauty had come full circle. Far from being a pathological outlier as believed by Einstein and Oppenheimer, they emerged as the epitome of austere mathematical and physical beauty in the cosmos.

Dirac seems to have been guided by beauty to an even greater extent than Einstein, but even there the historical record is ambiguous. When he developed the Dirac equation, he was very closely aware of the experimental results. His biographer Graham Farmelo notes, "Dirac tried one equation after another, discarding each one as soon as it failed to conform to his theoretical principles or to the experimental facts". Beauty may have been a criterion in Dirac's choices, but it was more a way of serving as an additional check rather than a driving force. Unfortunately Dirac did not see it that way. When Richard Feynman and others developed the theory of quantum electrodynamics – a framework that accounts for almost all of physics and chemistry except general relativity - Dirac was completely unenthusiastic about it. This was in spite of quantum electrodynamics agreeing with experiment to a degree unprecedented in the history of physics. When asked why he still had a problem with it, Dirac said it was because the equations were too ugly; he was presumably referring to a procedure called renormalization that got rid of infinities that had plagued the theory for years.

He continued to believe until the end that those ugly equations would somehow metamorphose into beautiful ones; the fact that they worked spectacularly was of secondary importance to him. In that sense beauty and utility were opposed in Dirac's mind. Dirac continued to look for beauty in his equations throughout his life, and this likely kept him from making any contribution that was remotely as important as the Dirac equation. That's a high bar, of course, but it does speak to the failure of beauty as a primary criterion for scientific discovery. Later in his life, Dirac developed a theory of magnetic monopoles and dabbled in finding formulas relating the fundamental constants of nature to each other; to some this was little more than aesthetic numerology. Neither of these ideas has become part of the mainstream of physics.

It was the quest for beauty and the conviction that fundamental ideas were the only ones worth pursuing that turned Einstein and Dirac from young revolutionaries to old conservatives. It also led them to ignore most of the solid progress in physics that was being made around them. The same two people who had let experimental facts serve as the core of their decision making during their youth now behaved as if both experiment and the accompanying theory did not matter.

Yet there is something to be said for making beauty your muse, and ironically this realization comes from the history of the Dirac equation itself. Perhaps the crowning achievement of that equation was to predict the existence of positively charged electrons or positrons. This discovery seemed so alien and unsettled Dirac so much at the beginning that he thought positrons had to be protons; it wasn't until Oppenheimer showed this could not be the case that Dirac started taking the novel prediction seriously. Positrons were finally found by Carl Anderson in 1932, a full three years after Dirac's prediction. This is one of the very few times in history that theory has genuinely predicted a completely novel fact of nature with no experimental basis in the past. Dirac would claim that it was the tightly knit elegance of his equation that logically ordained the existence of positrons, and one would be hard pressed to argue with him. Even today, when experimental evidence is lacking or absent, one has to admit that mathematical beauty is as good a guide to the truth as any other.

Modern theoretical physics has come a long way from the Dirac equation, and experimental evidence and beauty still guide practitioners of the field. Unfortunately physics at the frontiers seems to be unmoored from both these criteria today. The prime example of this is string theory. According to physicist Peter Woit and others, string theory has made no unique, experimentally testable prediction since its inception thirty years ago, and it also seems that its mathematics is unwieldy; while the equations seem to avoid the infinities that Dirac disliked, they also presents no unique, elegant, tightly knit mathematical structure along the lines of the Dirac equation. One wonders what Dirac would have thought of it.

What can today's revolutionaries do to make sure they don't turn conservative in their later years? The answer might come not from a physicist but from a biologist. Charles Darwin, when explaining evolution by natural selection, pointed out a profoundly important fact: "It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is most adaptable to change". The principle applies to frogs and butterflies and pandas, and there is no reason why it should not apply to theoretical physicists.

What would it take for the next Dirac or Einstein to make a contribution to physics that equals those of Einstein and Dirac themselves? We do not know the answer, but one lesson that the lives of both these physicists has taught us – through their successes as well as their failures – is to have a flexible mind, to always stay close to the experimental results and most importantly, to be mindful of mathematical beauty while not making it the sole or even dominant criterion to guide your thought processes, especially when an "uglier" theory seems to agree well with experiment. Keep your eye fixed on the facts of nature, not just on the dream of your imagination.