The birth of a new theory: Richard Feynman and his adversaries

Leading physicists discuss their field's most pressing problems
at Shelter Island in April 1947; including, among others, Julian
Schwinger, Richard Feynman and J. Robert Oppenheimer

This post was written on occasion of Richard Feynman's 100th birthday on May 11th and was first published on the website 3 Quarks Daily. It's the second in a pair of articles about two landmark meetings in postwar American physics.

A new theory seldom comes into the world like a fully formed, beautiful infant, ready to be coddled and embraced by its parents, grandparents and relatives. Rather, most new theories make their mark kicking and screaming while their fathers and grandfathers try to disown, ignore or sometimes even hurt them before accepting them as equivalent to their own creations. Ranging from Darwin’s theory of evolution by natural selection to Wegener’s theory of continental drift, new ideas in science have faced scientific, political and religious resistance. There are few better examples of this jagged, haphazard, bruised birth of a new theory as the scientific renaissance that burst forth in a mountain resort during the spring of 1948.

April 2, 1948. Twenty-eight of the country’s top physicists met at the Pocono Manor Hotel near the Delaware Water Gap in Pennsylvania. Kept apart from their first love of fundamental research in physics by the war, they were eager to regroup and rethink the problems which had plagued the heights of their profession before they were called away for war duty to Los Alamos, Cambridge and Chicago.

The listing of participants provides a rare snapshot of one of those hallowed transitions in the history of science, a passing of the torch. Both the old and the new guards were there. The old guard was represented, among others, by Niels Bohr, Paul Dirac and Eugene Wigner – the men who had formulated and then shaped the material world in its quantum mechanical image during the 1920s and 30s. The new guard was represented by Richard Feynman, John Wheeler and Julian Schwinger – the swashbuckling young theorists who wanted to take quantum theory to new heights, even if it meant challenging the old wisdom. J. Robert Oppenheimer who led the conference represented a prophet of the middle ground; a guide joining old hands with new. In retrospect, a clash of worldviews seems almost inevitable.

The problem that was at the forefront of everyone’s attention was the plague of infinities. The infinities had started showing up just a few years after the quantum revolution had burst upon the world. Within a short span of five years or so, between 1925 and 1930, a handful of theorists in their twenties and thirties including Dirac, Werner Heisenberg, Erwin Schrödinger, Max Born and Wolfgang Pauli had completely reworked the foundations of our physical picture of the world. Niels Bohr who along with Albert Einstein and Max Born had kicked off the revolution a decade before was their avuncular godfather; Einstein himself was a reluctant pioneer. The father of quantum mechanics, Max Planck, had showed that energy came only in discrete packets; now these new frontiersmen extended the concept to every physical entity in the universe. The work done by the quantum pioneers revealed a world steeped in probabilities rather than certainties, a world where you could not know the values of even simple parameters like a particle’s position and momentum to infinite precision, a world where particles and waves blurred themselves into each other in a mirage of probability amplitudes and wavefunctions. It was this fundamental ambiguity about what you could know about subatomic entities that led to Einstein’s famous remark about God playing dice.

And yet the theory was accuracy exemplified. Whatever its mathematical and philosophical ambiguities, it kept on providing astonishingly accurate answers to both old and new problems in disparate branches of physics. Whether it was a matter of calculating the frequency of radiation emitted by electrons transitioning in an atom or the resistance of metals to electrical current, quantum theory gave you the right answers, marching in perfect lockstep with numbers from experiment. It seemed to work for virtually every problem you threw it at. Except one.

That puzzle was the interaction between light and matter. It turned up in the simplest of situations, such as calculating the energy of an electron in an atom, and was recognized by the father of quantum field theory, Paul Dirac. Quantum field theory is the most comprehensive description of the world of subatomic particles, and in its simplest sense involves subjecting both particles and the electromagnetic fields which surround them to the rules of quantization. But even a cursory glimpse at the issue made the intractability clear: the energy of a charge in an electromagnetic field – called the ‘self energy’ – is given by the ratio of the strength of the field at various points and the distances between the charge and these points. One calculates the total self-energy by summing up the values at every point. The difficulty is obvious when you think about it: at distances close to zero, you divide by an increasingly smaller number, blowing up the value precipitously. Exactly at the location of the charge, where the distance is zero, the energy becomes infinite; the writers Robert Crease and Charles Mann have described it as a plane being blown to smithereens in its own wake. Clearly this is an absurd result since every energy value which you measure in a real world laboratory is finite.

Starting in about 1930, this problem of infinite self-energy was tackled by many of the most brilliant theoreticians of their time without resolution: Oppenheimer, Heisenberg, the acerbic Wolfgang Pauli and his mild-mannered assistant Victor Weisskopf all described the problem and tried to resolve it in various ways. If anything the picture got even worse; instead of just one infinity, other infinities started rearing their ugly heads like the heads of the mythical Hydra. One of these infinities was pointed out by Dirac. It turns out that during its transition in an atom, an electron can briefly spit out a photon and reabsorb it; this seemingly ex nihilo act of creation is allowed by quantum mechanics as long as it’s done in an exceedingly small amount of time. The problem is that the energy between the electron and this so-called ‘virtual’ photon can be apportioned again in an infinite numbers of ways. If you sum up all these ways you again get the dreaded explosion of infinity.

There seemed to be no end to attempts to exorcising these infinities. Then war intervened, the community of American physicists was drawn up for work on radar and the bomb, and the community of European physicists, many of whom had already fled from Hitler and Mussolini and were scattered across at least two continents, joined them. There the matter of the infinities rested until 1947, when an extraordinary conference of physicists was organized at a small inn off the coast of Long Island near New York City. The Shelter Island conference later went down in history as the conference that kicked off the postwar rejuvenation of particle physics, but in April 1947 it still represented the first stirrings of a revolution. The conference was again chaired by Oppenheimer and included a mix of the old and young guards.

The attention of the participants at Shelter Island was focused on one number of singular importance. Sometimes it takes hard experiment to cut the theoretical Gordian knot. While the theorists had struggled with infinities even before the war, they were galvanized by the experiments of Willis Lamb and his colleague Robert Retherford. Lamb was one of those rare breeds of scientist who are comfortable with both theory and experiment. Combining highly skilled techniques in microwave spectroscopy developed during wartime work on radar with a good understanding of the problems with infinities plaguing quantum field theory, Lamb and Retherford discovered a slight difference in energy between two states of the hydrogen atom at a place where the original Dirac theory predicted no difference. In science revolutions are sometimes engineered by the slightest and most mundane-looking discrepancies in the behavior of matter – Arthur Eddington’s measurement of a tiny shift in the position of the stars predicted by Einstein’s general theory of relativity comes to mind – and the Lamb Shift is as good an example as any of this pivot point in scientific history.

The Lamb Shift is also a telling example of what happens when multiple ideas are in the air, vying with each other for publicity and survival. In the conference Weisskopf had already presented a calculation that could potentially explain the shift, and so had one of the members of the old guard, Hendrik Kramers, who had been Niels Bohr’s assistant. But neither of these efforts got rid of the infinities. It took Hans Bethe with his absolutely mastery of synthesizing different ideas to take the Lamb Shift to its logical conclusion. Nobody surpassed Bethe in his knowledge of multiple branches of physics and his ability to calculate real world answers using the right combination of mathematical techniques and approximations. During a train journey back from Shelter Island, Bethe had the stroke of insight to attempt a calculation of the Lamb Shift using a non-relativistic approximation that ignored effects due to Einstein’s special theory of relativity. In addition, he introduced a physically sensible cutoff for the infinities to get a finite answer. Everyone knew that a correct quantum field would have to include special relativity, so it took some courage on Bethe’s part to attempt a non-relativistic calculation. Strikingly, the result was very close to experiment; 1040 MHz vs 1000 MHz. It still wasn’t the exact answer, but Bethe’s calculation was a shot in the arm, a signal that the theorists’ thinking was on the right track. It was also a fine illustration of how sometimes even a strictly non-realistic, approximate model can guide you in the right direction.

Bethe’s work breathed new life into the work of many others, including Weisskopf and Lamb, both of whom kicked themselves for not thinking about it first. But the biggest impact was on two young members of the group who had already distinguished themselves by their brilliant work during the war – Julian Schwinger and Richard Feynman.

Feynman and Schwinger were two of the earliest products of the American school of theoretical physics. Until the 1930s or so, most American theorists had to go to Europe to learn quantum mechanics at the feet of the masters: Niels Bohr in Copenhagen, Max Born in Göttingen and Arnold Sommerfeld in Munich. In the 30s the center of research started moving to the United States, partly engendered by the exodus of Jewish refugee physicists and partly because of the creation of prominent schools of physics by American physicists themselves. Two of the most prominent schools were Robert Oppenheimer’s at Berkeley and John Archibald Wheeler’s at Princeton. Schwinger came from Oppenheimer’s school; Feynman came from Wheeler’s.

Both Schwinger and Feynman were from New York, but otherwise were very different characters. Feynman was a practical joker who cracked safes, disdained pretension, played the bongos and spoke in colloquial New York City slang. While he had been recognized as a brilliant physicist, he still did not enjoy the star power that Schwinger – a child prodigy who had written his first paper on quantum electrodynamics when he was sixteen – did. Unlike Feynman, the leonine Schwinger wore expensive suits, drove a Cadillac and was the very picture of the distinguished academic. Physicists of the stature of Bethe and Fermi had already paid homage to Schwinger and everyone thought him to be the future. At Shelter Island they had listened to him with reverence; as Oppenheimer put it, “When other physicists do a calculation they want to tell you how they do it; when Schwinger does a calculation he wants to tell you that only he can do it.”

After Shelter Island, the physicists went off to their universities and laboratories, attempting a full calculation of quantum electrodynamics that was relativistic. Schwinger managed to calculate a precise value for the magnetic moment of the electron – a parameter for which comparison between theory and experiment would come to represent the most accurate agreement in all of physics – for the first time in November 1947. Most importantly, the calculation gave a finite answer. One of the elder statesmen of physics, a tough-minded New Yorker named Isidor Rabi, rushed off a note to Bethe: “Schwinger’s calculation is as accurate as yours. God is Great!”.

Feynman was on a completely different track. Working with John Wheeler, he had come up with a novel approach called the path integral approach that included particles traveling backward and forward in time. His bookkeeping technique used a principle familiar from classical mechanics, the principle of least action, that minimized the energy a particle takes in order to travel from A to B. For quantum theory, in Feynman’s hands, one had to consider every single trajectory that the particle would take in order to calculate the most probable one. This so-called 'sum over histories' approach was completely different from anyone else's, although its first trappings had been anticipated by the always prescient Dirac in a paper which Feynman had eagerly read in the Princeton library as a graduate student. Feynman represented his calculations in the form of squiggly and straight lines symbolizing virtual and real particles traveling backward and forward in time. When the Pocono Conference rolled around, he was ready to dazzle his listeners.

Unfortunately Feynman was up against two major obstacles. One was the traditional and hidebound old physics establishment. The other was Julian Schwinger. Schwinger had just given a marathon six-hour talk in which he brought all the formal machinery of mathematical physics to bear on calculating finite answers for electron-photon interactions. His talk was described by some as a virtuoso violin performance, more technique than comprehension. By one account, only Hans Bethe and Enrico Fermi – men who were particularly known for their stamina and powers of concentration – stayed awake and alert enough to follow the entire presentation.

Then Feynman took the podium. Knowing that his listeners would have trouble following the novel derivation of his results, he instead proceeded to simply show them worked out examples. His strategy was understandable, but he was attempting something akin to simply showing worked out examples in a mathematics textbook without showing the underlying theory. For the mandarins of theory who had spent their entire careers trying to take apart and understand all the gory details of how nature worked, this impressionistic-looking display was most unsatisfactory. Immediately they interrupted.

Edward Teller, the Hungarian-born physicist who hadn’t yet achieved the infamous moniker of ‘father of the hydrogen bomb’, thought that Feynman was violating the exclusion principle, a central tenet of physics and chemistry discovered by Wolfgang Pauli that precludes having two electrons with the same energy and spins in the same state. Dirac asked Feynman about a mathematical matrix that carried particle probabilities forward in time. He was wondering about a recondite mathematical property of a matrix called the unitary property that had nonetheless been key in understanding all particle interactions in quantum mechanics. Finally, the elder statesman of physics, the father of them all, Niels Bohr interrupted. Bohr had been impressed at Los Alamos by Feynman’s willingness to brazenly question all authority, including Bohr’s. He now took umbrage at the unfamiliar thicket of squiggles representing particle trajectories. Already in the 1930s, Bohr said in his soft but firm voice, we knew that the classical notion of a trajectory does not make sense in quantum mechanics. Now Feynman seemed to be violating this basic tenet of quantum theory. Bohr strode up to the stage and, standing next to Feynman, speaking in his notorious mumble, delivered a humiliating lecture that seemed to convey Feynman’s lack of understanding of even elementary ideas.

Feynman realized that it was hopeless; Teller was obsessed with a basic fact of quantum mechanics, Dirac was hung up over mathematical formalism, Bohr was still stuck in the 1930s. Clearly his approach was too unconventional and too novel for the old guard. The only way they would listen would be if he laid it all out in an academic paper. History was witnessing the passing of the torch between generations, but for the time being it would have to allow the old generation to win the battle, even if they lost the war. Feynman was undoubtedly on the right track. His new theory had given the right answers for all outstanding problems posed by the new physics. And after his talk, in the next few days, he compared his results with Schwinger’s. These two rivals nonetheless had a healthy respect for each other’s unique approaches, and they realized that were both traversing different trajectories on the mountain of truth.

Within a year Feynman had written up a seminal paper spurred by the disappointment and urgency he witnessed at Pocono. “Space-Time Approach to Quantum Mechanics” would become one of the most important physics papers of the twentieth century. In time, Feynman diagrams would come to dot the pages of the leading physics journals like an art form, much like the native art found on the caves at Lascaux represented its creator’s innermost desires and motivations. And like God bringing, in Schwinger’s words, “computation to the masses”, Feynman would have his own prophet: his colleague Freeman Dyson would unify Schwinger and Feynman’s versions of the promised land and deliver a set of powerful tools that would allow physicists to apply the duo’s techniques to problems in fields ranging from particle physics to astrophysics. And finally, like a voice from the deep, a lonesome letter would come floating to America from the troubled East, where a physicist named Sin-Itiro Tomonaga would have astonishingly worked out Schwinger’s formulation of QED in the isolation and destruction of wartime Japan. In time, QED would provide the most astonishingly accurate between theory and experiment in the history of physics.

But it had all started at Shelter Island and Pocono, where history changed hands and took a new direction, where a thirty year old physicist presented a novel vision; in Dyson’s words, “this wonderful vision of the world as a woven texture of world lines in space and time, with everything moving freely, a unifying principle that would either explain everything or explain nothing.”