Field of Science

The fundamental philosophical dilemma of chemistry

The classic potential energy curve of chemistry
hides a fundamental truth: bonds mean short distances,
but short distances don't mean bonds
Every field has its set of great philosophical dilemmas. For physics it may be the origin of the fundamental constants of nature, for biology it might be the generation of complexity by random processes. Just like physics and biology chemistry operates on both grand and local scales, but the scope of its fundamental philosophical dilemmas sometimes manifests itself in the simplest of observations.

For me the greatest philosophical dilemma in chemistry is the following: It is the near impossibility of doing controlled experiments on the molecular level. Other fields also suffer from this problem, but I am constantly struck by how directly one encounters it in chemistry.

Let me provide some background here. Much of chemistry is about understanding the fundamental forces that operate within and between molecules. These forces come in different flavors: strong covalent bonds (dictated by the sharing of electrons), hydrogen bonds (dictated by weak electrostatic interactions), strong charge-charge interactions (dictated by attraction between unlike charges), hydrophobic effects (dictated by the interaction between 'water-loving' and 'water-hating' parts of molecules) etc. The net interaction or repulsion between two molecules results from the sum total of these forces, some of which may be attractive and others might be repulsive. Harness these forces and you can control the structure, function and properties of molecules ranging from those used for solar capture to those used as breakthrough anticancer drugs.

Here’s how the fundamental dilemma manifests itself in the control of all these interactions: it is next to impossible to perform controlled experiments that would allow one to methodically vary one of the interactions and see its effect on the overall behavior of the molecule. In a nutshell, the interactions are all correlated, sometimes intimately so, and it can be impossible to change one without changing the other.

The fundamental dilemma is evident in many simple applications of chemistry. For instance, my day job involves looking at the crystal structures of proteins involved in disease and then designing small organic molecules which bind to and block such proteins. For binding to their target protein, these small molecules exploit many different interactions including hydrogen bonds, charge-charge interactions and hydrophobic effects to bring about a net lowering of their interaction energy with the protein. The lower this interaction or "free energy" the better the interaction. Unfortunately, while one can visualize the geometry of the various interactions by simply looking at the crystal structure, it is very difficult to say anything about their energies, for to do so would entail varying an interaction individually and looking at its effects on the net energy. Crystal structures thus can be very misleading when it comes to making a statement about how tightly a small molecule binds to a protein.

Let’s say I am interested in knowing how important a particular hydrogen bond in the small molecule is. What I could do would be to replace the atoms comprising the hydrogen bond with non hydrogen-bonding atoms and then look at the change in the affinity of the resulting molecule for the protein, either computationally or experimentally. Unfortunately this change also impacts other properties of the molecules; its molecular weight, its hydrophobicity, its steric or spatial interactions with other molecules. Thus, changing a hydrogen bonding interaction also changes other interactions, so how can we then be sure that any change in the binding affinity came only from the loss of the hydrogen bond? The matter gets worse when we realize that we can’t even do this experimentally; in my colleague Peter Kenny’s words, an individual interaction between molecules such as a hydrogen bond is not really an experimental observable. What you see in an experiment is only the sum total, not the dissection into individual parts.

There have of course been studies on ‘model systems’ where the number of working parts is far less than those in protein-bound small molecules, and from these model systems we have gotten a good sense of the energies of typical hydrogen bonds, but how reliably can we extend the results of these systems to the particular complex system that we are studying? Some of that extrapolation has to be a matter of faith. Also, model systems usually provide a ranges of energies rather than a single value and we know that even a tiny change in the energy of binding can correspond to a substantial loss of effective blocking of a protein, so the margin of error entrusted to us is slim indeed.

It is therefore very hard, if not impossible, to pin down a change in binding affinity resulting from a single kind of interaction with any certainty, because changing a single interaction potentially changes all interactions; it is impossible to perform the truly controlled experiment, a concept which has been at the heart of the scientific method. Sometimes these changes in other interactions can be tiny and we may get lucky, but the tragedy is that we can’t even calculate with the kind of accuracy we would like, what these tiny increments or reductions might be. The total perturbation of a molecule’s various interactions remains a known unknown.

The roots of the problem run even deeper. At the most elemental level, all interactions between molecules are simply a function of one of the four fundamental forces known in nature - the electromagnetic force. Of the four basic forces, gravity is too weak to play a role, while the strong and weak nuclear forces don't usually apply to molecular interactions since such interactions only involve the sharing of electrons. It is the electromagnetic force that is thus ascendant in mediating every single molecular interaction in the universe. When we divide this force up into hydrogen bonds, electrostatic interactions, hydrophobic interactions etc. what we are doing is imposing an artificial division on an indivisible fundamental force, purely for our convenience. It's a bit like the parable of the blind men and the elephant - there is only one electromagnetic force, just like there is only one elephant, but each of us describing that force divides it up into multiple flavors. No wonder then that we are led astray when we think we are doing a controlled experiment, since whenever we think we are varying one flavor or another we are actually varying the same basic parameter and not its independent components. That is because there are no independent components in the true sense of the term.

This inability to perform the truly controlled experiment is thus what I call the great philosophical dilemma of chemistry. The dilemma not only makes the practical estimation of individual interactions very hard but it leads to something even more damning: the ability to even call an interaction an 'interaction' or 'bond' in the first place. This point was recently driven home to me through an essay penned by one of the grand old men of chemistry and crystallography – Jack Dunitz. Dunitz’s point in the essay is that we are often misled by ‘short’ distances between atoms observed in crystal structures. We ascribe these distances to ‘attractive interactions’ and even ‘bonds’ when there is little evidence that these distances are actually attractive.

Let’s backtrack a bit to fundamentals. The idea of ascribing a short distance to an attractive interaction comes from the classic van der Waals potential energy curve (figure above) that is familiar to anyone who has taken a college chemistry class. The minimum of this curve corresponds to both the shortest distance (called the van der Waals distance) between two molecules and the lowest energy, typically taken to signify a bond. However this leads to a false equivalence that seems to flow both ways: van der Waals distances correspond to bonds and bonds correspond to van der Waals distances.

In reality the connection only flows one way. Bonds do correspond to short distances but short distances do not necessarily correspond to bonds. So then why do we observe short distances in molecules in the first place? Again, Dunitz said it very succinctly in a previous review: simply because ‘Atoms have to go somewhere’. The fact is that a crystal structure is the net result of a complex symphony of attractive and repulsive interactions, a game of energetic musical chairs if you will. At the end, when the dust has settled everyone has to find a chair, even if it means that two people might end up uncomfortably seated on the same chair. Thus, when you see a short distance between two atoms in a crystal, it does not mean at all that the interaction between them is attractive. It could simply mean that other interactions between other atoms are attractive and that those two atoms have simply then settled where they find a place, even if the interaction between them may be repulsive. 

The message here is clear: it is folly to describe an interaction as ‘attractive’ simply because the distance is short. This applies especially to weaker interactions like those between aromatic (benzene) rings. I am always wary when I see a benzene ring from a small molecule nicely sandwiched between another benzene ring in a protein and hear the short distance between the two described as a ‘stacking interaction’. Does that mean there is actually an attractive stacking interaction between the two? Perhaps, but maybe it means simply that there was no other place for the benzene ring to be. How could I test my hypothesis? Well, I know that varying the substituents or groups of atoms attached to benzene rings is known to vary their energies of interaction with other benzene rings. So I ask the chemist to make some substituted versions of that benzene ring. But hold on! Based on the previous discussion, I just remembered that varying the substituents is not going to just change the stacking energy; it’s also going to change other qualities of the ring that mess up the other interactions in the system. It’s that problem with performing controlled experiments all over again - welcome to the fundamental dilemma of chemistry.

The fundamental dilemma is why it is so hard to understand individual interactions in chemical systems, let alone exploit them for scientific or commercial gain. We see it in a myriad of chemical experiments, from investigating the effects of structural changes on the rates of simple chemical reactions to investigating the effects of structural changes on the metabolism of a drug. We can’t change one component without changing every other. There may be cases where these other changes might be minuscule, but in reality the belief that they may be minuscule in a particular case will always remain a matter of faith than of fact.

The fundamental dilemma then is why drug design, materials designs and every other kind of molecular design in chemistry is so tricky. It is why so much of complicated chemistry is still trial and error, why observations on one system cannot be easily extrapolated to another, and why even supercomputers are not yet able to nail down the precise balance of forces that dictate the structure and function of specific molecules. In a nutshell, the fundamental dilemma is why chemists are always ignorant and why chemistry will therefore always be endlessly fascinating.


  1. "bonds mean short distances,but short distances don't mean bonds" This reminded me of Cotton's words on metal-metal bonding. Let me quote from the book Multiple Bonds Between Metal Atoms:

    “Bond lengths are facts; bond orders are not. In the process of inferring a bond order from a bond length one is often getting out no more than what was put to begin with. …Experience is the only test, and experience thus far has shown that M-M bonds cannot usefully be treated in such a way. ..We condemn as foolish and hopeless any effort to associate a unique, precise, quantitative bond order with each and every M-M internuclear distance.”

    F. A. Cotton, C. A. Murillo and R. A. Walton, Multiple Bonds Between Metal Atoms, Springer Science and Business Media. Inc., New York, 2005, pp 707

  2. Hi Ash, There is also a more subtle issue here and I’ll use your example of trying to assess the contribution of a hydrogen bond. Suppose we could magically switch the hydrogen bonding off by turning the amide into something that looked like hydrocarbon? The problem is that the ‘hydrocarbonized’ amide still interacts with the environments in which it exists. We can use matched molecular pair analysis to show that one hydrogen bond contributes more (relative to a given reference state) than does another hydrogen bond acceptor. However, we can’t assign absolute contributions to the two hydrogen bonds.

    As you’ve correctly noted, the contribution of an intermolecular contact to affinity is not an experimental observable. Molecular recognition in aqueous media is controlled by non-local interactions. Some components of binding free energy (e.g. entropy changes, conformational energy penalties) are fundamentally non-local and it would not be possible to assign them to specific intermolecular contacts even in gas phase.

  3. Good point Pete: You can't truly turn off an interaction and keep the effect local. I suppose you could do it computationally to some extent (and people do do things like turning off electrostatics or vdW interactions) but as we all know that comes with its own litany of problems...

  4. Thanks for writing this so clearly and entertainingly. I'm a math teacher now (and saw a link to this from @MrHonner on Twitter, another math teacher), but worked for in the 1990s (I was employee #5, I see they're now at 300+), so I chuckled at "I suppose you could do it computationally... but as we all know that comes with its own litany of problems." Yes, as the grant writer/customer support person, I remember trying to talk our way through some of those.

    1. Thanks for commenting, and great to know that you were one of the original 'pioneers' at Schrodinger! I bet you know the travails of doing something like this as well as anyone...

  5. Ash, I think you might be needlessly giving chemistry as a field the short stick with this one. In molecular and cellular biology we consider quality knock-outs to be single variable changes, but in reality they are not even in absence of off-target effects. We pretend they change one variable (the target of interest) but in reality we know they change the system. I suppose those fall into the chemical systems you cover in the last paragraph.

    1. That's a good point. The lack of controlled experiments is a problem with biology too.

  6. I am going to paraphrase this a bit and you tell me if I am completely off.

    The issue is that you are dealing with a multi-particle system and that you cannot remove a piece of the puzzle and expect the landscape to change just locally. The impact of any changes is to change the global landscape and so it is very hard to build an overall change in the global landscape through minute changes in each local phase space changes.

    Kind of like how in GR you can build a full picture of a physical interaction by integrating over many many different inertial frames each of which is sort of a small change in the grand scheme of things, but adds up to the global one (however complicated and horrific this integral might be, there is at least in theory a path). However no such thing exists in Chemistry?

  7. Hi I enjoyed and (amazingly!) understood this post. There is a lucid beauty in looking at things as they are and appreciating the knowns and the unknowns. Posts like this can make daunting science less so. Congrats on the prize.

  8. Excellent article, but things are clearly not that hopeless. We can use these concepts to predict reactivity and reaction mechanisms, and do complicated organic syntheses. When considering drug protein interactions, it is impossible to systematically vary every amino acid in the protein -- there aren't enough atoms in the universe. Even doing this for 10 or so surrounding the target for a ligand involves 20^10 power. We are just beginning to find out how subtle and far reaching allosteric effects can be. A fascinating study of everything changing, when one very small ligand moves in a fairly simple protein is -- [ Science vol. 350 pp. 381, 445 - 450 '15 ] when CO is photo-dissociates from heme -- everything starts moving in under 1 picosecond.

  9. Could one, theoretically, come up with a program that takes all interactions (and all known information) into consideration and run the experiment through that on a "smaller" scale? I have seen some that do this for physics labs. Is it possible with Chemistry? I think that making a program like this would be a really cool project. It would be a lot of work to input all the data but the outcome would be a pretty awesome virtual chem lab. You could change scale and slow down or speed up reactions. Chem is not my strong suit, so excuse me if this is an ignorant question.


Markup Key:
- <b>bold</b> = bold
- <i>italic</i> = italic
- <a href="">FoS</a> = FoS