Protein-ligand crystal structures: WYSI(N)WYG

Crystal structures of proteins bound to small molecules have become ubiquitous in drug discovery. These structures are routinely used for docking, homology modeling and lead optimization. Yet as several authors have shown over the years, many of these structures can hide flaws that don't become apparent until they are actually revealed through analysis. Worse still, there may be flaws that never become apparent because nobody takes the trouble to look at the original data.

A recent paper from the group at OpenEye has a pretty useful analysis of flaws in crystal structures. They carry on a tradition most prominently exemplified by Gerard Kleywegt at Uppsala. The authors describe common metrics used for picking crystal structures from the PDB and demonstrate their limitations. They propose new metrics to remedy the problems with these parameters. And they use these metrics to analyze about 700 crystal structures used in structure-based drug design and software validation and find out that only about 120 structures or so pass the rigorous criteria used for distinguishing good structures from bad ones.

The most important general message in the article is about the difference between accuracy and completeness of data, and the caveat that any structure on a computer screen is a model and not reality. Even very accurate looking data may be incomplete and this flaw is often neglected by modelers and medicinal chemists when picking crystal structures. For instance, the resolution of a protein structure is often used as a criterion for selecting one among many structures of the same protein from the PDB. Yet the resolution only tells you how far apart atoms can be distinguished from each other. It does not tell you if the data is complete to begin with. So for instance, a crystallographer can acquire only 80% of the theoretically maximum possible data and present it with a resolution of 1.5 A, in which case the structure is clearly incomplete and possibly flawed for use in structure-based drug design. Another important metric is the R-free factor which is obtained by omitting certain parts of the data and refitting the rest to the model. A difference between R-free and the R-factor (a factor denoting the original difference between the full set of data and the model) of more than 0.45 for a structure with resolution 3.5 A or more is a red flag.

The OpenEye authors instead talk about a variety of measures that provide much better information about the fidelity of the data than resolution. The true measure of the data is of course the actual electron density. Any structure that is seen on the computer screen results from the fitting of a model to this electron density. While proteins are often fit fairly well to the density, the placement of ligands is often more ambiguous, partly because protein crystallographers are not always interested in the small molecules. The paper documents several examples of electron density that was either not fit or incorrectly fit by ligand atoms. In some cases sparse or even non-existent density was fit by guesswork, as in the illustration above. All these badly fit atoms show up in the final structure but only an expert would know this. The only way to overcome these problems is to take a look at the original electron density, which is unfortunately a task that most medicinal chemists and modelers are ill equipped for.

The worst problem with these crystal structures is that because they look so accurate and nice on a computer screen, they may fall into Donald Rumsfeld's category of "unknown unknowns". But even with "known unknowns" the picture is a disturbing one. When the authors apply all their filters and metrics to a set of 728 protein-ligand structures frequently used in benchmarking docking programs and in actual structure-based drug design projects, they find that only 17% or so (121) of the structures make it through. They collect these structures together and call the resulting database Iridium which can be downloaded from their website. But the failure of the 580 or so common structures used by leading software vendors and pharmaceutical companies to pass important filters leaves you wondering how many resources we may have wasted by using them and how much uncertainty is out there. 

Something to think about, especially when considering the 50,000 or so protein structures in the PDB. What you see may not be what you get.