By Vicki Glaser
March 28, 2008 | SANTA FE, NM -- OpenEye Software wrapped up its ninth annual CUP (Customers, Users, Programmers) meeting last week by announcing the results of SAMPL-1 (Statistical Assessment of the Modeling of Proteins and Ligands), a prospective, blind challenge designed to evaluate protein and ligand modeling tools and methods and the predictive power of computer-assisted molecular design (CAMD).
There were no official winners or losers in SAMPL-1, and no rankings posted. In fact, participants could choose to remain anonymous or not reveal their results. “We need ongoing, independent, blind tournaments” such as SAMPL, which expose competitors to unseen data, said Paul Labute, President and CEO of Chemical Computing Group (CCG).
“Failure is a great opportunity,” said Dave Mobley, a postdoc in Ken Dill’s group at UCSF. “We did reasonably well, better than I thought we would, even with a very challenging test set,” he added.
In all, there were 54 participants, spread equally from academia/government and industry, representing both vendors and third-party users. They submitted a total of 205 predictions to OpenEye, generated using a variety of algorithms including ligand-based, structure-based, and QSAR methods, combined with hand docking techniques and other intuitive strategies.
The predictions were based on several core datasets. Phama provided the protein-ligand binding datasets for the virtual screening and binding affinity components of SAMPL-1: 52 molecular structures from Vertex for JNK3 kinase; and 27 compounds from Abbott Labs for the urokinase protein target, all with crystal structures, as well as a large collection of decoys. Peter Guthrie, from the University of Western Ontario, contributed 63 previously undisclosed vacuum-water transfer energies, from which participants derived 3-D coordinates, conformations, tautomers, and charge distributions.
A SAMPL Plan
The motivation for SAMPL was clear — design a prospective challenge focused on molecular modeling that would deliver an unbiased evaluation of how well modeling and simulation software tools and computational strategies for drug design and lead optimization are faring. The evaluation had to be a prospective one to provide a realistic view of the quality and value of predictions made by modelers and computational chemists.
It is too easy “to parameterize away a problem and get the answer you already know” with retrospective evaluations, says Nicholls, who conceived the SAMPL challenge. That approach may well yield a workable model, but by trying to fit data to an existing theory or preconceived notion, the model will not be generalizable across targets or ligands and will be of little use in real-world situations. In contrast, prospective science has the potential to yield new, unknown data and conclusions.
Although Geoffrey Skillman, VP research at OpenEye, and mastermind and results tabulator for SAMPL-1, had not finished analyzing the large amounts of data still flowing into OpenEye up to the submission deadline, each participant received a summary of their results in advance of CUP IX. The preliminary results were generally enlightening — simultaneously encouraging and discouraging.
Overall, concluded Anthony Nicholls, president and CEO of OpenEye, “we didn’t do that badly.” While hardly a rave review of the predictive powers of molecular modelers, Nicholls argued that the challenge was an irrefutable success, judged less by whether participants succeeded in the task at hand, but whether they learned something from the process.
The meeting kicked off with a series of candid and humbling presentations by programmers and modelers from academia and industry on the status quo, notable for their probing self-assessment and the candid dialog they elicited. Stagnation is pervasive in the pharmaceutical industry, and poor productivity has been the appreciable outcome. Nicholls contends that drug discovery is suffering from a bias toward “’discovering’ what is already known” — marked by a lack of innovation resulting in the decades-long spate of me-too drugs and the rare eureka moment when a truly unique small molecule drug debuts.
The company plans to publish a review of the SAMPL process and results in a peer-reviewed journal.