Nov. 15, 2006 | Remember CASP? Critical Assessment of Techniques for Protein Structure Prediction — the biennial competition to computationally predict the precise 3-D shape of an array of target proteins from their amino acid sequences. CASP is now 12 years old, quite an accomplishment really, and a meeting to evaluate the results of the CASP7 predictions is scheduled for this month (Nov. 26-30) at Asilomar Conference Center, Pacific Grove, Calif.
In CASP1 (1994) there were 33 prediction targets, and 35 predictions groups participated. In CASP7, 104 targets have been released, and 200 “human” groups of predictors and 98 “prediction servers” have registered. I don’t fully know what the latter group constitutes but will look into it. You may recall that HHMI investigator David Baker’s predictions were consistently near or at the top. He seems to have the gene for protein structure prediction (see “Computational Biologists Join the Fold,” Bio-IT World, June 2002).
Now comes DREAM — Dialogue on Reverse Engineering Assessment Methods — which hopes to conduct a systems biology contest and education program that’s modeled loosely on CASP. The D can interchangeably be used for Database, says DREAM co-organizer Gustavo Stolovitzky, manager of functional genomics and systems biology, the computational biology center at IBM T.J. Watson Labs.
Working with NIH funding and support from the New York Academy of Sciences, DIMAC (Center for Discrete Mathematics and Theoretical Computer Science), and IBM, the DREAM project plans to collect data and techniques researchers can use to understand how well their various reverse-engineering methods actually work. Andrea Califano of Columbia University and Stolovitzky are co-organizers, and the group held its first workshop in September.
DREAM is an excellent idea. At Bio-IT World’s annual conference this year, directly following a session on systems biology, one attendee and I talked about how terrific it would be to have a CASP-like shoot-out for pathway prediction tools. Little did we know that DREAM’s first organizational meeting had already taken place (March). And DREAM is much more than a shoot-out. Speaking at last month’s Regulomics Symposium, organized by Cambridge Healthtech Institute, Stolovitzky provided an update.
“We have seen today several very interesting attempts to reverse-engineer or infer biochemical pathways from data,” Stolovitzky told the audience. “So the reason the DREAM project came to life is because we don’t know whether these methods are giving us the right results or if sometimes we “cherry pick” the methods results that best allow us to publish a paper.”
One problem is the lack of a gold standard to measure results against. CASP, of course, uses the structure as determined by x-ray crystallography. “We are trying to determine what would be a gold standard now and what kind of metrics to use to determine if one reverse-engineering method is better than another,” says Stolovitzky. “We’re also trying to decide whether a database would [help] unify this field.”
Not everyone is excited about the idea, concedes Stolovitzky. Someone’s favorite algorithm might get dinged; papers based on the algorithm might be called into doubt; grants might become imperiled. As Stolovitzky points out, “a lot of us have been publishing these networks [and] they enrich the literature and [sometimes] they pollute it.” A survey conducted among attendees to the first workshop was positive but, interestingly, the proportion of thumbs up from informaticists rather outnumbered those from traditional researchers.
Working out kinks will take time, but this is a DREAM worth pursuing. Extracting real biological understanding from the staggering amount of omic data is the next big challenge. Static signatures, though useful, aren’t enough. We need to know what the biology is. Pointing to a particularly nasty network diagram, Stolovitzky recalled comments made by a colleague about one of his students who coined the term — Rediculome. The student noted, “We put together all these rediculograms and you will have a Rediculome.” Great line, whether original or not. Stolovitzky says the first DREAM competition is probably a year away. More information can be found at the New York Academy of Sciences website.