Oak Ridge Tackles Opioids With Assistance From World’s Fastest Supercomputer

By Benjamin Ross

December 14, 2018 | The US Department of Energy’s Oak Ridge National Laboratory (ORNL) is tackling the opioid crisis by using the world’s fastest supercomputer. With the help of IBM’s recently launched Summit supercomputer, the ORNL team was able to analyze 85 million genetic markers and understand  complex, epistatic genetic architectures responsible for challenging traits in phenotype.

The research earned the ORNL team the 2018 Association for Computing Machinery (ACM)’s Gordon Bell Prize, an annual award given in conjunction with the SC conference series that recognizes outstanding achievement in high-performance computing.

Dan Jacobson, chief scientist for computational systems biology at ORNL, told Bio-IT World the typical approach to studying genomes are unable to analyze the full scope of diseases.

“Opioid addiction is clearly multi-genic,” said Jacobson. “Straight forward mendelian genetics and even traditional [genome-wide association studies] are less likely to find really causative genetic causes for a range of things like addiction phenotypes, as well as autism, schizophrenia, and predilections to suicide. Pretty much any complex clinical trait you can think of.”

The ORNL team saw these limitations as motivation to step outside the box of convention. A little over three years ago Jacobson began discussing the issue with Wayne Joubert, a computational scientist at ORNL, who told Bio-IT World the process from discussion to implementation was expedient.

“Within about a month we had a prototype code using a Titan GPU that was 180 times faster than the original C code,” Joubert said. “Dan made a comment at that point that this opened up a whole new way of thinking about the possibility of what can be done.”

The algorithm developed by Joubert and his team is a Custom Correlation Coefficient method, implemented in the Combinatorial Metrics (CoMet) application, which compares variations of a given gene in certain populations, and maps it to Summit’s architecture.

The Tensor Core matrix multiplication capabilities, built into Summit with NVIDIA’s Volta graphics processing units (GPUs),  also gave researchers the ability to transfer and analyze large quantities of data, said Joubert.

“Although Tensor Cores weren’t designed with genomics data analysis in mind, as scientists we wondered if we could adapt our application to take advantage of the high performance offered by this NVIDIA feature,” Joubert said in an ORNL blog post. “In this case, we found a way to recast our problem to fit the hardware without losing accuracy and the results are pretty exciting. In one hour on Summit, we can solve a problem that would take 30 years on a desktop computer.”

The result was ORNL reportedly “breaking the exascale barrier,” Jacobson said, by analyzing genomic data at a peak throughput of 2.36 exaops (the equivalent to carrying out over 2 billion billion calculations per second).

Summit’s memory architecture also played a huge role in ORNL’s success, Jacobson added.

“Having half a terabyte of RAM per node is literally allowing us to solve problems that we had never been able to solve using previous architectures,” said Jacobson. “The collective architecture is allowing us to solve all sorts of problems, including machine and deep learning problems at scale, and at a rate we couldn’t have done before.”

The ORNL team is looking to apply the same approach to analyzing genomes in other conditions, including Alzheimer’s, cardio-vascular disease, and multiple sclerosis.

This effort is what he was brought on board at ORNL to do, Jacobson said.

“We came to Oak Ridge with the mission of putting big data and big computing together to solve complex biological problems,” Jacobson said. “To be honest there hasn’t been a lot of systems biology in supercomputing, and that’s something we decided was core to our mission and we think brings a large future audience to supercomputing.”

“This was literally impossible a few months ago,” said Jacobson. “So I think there’s going to be all sorts of fun science coming out of this.”