By Kevin Davies
April 28, 2009 | BOSTON – In the opening keynote of the 2009 Bio-IT World Conference & Expo, Chris Dagdigian delivered a candid assessment of the best, the worthwhile, and the most overhyped information technologies (IT) for life sciences.
Some formerly hyped technologies were now mainstream, including virtualization and storage. Others were over hyped but still on balance worthwhile, such as green IT and utility computing. And in the future, Dagdigian saw major benefits accruing from trickle-down best practices and federated storage.
Dagdigian, a founding partner for the BioTeam IT consultancy, is a regular speaker at the annual Bio-IT World Expo. This year, he brought his trademark “trends in the trenches” talk to the opening plenary session. Offering his customary disclaimers, he said he was “comfortable with deliverables, but not comfortable with being a talking head or a pundit.” The audience begged to differ.
Already mainstream
Dagdigian divided his talk into three sections: reviewing what he called “old news,” discussing “currently exciting” technologies, and those that could be exciting in the future. But, he warned, there is “a ridiculous list of stuff on the ‘hypemeter.”
In the “already mainstream” category, Dagdigian tagged virtualization and the “bio-IT storage tsunami.” Most of his recent work was in helping “people getting buried by instruments.” The community had been talking about the “data deluge” for the past four years. It was “still buried,” said Dagdigian, “but the problem domain is understood. Our smallest customers aren’t losing hope, and our biggest customers are staying ahead.”
Last year, Dagdigian saw the first “100-Terabyte (TB) single namespace project.” But, he cautioned, he had also witnessed for the first time a “10 TB catastrophic data loss” with consequent job losses. The loss, which occurred in a government lab, resulted primarily from double-disk failure in a RAID5 volume holding SAN F5 metadata. Dagdigian said he used to be “huge fan of RAID5,” but abandoned it last year. “The statistical probability [of failure] is too high – it’s going to happen!” Everything is now RAID6, with maximum attention paid to monitoring and maintaining data integrity.
In a welcome development, data triage discussions were spreading beyond cost-sensitive industry organizations. “Everyone has come to the realization that data triage is a given,” said Dagdigian. Displaying his favorite slide, Dagdigian showed a screen short of an 82-TB folder on a Mac. Even more impressive was a 1-Petabyte (PB) output mounted on a Linux system output.
While storage was cheap (and getting cheaper), operational costs, staff, tape and backup costs were still fairly constant. Users, enamored with the plummeting price of TB storage devices from Costco, did not understand enterprise IT and backup requirements. “IT organizations need to set expectations, because the electronics market is skewing expectations,” said Dagdigian. Many next-generation sequencing grants simply didn’t budget for storage, let alone a 100+ TB storage system. Meanwhile, Dagdigian noted, the Broad Institute already has more than 1 PB of storage.
“Unlimited data storage is over,” noted Dagdigian. It’s simply not possible to back up all data, keep it safe, secure, and so on. “Sometimes,” he said, “it’s better to go back to the -40 F freezer and repeat the experiment.”
In short, storage was no longer a major bottleneck – rather, that fell to chemistry, reagents, and human factors. Customers were starting to trust instrument vendor software more. ”The problems are not as scary as they once seemed,” he said. Dagdigian also noted that storage devices are running more 3rd party software, such as Ocarina Reader software on Isilon, and Ocarina Optimizer on BlueArc.
Virtualization offered the “lowest hanging fruit,” said Dagdigian. The tipping point, Dagdigian said, was the live migration of a VMS (virtual memory system) without requiring a proprietary file system underneath. In 2009, he helped build and design a virtual collocation facility for an academic west coast campus, which was experiencing limits imposed by electrical power and air conditioning. “400 servers are currently virtualized on a lightweight simple platform,” said Dagdigian. Large numbers of physical servers had been shut down, realizing significant savings from de-duplication, compression and thin provisioning, not to mention electricity. Moreover, scientists now had full administration control.
Coming soon, said Dagdigian, “virtualized cluster head nodes.” Not coming soon: grids and clusters distributing entire VMS for task execution. It wasn’t practical, argued Dagdigian but rather a case of “marketing winning out over practical stuff.”
Green IT
In the category of “hyped beyond all reasonable measure – but still worth pursuing,” Dagdigian said green IT could deliver real electrical savings. “Use green IT for political cover,” he urged the audience. A deployment of a Nexan SATABeast had led to a 30% reduction in power draw with no impact on cluster throughput. One of his best 2009 moments came in deploying a Linux HPC cluster for a west coast organization. The system interface talks to Platform LFS, powering nodes up and down and sending automatic email alerts to management such as: “Hello, I’ve saved $80K in facility costs this year.”
Utility (or cloud) computing was “not rocket science, but fast becoming mainstream,” said Dagdigian. “Amazon web services is the cloud,” he said. “It’s simple, practical, and understandable,” and enjoyed a multi-year head start on the competition. The rollout of features was “amazing,” such as Hadoop and applications for short-read sequence mapping. “I drank the EC2 Kool-Aid: I saw it, I used it, I solved real-world problems,” said Dagdigian.
The biggest problem was ingesting data into the cloud. “There is no easy solution,” he said. How does one push 1TB/day into Amazon? “Have patience,” said Dagdigian, expressing “100% confidence” that Amazon is working on the problem. “If we can solve data ingestion problem, I see a lot of scientific data taking a one-way trip into the cloud. Data would rarely, if ever, move back. If I take it back, it’s going to be really obnoxious -- big data pipes or people driving minivans of USB drives.”
Worth Watching
In his “worth watching” category, Dagdigian cited federated storage and the trickle down of best practices from Amazon, Google and others. Amazon, Google and Microsoft had all been computing at such a scale that it was too much of a trade secret, he complained. But there were signs, such as a recently released video of the Google data center circa 2004, that their best practices will eventually trickle out, benefiting the entire community.