By Salvatore Salamone
July 15, 2003
Plugged in: David Lucas, director of portal and therapeutics services at Celera, says a good connection with the vendor is essential. "It's like getting to date before you get married."
| What do cutting-edge life science research and The Beatles have in common? On the surface, not much. But when Celera Genomics — the company that ignited the race to complete the human genome — decided to upgrade its entire computing infrastructure, it had to be humming "Come Together."
Celera was founded in 1998 as a sister company to Applied Biosystems, with the avowed aim of cracking the genome faster and cheaper than the international consortium laboring to sequence 3 billion letters of DNA. The centerpiece of its operation, based in Rockville, Md., was a state-of-the-art Compaq supercomputer designed to assemble literally millions of DNA fragments.
But now Celera is replacing its Compaq Alpha processors with an entirely new system from IBM and EMC. The procurement process was very different than the norm. Internally, it involved bringing together legal, business development, and technical people from the start. And externally, it featured a much more open dialogue than usual with vendors competing for the deal.
As a result of this collaborative effort, Celera was able to go from an initial request for proposal (RFP) to a signed deal in roughly one year — a deal that saw the company purchase 12 IBM p690 Regatta computer systems and three EMC Symmetrix Storage Frame systems, for aggregate processing power of about 2 teraFLOPS (two trillion floating-point operations per second) and 150 terabytes of raw storage capacity. The system, to be fully operational by the end of the year, will rank as one of the top commercial supercomputers in the world.
The three Celera people responsible for the RFP were John Reynders, vice president of informatics; attorney Daniel Mazella; and David Lucas, director of portal and therapeutics services.
As he contemplated the task of upgrading Celera's massive computational infrastructure, Reynders wanted to do something different. He and Mazella were veterans of the government IT procurement process. Reynders got his experience working at Los Alamos National Laboratory; Mazella got his working at the U.S. Department of the Treasury.
One disagreeable facet of the government RFP process is the blackout period, during which a vendor submits a proposal, a procuring agent evaluates it, and then replies — without any interaction between the two sides. Reynders wanted more give and take. "We wanted a dialogue with the vendors," he explains. "That was one reason we put out a draft RFP — many vendors responded saying, 'Did you consider this or that?'" (Reynders would not comment on precisely how many vendors participated, due to nondisclosure agreements.)
Additionally, Celera was looking for more in the deal than just an equipment supplier. "It was not just 'Hi, we want to buy a lot of boxes,'" Reynders says. "We were looking for a partnership."
"It's like getting to date a while before you get married," Lucas says. "You have a sense of what an organization can provide."
More Floating-Point Operations
Since its founding in 1998, Celera boasted an impressive computational center comprised of Compaq AlphaServers and StorageWork storage systems (see: "Scaling Celera's Mountain of Data," April 2002 Bio·IT World). So why the sudden shift in computing architectures? After all, some companies go decades with the same basic systems in place.
Industry speculation on the reasons for Celera's abrupt change has centered on the fallout from the acquisition of Compaq by Hewlett-Packard. Some industry sources (none willing to be quoted) attribute the change to HP's decision last year to discontinue Compaq's AlphaServer line. Others suggest it was fear that the Compaq products would not get the technical support required from HP in the long term.
But Celera says the reason for re-evaluating its computational infrastructure is the dramatic change in the company's business direction — and thus its IT requirements — over the past two years. Additionally, much of the HP/Compaq gear was coming off lease.
"When I came on board about two years ago, the platform in place was appropriate for the [computational] task of assembling the [human] genome," Reynders says. "At that time, the platform was also appropriate for the business, which focused on supplying [informatics] tools and data." Reynders insists there was nothing but satisfaction with the installed HP/Compaq gear.
However, since Reynders joined Celera, the company has refocused from being a tools and data supplier to a company dedicated to therapeutics and diagnostics. "This refocusing drives down throughout the organization," he says.
| "These ongoing discussions gave us valuable insight into how we could work with a vendor as a partner."
vice president of informatics
In moving forward into therapeutics, Reynders explains, "We looked at our heritage — our bioinformatics, our supercomputing heritage — and asked, 'Is this the right platform?' Instead of supporting a single task — [assembling the genome] — new needs emerged, including supporting a wider range of applications and scientific disciplines, such as cheminformatics, SNP analysis, more types of bioinformatics, and calculations of binding energy."
"We looked at a companywide system that would support all the business functions," Reynders says. "We wanted a flexible environment," he adds, noting the need to support a diversity of applications. Also driving the need for a new system was the fact that the mix of applications would change as the company moved beyond its roots.
For instance, when supporting the original genomics sequencing work, most of the computations were integer-based. "In genomics, it's BLAST this, align that," Reynders says. "Now, there are many more floating-point operations required of our system as compared to earlier, genomics-focused computation."
In proteomics and therapeutic work, he points out, the types of computations are things like simulating how molecules dock. And there are lots of data coming from mass spectrometers. Additionally, Celera is doing more computational chemistry. All these operations involve floating-point calculations and moving lots of data around.
Some life science companies would apply different computing resources to the various compute tasks. For instance, at the Virginia Bioinformatics Institute at Virginia Tech, scientists are given a choice of running routines on either a Sun Fire 15K supercomputer or an IBM 102-node Linux cluster. This entails different processors and different operating systems for different tasks.
This approach works in some environments, but Celera felt there were more benefits to using a single type of processor and a single operating system.
Celera generated a draft RFP in the late summer of 2001 that included a description of system performance, storage, and other pure technical specifications. This draft RFP was floated at the 2001 Supercomputing conference in Denver. Vendors were given about a month to chime in with initial comments.
"This first RFP almost brought it down to the box level," Lucas says. "We decided to take a step backwards. Why dictate something that was good for genomics? Let's be more vague and let the vendor specify the architecture."
So Celera decided to release a GFP — a guideline for proposals — rather than an RFP. This provided more leeway for the vendors to be creative. "They could choose clusters, [symmetrical processor] systems, whatever," Lucas says.
Celera's GFP went beyond the straight list of technical specifications that needed to be satisfied. The GFP had "a very large technical requirement, a pricing component, and a business partnering element," Reynders says. "We made it clear they had to cross the technical bar first. But the other elements were also very important."
| "We asked ourselves, were we picking a proposal or a partner? We decided we were picking both."
Daniel Mazella, attorney
M azella agrees. "We asked ourselves, were we picking a proposal or a partner? We decided we were picking both."
When it came to the partnering side, Celera was faced with the same problem many companies encounter when setting out to work with a vendor: How do you judge beforehand how the interaction with a vendor will go?
Issuing the GFP with no blackout enabled Celera to receive continuous feedback from vendors. "These ongoing discussions gave us valuable insight into how we could work with a vendor as a partner," Reynders says.
"The ongoing dialogue prevented misunderstandings," Mazella adds. "And the dialogue approach is faster than sending a document and waiting for someone to look at it and reply, as is the case when long blackout periods are [in place]." Mazella notes that in the proposal process, Celera would sometimes have no communication with vendors for a week or two, but such periods are minor compared to the formal blackouts used in traditional RFPs.
Celera also took a different tack when it came to benchmark testing within the proposal process. Vendors are typically asked to run a set of relevant tasks to gauge the raw performance of their systems. But Celera gave vendors a suite of computational tasks and some tasks involving data management and I/O. Rather than asking the vendors to simply run the suite of tasks without any modifications, Celera worked with each company to optimize and recompile the code for the specific system it was proposing.
"We got two things from this approach to benchmarking," Reynders says. "We got a hard fact — here's the speed — and a soft fact — here's how we work together. This is a better benchmark."
Lucas adds, "This is a far more thorough process than normal."
While Reynders puts considerable emphasis on the importance of vendors meeting technological and partnering criteria, his comments on the third crucial aspect of the RFP — pricing — are closely guarded. Price alone, he notes, was not the key to winning the deal. "I didn't care if I got the [system] for free if it didn't meet our needs," Reynders says.
He cannot disclose the value of the contracts because of nondisclosure agreements, but he says Celera got a fair deal. At the time of the announcement, industry analysts estimated the package to be worth from $10 million-$25 million.
Due to the overall economic slump, Celera was in a good negotiating position. How good a position? After the contract was announced, HP's Mark Gonzalez, vice president of storage sales, told Computerworld that HP was not "willing to give away the hardware to keep the account." (Celera executives responded that they did not know what Gonzalez was talking about.)
Judging by other recent RFPs, Celera was shopping in a buyer's market. For example, when biotech company diaDexus was looking for a 16-node Linux cluster to replace its SGI system, competing vendors slashed prices. "IBM dropped its price by almost 50 percent," says Tim Burcham, diaDexus' senior director of bioinformatics. Cluster vendor Linux NetworX "couldn't go down that much, but did drop its price," which included installation, Burcham says. "I don't think we could have built the cluster ourselves for the same price." As a result, dia-Dexus opted for the Linux NetworX solution.
|J. Craig Venter, IT Consumer
|Former Celera President J. Craig Venter spoke with Bio·IT World last year about the history and future of IT at his former company...
The pressure companies can exert on vendors seems incredibly strong today. Vendors complain they are not only being asked to cut prices but also often required to participate in "reverse auctions." (These auctions are termed "reverse" because the price typically goes down rather than up.) Such auctions have vendors bidding, usually online, directly against each other.
The consultancy Giga Information Group reports that reverse auctions can save companies from 10 percent to 30 percent on their IT purchases. But there is no rule of thumb. If the bids have to do with commodity products, there is a better chance of savings. But auctions on something less quantifiable, like services, may not yield much cost benefit.
Making the Transition
The collaborative spirit did not end once Celera had selected IBM and EMC. In fact, the same internal teaming was applied to the transition from the old HP/Compaq equipment to the new IBM and EMC gear. "We could have come in with bulldozers and mandated the change," Lucas says. "But we worked with the business units managing the migration to budgets and schedules."
One approach Celera considered was a single migration: Everything — applications and data — would be moved off the HP/Compaq systems and onto the IBM and EMC equipment. Instead, Celera opted for a dual migration strategy, which essentially entailed moving the data first, then the applications. In this way, "We could return the expensive storage [equipment] now, and we could work with the business units to port their applications," Lucas says.
He notes that part of the migration process involved porting and testing the applications. To help the departments, Celera set up "sandboxes" on the new system — guarded areas that let programmers try out software without compromising existing applications of data.
Within budgetary constraints, Lucas is letting individual groups make their own decisions as to when to migrate everything to the new systems. The approach has been well received. "I've been surprised how many people have gone off and ported their own code," Lucas says. The company expects the entire migration to be completed by the end of 2003.
Celera's approach of directly involving the three internal groups — technical, business development, and legal — in its three-pronged proposal process — technical specs, partnering, and pricing — has proven to be a success.
As The Beatles computed in "Come Together," it all adds up. "One and one and one is three."
PHOTOGRAPHS BY CHRIS HARTLOVE