TurboWorx, in a partnership with IBM, has extended its workflow and distributed computing software to include IBM’s On Demand computational services.
As a result, computation jobs, which up to now TurboWorx could run on a variety of in-house compute nodes and clusters, can also run on systems in IBM’s managed, high-performance On Demand computing centers.
The practical consequence of the partnership is that life science organizations will now be able to integrate the running of in-house and on-demand compute jobs – something that is not commonly done today.
“With this partnership, [life science organizations] can set up workflows that extend their in-house capacity to take advantage of IBM’s On Demand computing services,” says Jason Alter, vice president, marketing and services at TurboWorx.
This very different from the norm. On-demand computing services have been offered for several years to handle peaks in computing needs. But up to now, these services have typically been used for specific, dedicated projects and computing jobs.
With the partnership, a life science organization using TurboWorx’s software could design a computing workflow that automatically accesses IBM’s On Demand services within the execution of a job. For example, if a cluster is running one application and a second job needs more processing power than is available, the TurboWorx software could be set up to automatically run the second job using the On Demand service.
Similarly, the incorporation of on-demand computing power could be used in computing workflows where certain steps in a workflow require more processing power than might be available at the time the step is executed.
Thought Experiment
When discussing the partnership, a TurboWorx customer came up with a hypothetical application where this linkage between in-house and on-demand might help speed its research efforts.
The customer (a research organization that did not want to be named) needs lots of computing to help with its microscope image processing. One idea it had is to tap the processing power of Blue Gene as offered through an IBM On Demand service.
The application it has in mind involves the deconvolution of its microscope images. Deconvolution is a technique commonly used in image analysis. The need for this technique lies in the fact that image data is a combination of the true image plus factors such as distortions or variations due to the instruments and techniques used to capture the image. If one can determine the impact of these extra factors and compensate for them, the true image can be recovered.
An analogy can be made with a 35-mm camera that accidentally over-exposes an entire roll of film. A person developing the film could correct each photo one-by-one. But if that person knows that all of the photos have been over-exposed by the same amount, he or she could determine a corrective action and apply the same correction to all the photos on the roll.
The deconvolution process with a research organization’s microscope images has several steps. First, stored compressed images must be decompressed. Then all images are run through deconvolution algorithms to search for and determine any factors that contribute to image distortion.. Once this is done, any image called up to be viewed or analyzed is corrected using the information derived from the deconvolution analysis of the entire set of images.
In such a workflow, things like the compression, decompression, display, and analysis of individual images could be handled with in-house computing facilities. But the deconvolution of the entire set of images is something that could use on-demand processing power.
Currently, deconvolution is run on large, shared-memory systems. While the algorithms are highly parallel, they have not typically been run on distributed-memory systems because of the huge amount of overhead required to move the large volumes of data around. But Blue Gene lends itself to this application because of its very large processing power. That is why this hypothetical workflow would be a good application of incorporating in-house and on-demand computing power.
There are many other life science applications that have this asymmetry in their computing workflows that would also lend themselves to a tight integration between in-house and on-demand services.