By Salvatore Salamone

March 26, 2008 | It’s no secret that life sciences organizations must deal with ever-growing volumes of data. New lab equipment, lab automation, and computer simulations are increasingly generating more and larger data files, all of which must be stored, backed up, and managed.

Unfortunately, the data management challenge will likely only get worse. The life sciences, like many other fields, are undergoing an unprecedented data explosion, according to new research released this month by IDC.

In the study “The Diverse and Exploding Digital Universe,” IDC estimates that by 2011, the total amount of electronic data created and stored will grow to 10 times the 180 exabytes that existed in 2006. That represents a compound annual growth rate of almost 60 percent.

Interestingly, the report notes that in addition to the increase in the volume of data, there is also an increase in its diversity, due to the use of such things as video, voice over IP (VoIP), and RFID. IDC notes that this complicates data management since the number of electronic information containers (files, images, packets, meta-tags, etc.) is growing 50 percent faster than the number of gigabytes. In fact, IDC estimates that the information created in 2011 will be contained in more than 20 quadrillion containers.

With respect to data management, there is good and bad news.

IDC estimates that less than 5 percent of all data emanates from datacenter servers, and only about 35 percent emanates from the enterprise overall, mostly from workers at their desks, on the road, or working at home.

However, the report notes: “While 70 percent or more of the digital universe is created, captured, or replicated by individuals — consumers and desk and information workers toiling far away from the datacenter — enterprises, at some point in time, have responsibility or liability for 85 percent of the data."

How can that be? Well, many users store personal digital photos on company computers. Or, they may download pirated MP3s to the office computer or upload copyright protected videos to YouTube from work.

All of this has great implications for a life sciences organization’s data management practices, including information security, privacy protection, copyright protection, screening for obscenity, detecting fraud, reporting on and archiving the content, searching and retrieving, and disposal.

To address these issues, the IDC report recommends that IT departments:

• Transform their existing relationships with the business units. These are the groups that will classify information, set retention policies, and face the public if data is lost, breached, compromised, or simply handled badly.
• Spearhead the development of organization-wide policies for information security, information retention, data access, and compliance. And IT must extend these policies to business partners.
• Rush new tools and standards into the organization, including tools for storage optimization, unstructured data search, virtualization to pool resources, and management and security tools.

Embracing these three practices will help organizations better deal with the data explosion that will continue, unabated, over the years to come.