By Ann Neuer
February 2, 2009 | The Society for Clinical Data Management (SCDM) awards its Data Driven Innovation Award to a vendor/sponsor team for excellence in the advancement of clinical data management. The 2008 award went to Arrowhead Electronic Healthcare and Bayer HealthCare Pharmaceuticals for their innovation dubbed the “Dumb Monkey”, a catchy name for an electronic patient reported outcomes (ePRO) automated testing and documentation tool.
Shae Wilkins, VP Business Development at Arrowhead, says the purpose of the Dumb Monkey is to address one of the key challenges for data managers who work with ePRO in clinical trials, namely, “Getting a reliable solution from the vendor that meets all of the sponsors’ requirements and enables the sponsor to test that solution.”
The Dumb Monkey was created in response to some very tight deadlines put forth in an agreement between Bayer and Arrowhead in late 2007. Arrowhead had a mere six weeks to produce and ship 142 forms for its ePRO-LOG device in three languages—Hebrew, Russian, and Arabic. Then it had to ship versions in 16 additional languages just two weeks later. All told, 2,698 different forms had to be produced—142 forms in 19 different languages and locales.
Arrowhead met the challenge with the aid of the Dumb Monkey, an unscripted automated tool that gives input to the eDiary software detailing its ability to meet the customer’s requirements. It does this by virtualizing the test cycles at every aspect of product development. It walks randomly through the diary application, recording everything it sees by taking screenshots of all screens, testing every option to build flow diagrams, and building translation verification documentations.
According to Daniel Derr, VP Technology, the monkey can traverse through the application in three hours, grabbing 150 screens to generate documents and build them into a translation verification document, which could consist of 140 pages. For the Bayer project, the monkey generated all of the documents in 42 hours—Arrowhead estimated it would have taken 312 human hours to do the same tedious work. This is largely due to the monkey’s ability to work 24/7 and click on options three times faster than humans. Another factor accounting for the huge time savings was that without the monkey, Arrowhead would have had to give the device to a translator for each language, then map a way to go through the applications of each screen, and then review the application for adherence to the requirements to generate the verification document.
Going forward, Arrowhead plans to use the Dumb Monkey as part of its testing for other clients and will factor in the monkey’s capabilities when preparing proposals. Shae explains, “When conducting user acceptance testing, you are looking for every event to happen that can possibly happen. It’s hard to find the nooks and crannies, but with the monkey’s help, we will be able to give sponsors confidence that they have complete coverage.” From Bayer’s perspective, there is still the burden of doing its own internal manual testing, but the monkey approach does increase the quality of the system and allows for better use of human resources by reducing the iterations of manual testing the sponsor needs to perform.
According to Paul Clarkson, Director Development Information Access at Genentech and a member of the Board of Trustees of the SCDM, the Arrowhead-Bayer project stood out in the competition because it had “the most impact in terms of reducing effort and pain on operational efficiency for the system’s users.” The decision was unanimous among the three judges, all certified clinical data managers.