BY MARK D. UEHLING
July 15, 2003 | When did Pfizer really know Viagra worked? Not from observing mice in the lab. Nor could nurses and lab tests confirm how Mr. Johnson really performed in the bedroom. But when it examined the diaries of patients taking Viagra, Pfizer knew that 48 percent of patients swallowing the little blue pill reported "successful" sexual encounters. That compared with a mere 12 percent of men taking a placebo before a big night out.
For a wide spectrum of conditions — incontinence, depression, and pain — patient diaries yield invaluable insights into whether any medicine is working. They are used in more than a quarter of all clinical trials. The nagging issue is what those diaries do not say. Physicians and regulators have been so intent on gathering patients' day-to-day experiences that they have overlooked chronic problems with the technology — and people — recording those experiences.
A recent study is making the inadequacies of paper-based diaries much more difficult to brush under the carpet. In a paper on chronic pain, published by psychiatrists at Stony Brook University in April ("Patient compliance with paper and electronic diaries," Control Clin Trials 24,182-99), the number of patients who recorded their experiences as instructed was only 11 percent among those using paper diaries, compared with 90 percent of patients with a personal digital assistant (see "Paper vs. Plastic").
Although patients intend to help, all-too-human tendencies to procrastinate and exaggerate overwhelm their promises to record their lives accurately. As the new study shows, many patients in clinical trials consistently fabricate their diary entries.
Lying to Your Diary
The Stony Brook study involved the use of both paper and PDAs for record keeping. As part of the research, cooperating scientists at Invivodata (a developer of electronic patient diaries) embedded a light sensor in the paper diaries to secretly record when patients opened their diary. That trick was revealing. On more than 30 percent of the days paper-based patients claimed to have made entries, they did not even open their diaries.
With electronic diaries, oddly enough, the rate at which patients cooperate is at the opposite end of the spectrum. In one recent clinical trial, patients using an Invivodata-supplied PDA and taking a pain medicine after corneal refractive surgery had 99-percent compliance. That drug, Allergan's Acular LS, is the first medicine approved with electronic patient diary data as a pivotal component of the approval decision. "This is a milestone for Invivodata," says Invivodata CEO Doug Engfer, "and it's a milestone for the industry. We're the first and only vendor to have cleared this hurdle."
Allergan seems to agree. "As primary endpoint data for this drug, it was critical that submitted data were collected using a proven methodology that would meet FDA 21 CFR Part 11 requirements," Elizabeth Bancroft, senior director of regulatory affairs at Allergan, said in a press release. "This approval validates our confidence in the self-report data collected."
Confidence, oddly enough, is not a word that can always be used with the diary data collected on paper. Michael Hufford, Invivodata's vice president of scientific affairs, ticks off journal after journal article in which patients fabricated events in the past — or fibbed about how they would feel in the coming days. But our brains do not contain alphabetized manila folders. "Instead," says Hufford, a clinical psychologist, "the brain reconstructs probably what happened. That process of reconstruction is both inaccurate and systematically biased."
Reality Check: The PDA
Hufford and the company's founders — who used Psion handhelds in National Institutes of Health grants at the University of Pittsburgh in the late 1980s — recognized how pervasively wrong paper-based data might be. "Electronic patient diaries really fix a fundamental problem that cannot be resolved in any other way," Hufford says.
Companies researching nicotine patches, for example, want to know how discouraged smokers are at the moment they elect to resume smoking after a period of abstinence. It turns out that in real time, most smokers are not so discouraged. When a clinician asks the same question a month later, Hufford says, "For folks that return to smoking, they systematically exaggerate how much they felt like giving up. That's one example of how people try to make sense of their lives, by changing the past to fit a coherent story."
Reading palms: Electronic diaries help patients record their symptoms and take their medication.
In other cases, drugs may be working much more effectively than life sciences companies can measure using paper diaries. Invivodata's research shows that the placebo effect (the tendency to believe a sugar pill is working) is less pronounced when human memory is removed from the equation. Says Hufford: "For patients who relied on recall, either the end of week or the end of month, they demonstrated a far more significant placebo effect than if you used the data captured in real time. Capturing real-time data from patients can help to minimize the placebo effect."
Other companies, including etrials, PHT, Logos Technologies, CRF Box, and Symfo, to name but a few, are also working on electronic diaries. All say that real-time devices generate more consistent, reliable data — partly by programming out the freedom to commit clinical misdemeanors. Unlike Nigel Tufnel in the movie This Is Spinal Tap, you can't give an "11" on a scale (of pain) that goes to only 10.
At the FDA, meanwhile, Robert Temple, director of the office of medical policy at the Center for Drug Evaluation and Research (CDER), is fully aware of the published failings of paper diaries. "I find those results plenty disturbing," he allows. Is the agency getting ready to issue any guidance encouraging the migration toward electronic diaries? In a word, no. "I don't think we've decided to do that yet," Temple says.
Getting the day or moment of a medical experience wrong, Temple says, does not mean that the underlying event — a seizure, say — is fabricated. It may just mean the time of the seizure is not recorded accurately. Most sufferers of migraines, to take another example, are probably not going to get the number of such headaches in a week wrong. Each one is all but unforgettable.
Temple believes the scientists who run clinical trials will probably migrate to electronic diaries on their own. What the FDA is actively trying to figure out is whether existing regulations protect the patient data once they are uploaded: "We need to worry about what the standards for maintaining these records are going to be. That's what we're thinking about most actively now."
It's not clear, Temple says, whether the agency's hallowed 21 CFR Part 11 regulation fully covers all of the necessary data-protection issues for electronic diaries. "The fundamental question we have," he says, "is: Can you make sure these data have the same sorts of properties [as paper diary data] of giving you the assurance that they have not been altered?"
Hufford, in response, says that the electronic password-protected and time-stamped diary entries are actually more difficult to tamper with than the contents of paper diaries, which may or may not be under lock and key. What's more, in paper diaries patients often put a single vertical hash mark along a horizontal spectrum — and such minimalistic lines cannot be authenticated in any meaningful way. "A compliant e-diary system ensures data security" at levels satisfactory to the financial industry or the U.S. military, Hufford says, "and provides a complete electronic audit trail for any changes made to the data, whether by the subject or the investigator."
Beyond the FDA concerns, Hufford says, IT helps set Invivodata apart. "Simply putting questions on a screen is not sufficient to capture real-time data," he says. Screen commands are ultra-simple. Fonts are huge. Even the beeping tones on its devices have been chosen to ring at frequencies that elderly patients can hear. In a recent prostate cancer trial using the company's diaries, for example, such seemingly simple features helped the patients achieve 91-percent compliance, despite an average age of 78. Hufford is irked that his competitors have the audacity to cite (and win business on) Invivodata's 40 scholarly articles, published in the likes of British Medical Journal, Pain, and Annals of Behavioral Medicine. But he cannot resist noting that in one incontinence study, a rival company's PDA reminded patients about their bladder problems so frequently it actually improved their incontinence. That may sound great, but it obscured whether the medicine in the trial worked.
The key, Hufford says, is to be a part of the patient's life — but not too intrusive: "Just beeping patients is not enough to get high rates of compliance. We are able to do that through the application of ecological momentary assessment, or EMA. It's the science of capturing real data in the real world."
Curing Double Data Entry?
On the IT side, some of the ingredients of the company's secret sauce are obvious. Invivodata uses the Palm OS, linking via a cradle or wirelessly to a Microsoft SQL Server database. Exporting into SAS or Oracle Clinical is no problem.
|"Just beeping patients is not enough to get high rates of compliance. We are able to do that through the application of ecological momentary assessment, or EMA. It's the science of capturing real data in the real world."
Michael Hufford, Invivodata
Other aspects are held closer to the vest. "We just closed and locked a database that had 10 million records," Hufford says. "If all you have [are] the uploaded data, you know nothing about how compliant the patient was with the protocol. The real challenge is: How do you aggregate the data in such a way to help the site coordinator quickly and efficiently give the patient feedback about how they're doing?"
So Invivodata strives to be able to help doctors and nurses running trials take action quickly. With paper diaries, that takes an eon. Why? Because of a widely accepted but grotesquely inefficient practice called double data entry, in which two different people manually transcribe each scribbled paper diary entry into computers.
In the end, Hufford says paper diaries simply don't enable conscientious patients to tell drug companies what's happening. "The problem with paper diaries," he says, "is that you are not helping the patient to succeed." If patients succeed, electronic diaries could provide a quicker, more convincing return on investment than other computerized data-gathering technologies for clinical trials. In the best scenario, cleaner data from such diaries could open sponsors' eyes to the advantages of moving other clinical information — from labs or healthcare providers — out of paper and into digital form from the start.