Despite potential advantages, pharma is taking a cautious approach to adaptive designs, resulting in a slow but sure restyling of the research enterprise.
By Deborah Borfitz
June 10, 2008 | Among the broad array of statistical methodologies being piloted in clinical research, none draw as much hope and consternation than adaptive designs that involve interim data analysis. (See “Real-Time Trials,” Bio-IT World, June 2006).
The potential of this approach is promulgated by vendors including Cytel and Tessella, which sell the accommodating software, together with a handful of evangelists within big pharma. Virtually unknown, however, is how these adaptive clinical trials (ACTs) play out in the real world and the overall tenor of regulatory agencies on the matter.
Aside from debate about the promise of ACTs in reducing development timelines and costs by utilizing actionable information sooner, much of the dialogue occurs behind closed doors. The FDA, which has been promising guidance on ACTs for more than a year, is currently handling novel designs on a “case by case basis” to give the agency “experience and definitions that will be useful for later advice,” states Crystal Rice, spokesperson for the FDA’s Center for Drug Evaluation and Research. The FDA does not track the number of adaptively designed protocols it evaluates, but “most all of the medical areas are receiving some protocols that have some aspect of the novel adaptive design associated with them.”
Generically, an ACT describes an assortment of statistical approaches, including widely accepted designs such as “early stopping” and “dose-finding,” says Donald Berry, head of the division of quantitative sciences and chairman of the department of biostatistics at the University of Texas MD Anderson Cancer Center. (Berry is also an independent adaptive design consultant.) Seamless trials, notably oncology studies that combine phases I and II, are also fairly common. Given the problems pharmaceutical companies have had in accurately establishing dosage, the FDA actively encourages adaptive approaches in early phase trials.
But in later stage ACTs, Berry says the FDA worries about the intrusion of bias and the inability to accurately read a treatment’s false positive rate. “Almost all of [the ACTs] I’ve done,” says Berry, “occurred before the confirmatory aspect kicked in.” The exceptions were a few seamless phase II/III trials.
The European Medicines Agency (EMEA) appears more comfortable with seamless ACTs than the FDA. Last December, the EMEA helped organize a workshop focused on adaptive phase III designs. Robert Hemmings, a member of EMEA’s Scientific Advice Working Party, reports that the number of adaptively designed phase III trials is growing so quickly that the agency has “stopped counting.” The EMEA received 15-20 scientific advice applications in the prior two years across a variety of therapeutic indications and the “vast majority” were confirmatory studies and, more specifically, seamless phase II/III combinations incorporating dose selection, sample size re-estimation or both. Approximately 50% were single pivotal studies.
Among the adaptations deemed “problematic” by the EMEA, Hemmings says, are those that adjust the randomization ratio, resulting in a possible shift in population recruited. The primary endpoint also needs to reflect patient benefit and be “independent” of interim data. The reasons for not endorsing several proposals for an adaptive design strategy included “lack of acceptable rationale, totality of evidence likely to be inadequate (due to early stopping), Type I error (false positives) not adequately controlled, concerns over dissemination of interim information, and inadequate pre-specification of intention to adapt.
The FDA comfort level with ACTs tends to be division-dependent, says Berry. “Usually, the FDA wants you to draw a line in the sand [regarding] when you switch from phase II to III and will only count data after that switch.” But with rare conditions, such as a spinal cord injury, or in an area of high unmet medical need, such as a treatment for pancreatic cancer, the agency is more apt to “lower the bar.”
There’s not much expertise in the “adaptive” business, says Berry. The FDA doesn’t know what to suggest. But absence of guidance on ACTs is not a key holdup in the adoption of adaptive designs.
It is, however, making companies almost universally mute on the topic. A half dozen companies that have publicly acknowledged doing ACTs declined to be interviewed about the particulars of any of their seamless, later-phase trials, citing bad timing (in one case, conflict with a study’s upcoming publication in the medical literature) and general discomfort in talking about cost savings of the approach. Normally talkative technology vendors have also been unusually quiet, hoping not to raise further the anxiety level of their clients.
For self-teaching purposes—and to please regulators—some sponsors have opted to do “inferentially seamless” trials that transition into traditional confirmatory trials once dosage is firmly established, says Berry. MD Anderson, together with the FDA and the National Cancer Institute, is eyeing the potential of adaptive designs targeting biomarker-defined populations. The idea is to look at the benefit of different therapies and combinations of therapies based on patients’ genetic makeup.
From the regulators’ standpoint, the overriding concern about ACTs seems to be who has access to information during interim analysis, says Trevor Mundel, global director of the immunology and infectious disease for Novartis Pharmaceuticals. Companies need an independent data monitoring committee (DMC), “probably outside the company,” and an independent statistician to do evaluations. The regulatory worry is whether a company will live up to the guarantee of data integrity implied by a DMC. “It was easier for us to go to regulators the second time…based on what was done in the past.”
The fear is that information leakage will potentially skew trial results as well as incite litigation by investors on Wall Street, says Berry. The problem is solvable by erecting firewalls. But that won’t necessarily stop speculation about “what it means” when the data safety and monitoring board (the formal term for DMC) wants to know more about a particular set of results or keep the study team from trying to “read” clues from the body language of the unblinded statistician.
Members of the DMC, as a rule, shouldn’t have much interaction with the sponsoring company’s senior executives, says Mundel. But there may be a clause in the DMC charter stating that any decision that would have major financial repercussions for the company, including stopping a study, would require talking to management first. “This clause may exist for Novartis,” says Berry, “but it is not standard in the industry.”
On confirmatory trials, the FDA may allow some senior managers of the sponsor company on the DMC—or none at all, says Michael Krams, assistant VP, adaptive trials, clinical development at Wyeth (See “Biting the Adaptive Trials Bullet,” Bio-IT World, May 2007). “There seem to be a lot of FDA ‘positions’ on that. My opinion is that sponsor involvement should not be totally excluded in meetings where the DMC reviews recommendations of the steering committee, because many of the decisions that have to be made are business decisions,” with potentially huge financial implications.
DMCs overseeing ACTs may well need someone who “understands the [protocol] design and what the algorithms are supposed to be” for troubleshooting between meetings, says Berry. (Until now, that role has often been played by Berry’s consulting firm.)
Shortage of Metrics
Based on the findings of an initial survey of 13 mid- and large-size pharmaceutical companies by the Pharmaceutical Research and Manufacturers of America, sponsor companies aren’t overly chatty with the FDA about their phase II ACTs unless the information is expected to be used for a regulatory submission.
Of 37 identified ACTs, three were phase I, one combined phase I and II, 15 were phase II, two combined phase IIA and IIB, nine combined phase II and III, four were phase III, and three were phase IV, according to Judith Quinlan, co-chair of the group’s case study work stream. “All but one focused adaption on dose.” Only nine of the 37, including seven of the phase II trials, used the Bayesian statistical method that provides a transition from sequential to continuous monitoring of trial data and allows for various parameters to be changed. Adaptations can be made to number of patients needed, eligibility criteria, and how patients get randomized to different treatment arms as well as drug dose.
Late-phase confirmatory trials were underrepresented due to confidentiality issues, Quinlan adds, although smaller companies are now stepping forward to share their cutting-edge case studies via statistical clinical research organizations and consultancies.
The medical literature is virtually devoid of reports on large, pivotal pharmaceutical ACTs and will be for at least another two years, says Jay Herson, senior associate in biostatistics at Johns Hopkins University. The notable exception is Pfizer’s oft-mentioned ASTIN (Acute Stroke Therapy by Inhibition of Neutrophils) study, published in Stroke in 2003, whose adaptive design was credited with killing the drug promptly and decisively. Only now are other companies beginning to approach the FDA with proposals for late-phase ACTs using novel methodologies.
Wyeth, a champion of the adaptive “Learn and Confirm” model of drug development (See “Ten Years After: Learn and Confirm,” Bio-IT World, February 2007), started six adaptive dose-ranging studies involving “dynamic termination rules” last year, says Krams. Eight more will launch this year. Mundel reports that Novartis is operating more than a dozen ACTs across all clinical phases.
Most ACTs that have been done so far appear to involve group sequential design and sample size re-estimation, says Herson. While the number of ACTs involving seamlessly-combined phase IIB/III trials is growing, they’re less understood than the older adaptive approaches and tend to stir controversy within the industry and, presumably, red flags within the FDA. “The big problem is that companies aren’t getting the dose right even when they do many phase II trials in the traditional fashion. Now they want to learn a lot in one trial and there’s a lot of room for error and a lot of room for the FDA to not approve a drug because [of recent product recalls and public safety concerns],” says Herson.
ACTs may be a convenient way for drug companies to size up questionable compounds. Berry points out that most ACTs his firm has done, including ASTIN, have been successful in that “they’ve killed off a drug very early. One possibility is that most drugs are duds. The other possibility, which worries me, is that companies use standard [trial] designs if they have a highly positive drug and do an [ACT] if they’re not sure the drug is any good. Proving a drug doesn’t work is a great service, but it’s not making us famous.”
Gains for Doctors and Patients
The major attraction of ACTs is the “higher information value for the research investment,” says Krams. Sponsor companies aren’t the only beneficiaries. “For investigators, [ACTs] are a more intelligent way to interpret the data. From the individual patient’s perspective, there’s a higher probability of getting a good treatment or not being allocated to a bad treatment.”
This is especially meaningful when it comes to adaptive randomization in trials for life-threatening conditions, including most cancers, says Mundel. “If you can do this and you start to see a subgroup respond better to your drug, aren’t you obliged to put more patients in that subgroup?”
Better treatment odds serve as compensation to investigators engaged in ACTs who face “an additional level of uncertainty” regarding study length and subject enrollment, as well as higher expectations in terms of timely data entry, says Krams. With adaptively designed studies, how long a study will run and how many patients will be involved “is not known at the beginning, but emerges as data accrues.” Most investigators have never conducted an ACT and are understandably skeptical, “but once they’ve gone through the experience, they clearly want to do it again.”
Study monitoring happens in an adaptive fashion, with site visits occurring “whenever there’s data to look at,” says Krams. Because data queries happen “without any delay,” they double as an educational tool for ACT investigators.
There is a “clear need” for additional information sharing with institutional review boards and ethics committees regarding the intention of an ACT, says Krams. “Some ethics committees have asked if we could share interim results with them whilst the trial is still ongoing. We believe that the only body which should see the data from interim analyses is the independent [DMC].”
Running the Numbers
Statisticians hold a special position in this arena, as they’re often the “gatekeepers of all that is quantifiable,” says Mundel. Unfortunately, he says statistics are “a black box” to a many scientists who implement, run and manage ACTs. Much of the decision making rests in the hands of the number crunchers.
This is problematic only if in-house statisticians are rigid about “statistical purity” in all they do, says Mundel. “Some [statistical] techniques have been carefully worked on and the theories are well understood and usually lead to the right conclusion,” he says, “but in the real world, there’s no proof that they actually work.” These include the routine practice of “analyzing data repeatedly over time when patients are dropping out of the study for various reasons.”
ACTs are not inherently “good” or “bad” and thinking of them that way can lead to tunnel vision, says Mundel. Statisticians who are fervently pro-Bayesian and push for every study to be adaptive could lead companies to make impractical investments in data collection technologies. (Berry notes one case in which investors pressured a biotechnology company into using a Bayesian methodology against its better judgment.) “A lot of organizations are attracted to adaptive designs because they believe they will yield results faster. But it can take companies a long time to get these kinds of studies started. The notion that an [ACT] can get rid of the white space between study phases II and III is the worst reason to do them.”
That’s because the “white space” simply gets moved to the front end of a trial, says Mundel, working out the details of study design, achieving consensus with regulatory authorities, and getting the necessary information technology in place. “Time savings are fictional,” he says. On the other hand, endless months of acrimonious debate about whether to utilize an adaptive design can also prove futile.
“From a design perspective, there’s an extensive up front investment in thinking time and to construct documents, such as a simulation report summarizing operating characteristics,” says Krams. “This is in addition to the protocol, interim and final statistical analysis plans, and DMC charter.”
None of this means that the time savings associated with ACTs are fictitious, says Berry. “First, the ‘white spaces’ are not equal. Usually, more time is saved than is used in setting up the trial. More importantly, there is a set-up cost associated with ACTs. The first one takes the longest—for the obvious reason that the company has never designed one before. After experience with setting up five or ten trials, the white space at the design stage is tiny, perhaps no longer than the time it takes to design a typical trial now.”
The expertise necessary to design and run ACTs is relatively scarce. Berry has helped design about 50 ACTs for two dozen pharma companies over the past couple of years. Quickly producing new randomization probabilities based on incoming data is a capability limited to a handful of companies, including Cytel, Tessella, and United Biosource Corporation. Among clinical research organizations (CROs), “maybe 5%” can handle ACTs. “They’re learning by doing, and we are teaching them,” says Berry. But someone at the CRO still has to “throw the switch” when, for example, patients begin to be randomized adaptively rather than equally across treatment arms.
The IT required for ACTs includes an interactive voice response system to accomplish adaptive randomization as well as electronic data capture to collect adaptive parameters in real time, says Mundel. The structure of the database also has to be finalized up front rather than mid-study.
Drug supply software is likewise a necessity, especially for adaptive dose-finding studies, to ensure there is sufficient quantities of the correct dose formulations at investigative sites, and “this has to be worked out well before the study starts,” says Mundel. Current drug supply technology is far from ideal due to lack of uniformity. “I think it’s the number one cause of delays across the board, not just for ACTs.”
The “dream” at Wyeth is a UPS-like setup for real-time drug supply chain management that is part of a fully integrated system housing all trial-related clinical and financial data, says Krams. “We do a good job integrating all the different functions on individual trials, but we want to develop the IT infrastructure to do it in a way that’s scalable.”
Technology is not the only and clearly not the biggest impediment to the adaptive approach, now widely regarded as valuable to both sponsoring companies and human subjects. The evolution to this promising new era of clinical research appears inevitable. But the transition may be hindered due to the shortage of information and experience, as well as informed leadership at the FDA.
Berry Consultants is one of the most experienced players in adaptive designs. This statistical consulting company has designed virtually every type of ACT, including adaptive sample size (stopping early or late for efficacy or futility), dose finding (and dropping), seamless phases (I/II and II/III), adaptive randomization (to better perform treatments), and identifying responding biomarker profile. Services include writing software, assessing a trial’s operating characteristics, and modeling disease course over time for trials with long-term endpoints. Frequently involved in monitoring trials to ensure the design is working as prescribed, senior statistical scientist Donald Berry has helped design about 50 ACTs so far.
The leading technology provider for ACTs is ClinPhone, which has delivered applications for more than 85 protocols with design adaptations. Tailored, fully integrated combinations of solutions cover central randomization, trial supply management, EDC, and electronic patient reported outcomes. ClinPhone can also provide real-time integration with sophisticated statistical software. It has been involved in all types of ACTs, including those involving dropping/adding treatment arms, modifying the randomization ratio and/or sample size reviews in dose-finding studies, adaptive cohort designs, and seamless II/III designs.
For the past decade, Tessella has been providing simulation and analysis tools for ACTs as well as the know-how to build and run the requisite information technology. It has implemented statistical models for phase I trials, phase II dose-finding studies using Bayesian statistics, and phase II/III seamless designs. Six top international pharmaceutical companies have worked with Tessella on ACTs.
United BioSource Corporation (UBC) offers virtually everything sponsors need to do ACTs, including study design and protocol development, study simulation and adaptive statistical calculations, clinical technologies, logistics and supply management planning, creating interim DMC reports with trial data tables, and establishing and managing the study DMC. UBC has been involved with more than 100 ACTs, including both Bayesian and frequentist methodologies.
On a consultative basis, clinical trial design firm Cytel has been involved with more than 30 ACTs—presumably, some using its East system for design, simulation, and monitoring purposes. Experience in a wide variety of ACTs, including proof of concept, dose-finding using Bayesian methods, seamless phase II/III, and phase III adaptive variants such as population enrichment and re-estimate duration. Cytel offers a full line of services vis-à-vis DMCs and has custom software and computational tools for simulating adaptive designs (used by Merck, among others). It trained the FDA/CDER in adaptive design methodologies.
This article appeared in Bio-IT World Magazine.
Subscriptions are free for qualifying individuals. Apply Today.