BMJ 2016; 354 doi: http://dx.doi.org/10.1136/bmj.i4437 (Published 16 August 2016)Cite this as: BMJ 2016;354:i4437
- Courtney Davis, senior lecturer1,
- Joel Lexchin, professor2,
- Tom Jefferson, honorary research fellow3,
- Peter Gøtzsche, director4,
- Martin McKee, professor of European public health5
- Correspondence to: C Davis firstname.lastname@example.org
- Accepted 11 August 2016
After the thalidomide tragedy governments in Europe and North America established new requirements for drug testing intended to protect the public from ineffective and unsafe drugs. As no drugs are entirely safe, regulators must determine whether manufacturers have provided enough evidence to show that drugs’ benefits outweigh their harms. Precisely how high the evidence bar should be set for marketing authorisation has been hotly disputed ever since the enactment of modern drug regulation. The industry has long contended that the process is too complex and expensive, delaying patients’ access to lifesaving drugs.
Despite three decades of streamlining of regulatory processes and drug testing,1 the industry, some patients’ advocates, lobbyists, and investors still argue that the current model of drug development and regulation is unsustainable. They have called for a “paradigm shift” to allow greater numbers of new drugs to be approved on the basis of preliminary data, with key information about benefits and harms to be collected only after drugs have come on the market and are being used by patients.2 There may be limited circumstances that justify rapid access to new drugs on the basis of minimal data, but evidence indicates that regulators have been too permissive in their interpretation of existing criteria for expedited approval,3 that current regulatory standards are already too lax, and that low standards have failed to incentivise genuine therapeutic innovation.1 4 5 6
The European Medicines Agency (EMA) has embraced a new model of drug testing and marketing called “adaptive pathways”—the brainchild of an industry funded think tank, NEWDIGS (New Drug Development Paradigms), at the Massachusetts Institute of Technology. The adaptive pathways model would skip several steps designed to protect patients from unsafe and ineffective drugs, allowing new drugs for “unmet medical needs” to be launched on the market faster, on the basis of fewer data. The EMA initiated a pilot project to test the new paradigm and reported final results this month.7 As the project was concluded before any of the pilot drugs in development had been granted marketing authorisation, it remains unclear how the EMA will define “unmet needs,” and crucial information relating to the design and reliability of studies to generate evidence of drug safety and efficacy is absent from the report.8 Nevertheless, the EMA has judged its pilot project a success and will integrate adaptive pathways into the existing EU regulatory framework.
The argument that adaptive pathways will benefit patients and public health is based on several important assumptions. Here we review those assumptions, finding that the evidence underpinning them is either lacking or contradictory, calling into question the EMA’s case.
Though the adaptive pathways model builds on existing regulatory mechanisms, the EMA’s proposal would extend the application of current approaches. For example, conditional marketing authorisation (which allows for approval at an earlier point in the drug development process) was justified on the basis that it would be reserved for drugs for seriously debilitating or life threatening conditions, orphan diseases, or emergency situations. But it does not seem that such restrictions would apply in the case of the new model, which envisages “adaptive licensing” for some drugs that would normally receive “standard” marketing authorisation.9
Also, the adaptive pathways model requires companies to collect and analyse “real world data”—an EMA term for observational data—to supplement evidence from preliminary clinical studies and, in some cases, as an alternative to randomised clinical trials (RCTs). Although effective post-marketing surveillance of drugs in general use has always been a crucial element in the protection of public health, using electronic health records and patient registries for evidence of drug efficacy is a problem.
Finally, the new model requires that companies seeking adaptive licensing also receive joint scientific advice, in the hope that early involvement of health technology assessment bodies in product development discussions would increase the likelihood that drugs would be reimbursed at the same time they receive marketing authorisation.
The EMA acknowledges that approval resulting from adaptive pathways would increase uncertainty about the benefit-harm balance of new drugs but claims that such uncertainty is ethically defensible where patients have unmet needs and that the uncertainty would progressively reduce as additional “confirmatory” data are collected in the post-marketing period.2 Critics have warned that drugs approved on the basis of limited evidence may harm patients,10 11 and health advocacy groups are concerned that adaptive pathways advance a deregulatory agenda that further erodes standards for the testing and approval of new drugs.12
“Iterative” drug development
Drugs approved through adaptive pathways would be granted an initial marketing authorisation earlier in the development process, on the basis of limited, preliminary data. For example, companies could test drugs in small, highly selected patient groups; conduct fewer, shorter term, and smaller clinical studies; or use surrogate instead of clinically meaningful endpoints. Or companies might be permitted to bypass phase III testing altogether and obtain marketing authorisation after completion of single arm, phase II studies.9 Product development would continue after new drugs have come on the market and are being used by patients, with companies seeking to expand from an initial, narrow indication to an increasingly wider patient population, or to confirm the benefit-harm balance of a new drug approved early in development or on the basis of a surrogate endpoint.7
A key assumption is that marketing approvals that are based on limited data can give regulators the information they need to determine whether manufacturers have shown a positive balance of benefits and harms to patients. But experience and evidence show that such confidence is misplaced.11 Nearly half of all investigational drugs that successfully complete phase II studies fail in phase III, mostly because of lack of safety or efficacy.13 14 This indicates that if new drugs are approved on the basis of phase II trials there is a 50:50 chance that they are unsafe, ineffective, or both. Systematic empirical evaluation of the clinical literature shows that smaller, short term, or single trials are more likely to overestimate, or to generate spurious, treatment effects.15 Evaluation of surrogate markers shows that they are often biased, may overestimate drug benefits,16 and correlate poorly with outcomes relevant to patients.17 18 There are several examples of significant harm to patients from reliance on surrogate endpoints.19 In view of the evidence that limited data are more likely to lead to false conclusions, it is difficult to see how the EMA can determine whether early evidence submitted by manufacturers provides a “sufficient” basis for judging “likely benefit.”20
A second assumption is that robust clinical studies can, and will, be conducted rapidly and efficiently once drugs are on the market, so as to mitigate risks to patients by confirming or refuting efficacy and safety as quickly as possible. Yet confirmatory studies are often slow to complete, and drugs with uncertain benefit-harm profiles may be prescribed to patients for many years, sometimes even until patent protection expires.3 The EMA claims that, while failures occurred in the past, the situation has much improved and that a 2011 EMA led study concluded that “compliance with the CHMP [the criteria of the EMA’s Committee for Medicinal Products for Human Use] to conduct studies is generally very good albeit with some aspects for improvement.”20 However, the most up-to-date evaluations of the status of conditionally approved drugs in the EU found that the median time before completion of confirmatory, post-marketing studies was four years,3 and that half of all studies were completed only after substantial delays.21 Bedaquiline, approved to treat multidrug resistant tuberculosis, is a recent example of the risk that long completion times can pose to public health. The EMA approved it in 2014 on the basis of a surrogate marker for efficacy, despite trial data indicating that it increased the risk of death fivefold.22 Instead of deferring approval until further RCTs could confirm or contradict this signal of harm to patients, the EMA agreed to a post-marketing confirmatory study of the drug’s long term effect on death, with study results not required until 2022.23
Reliance on “real world data”
The EMA’s adaptive pathways proposal specifies that the planned collection and analysis of “real world data” (non-randomised observational studies) is essential for approval through this route and that “repeat cycles of evidence generation” can “quickly refine or correct past decisions.”20 However, the EMA’s pilot project provides no support for these claims,8 and available scientific evidence tends to undermine them. With respect to safety, proponents of adaptive pathways claim that the availability of “big data” such as electronic health records will enable “rapid learning” in the post-marketing period.2 Yet past experience has shown that different observational studies investigating the same safety issue may produce conflicting results,24 and recent evaluations of regulators’ and researchers’ experience of using large, electronic health databases indicate mixed results with respect to signal detection and signal confirmation.25 26 27 28 One review found that large, publicly funded pilot studies in the EU and US have largely failed to provide credible evidence of new, unsuspected adverse effects or to yield reproducible results. The review authors argue that considerable progress is needed in database terminology and coding, data validation, and statistical methods before electronic health records can be a reliable resource for rapid assessment of drug safety.25 Such evaluations undermine confidence that observational studies can quickly reduce uncertainty. Pioglitazone for type II diabetes was approved by the EMA in 2000, but early evidence from animal studies and a subsequent trial in patients indicated that pioglitazone might cause bladder cancer. Now, more than a dozen observational studies have reached conflicting conclusions and failed to definitively resolve this safety concern,29and the drug (which is now off patent) continues to be marketed in many European countries.
The adaptive pathways paradigm also encourages the use of “real world data” to support claims about drugs’ efficacy and effectiveness, such as to confirm long term effects if initial approval was based on early or surrogate endpoints. However, reliance on observational data to evaluate drug efficacy is highly problematic,30 and the bias is, on average, larger than the estimated effect.31 There are many recent examples where observational studies that suggested a treatment benefit were overturned by RCTs. These include the protective effects of hormone replacement therapy (HRT) and non-steroidal anti-inflammatory drugs against dementia32 33; HRT against myocardial infarction34; and antioxidant vitamins against cardiovascular disease, cancer, and all cause mortality.35 3637 38 39 Moreover, when results of observational studies “go the wrong way” companies are likely to dispute unwelcome findings on the very grounds that such data are unreliable.40
Although RCTs too may produce false and exaggerated estimates of treatment benefit, RCTs are more likely to produce true results than epidemiological studies.41 Moreover, misleading results from RCTs most often arise because of inappropriate study design and conduct—a problem that more stringent regulatory standards would reduce.4 16 42 Concerns relating to the “generalisability” of RCTs43 could be tackled by adopting pragmatic approaches to the design of studies, while preserving the strengths of randomisation.44 45 Although rigorous observational studies serve to generate hypotheses and may complement RCTs that have robustly shown efficacy of a drug, they cannot be relied on to compensate for weak evidence of benefit or the uncertainties generated by adaptive pathways.46 47
Restricting or reversing use
Proponents of adaptive pathways assume that regulators, doctors, and third party payers can, and will, restrict prescribing or reverse use in response to emerging evidence of harms or lack of benefit. But when new drugs are trumpeted as “breakthrough” treatments it may be difficult to restrict use to a small patient population or to subsequently withdraw a drug from the market.48 Even among “regular” drugs many examples exist of off-label and contraindicated use continuing in the face of repeated warnings by regulatory bodies, and of monitoring requirements failing to prevent the occurrence of deaths and serious harm to patients.49 Studies of treatment practices that were subsequently contradicted by large, well designed RCTs show that doctors can be slow to abandon useless or unsafe practices.50 This is partly because new data may emerge piecemeal and are often disputed when financial rewards are high, conflicts abound, and the reputations of individuals—and sometimes whole specialties—are at stake.51
The EMA has claimed that recent experience with prospectively designed risk management plans shows that restrictions on use and regulatory warnings can be effective, and it cites two recent examples, though these are based on relatively small surveys.20
“Life cycle management”
The adaptive pathways model assumes that regulators can be trusted to implement “life cycle management” of new drugs and to act promptly when subsequent evidence changes the benefit-harm balance. Yet both the US Food and Drug Administration and the EMA have been criticised for neglecting to monitor or enforce post-marketing study commitments for conditionally approved drugs.3 52 The EMA claims that the situation in the EU has changed since implementation of the EU pharmacovigilance legislation in 2012.20 A report on the activities of the EMA’s new Pharmacovigilance Risk Assessment Committee does suggest that the EMA is taking a more proactive approach to pharmacovigilance and risk management, and it is to be commended for this.53 However, the report’s authors looked at the effect of specific practices on process measures rather than on patient safety, so it seems premature to draw strong conclusions about the success of the legislation.
Moreover, when regulators have failed to act on post-marketing evidence that drugs are ineffective or can no longer be safely used this is usually not because they lack the tools to act but reflects, rather, a culture in which regulators seem to have a high tolerance for drug harms, even when drug benefits are marginal or non-existent.54 5556
Public risk, private reward?
The EMA claims that early access to drugs will benefit patients and that adaptive pathways will not change regulatory standards or the requirement to show a positive benefit-harm balance to obtain marketing authorisation.7 However, accelerated approval and conditional marketing have already lowered the regulatory bar. And when evidence comes from uncontrolled studies of poorly predictive surrogate endpoints in a small number of patients, the EMA’s determination that benefits outweigh harms “may be no more than a guess.”3 Whether guesswork is an acceptable basis for approving new drugs is a question of fundamental importance to citizens and public health. Moreover, when potentially unsafe, ineffective, or marginally effective drugs are prescribed and reimbursed under the mistaken belief that they are of great clinical utility, the costs to patients and the public are high—in terms of the opportunity costs of wasted healthcare resources and the costs to patients who have been exposed to futile or dangerous treatments.
We would argue that neither a scientific nor an ethical case for adaptive pathways has been made. Whether—and, if so, when—the benefits to some patients of early approval might outweigh the risks to more patients in the future and to public health needs urgent discussion. This discussion must consider the lessons over more than 20 years of expedited drug development and review in the US and the EU and known examples of regulators failing to protect public health.57
European regulators are under pressure from the drug industry to accelerate approval of new drugs
One proposed mechanism, adaptive pathways, relies on preliminary clinical data, surrogate outcomes, and observational studies—all of which risk reaching incorrect conclusions about the benefit-harm balance of new drugs
Early approval assumes that reliable new data on benefits and harms will ensue rapidly once a drug comes to market and that early widespread use can be reversed, but the evidence does not support these assumptions
The European Medicines Agency’s new report on its adaptive pathways pilot is disappointing and leaves important questions unanswered. A transparent and inclusive discussion about the wisdom of adaptive pathways is urgently needed
Contributors and sources: This paper arose from a meeting in Brussels in April 2016 on adaptive pathways and access to drugs, convened by the European Public Health Alliance. CD and JL have written extensively on medicines regulation. TJ (Cochrane Collaboration) is interested in the use of regulatory evidence for research synthesis, and PG (director of the Nordic Cochrane Centre) is an expert on research methodology. MM until recently chaired a committee of the Open Society Foundations overseeing a programme on access to medicines. The initial drafts were written by CD and MM, with input from JL, TJ, and PG. CD is guarantor.
We have read and understood BMJ’s policy on declaration of interests and declare the following interests: TJ was a recipient of a UK National Institute for Health Research grant for a Cochrane review of neuraminidase inhibitors for influenza. TJ has acted in various capacities in influenza related litigation cases. In 2014-16 TJ was a member of three advisory boards for Boehringer Ingelheim and is holder of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial of a flu vaccine. CD and JL are members of Health Action International. MM was a member of the European Commission’s Expert Panel on Investing in Health until May 2016 and chaired the Global Health Advisory Committee of the Open Society Foundations until November 2015.
Provenance and peer review: Not commissioned; externally peer reviewed.
- Specialists attack drug agency’s fast track approval scheme
- Letting post-marketing bridge the evidence gap: the case of orphan drugs
- German body calls for pause in European plan for fast track drug approval
- The European Medicines Agency is still too close to industry
- Comparison of rates of safety issues and reporting of trial outcomes for medical devices approved in the European Union and United States: cohort study