Out of Sight, Out of Mind, Out of the Peer-Reviewed Literature
ABSTRACT
Importance Every year, the US Food and Drug Administration (FDA) inspects several hundred clinical sites performing biomedical research on human participants and occasionally finds evidence of substantial departures from good clinical practice and research misconduct. However, the FDA has no systematic method of communicating these findings to the scientific community, leaving open the possibility that research misconduct detected by a government agency goes unremarked in the peer-reviewed literature.
Objectives To identify published clinical trials in which an FDA inspection found significant evidence of objectionable conditions or practices, to describe violations, and to determine whether the violations are mentioned in the peer-reviewed literature.
Design and Setting Cross-sectional analysis of publicly available documents, dated from January 1, 1998, to September 30, 2013, describing FDA inspections of clinical trial sites in which significant evidence of objectionable conditions or practices was found.
Main Outcomes and Measures For each inspection document that could be linked to a specific published clinical trial, the main measure was a yes/no determination of whether there was mention in the peer-reviewed literature of problems the FDA had identified.
Results Fifty-seven published clinical trials were identified for which an FDA inspection of a trial site had found significant evidence of 1 or more of the following problems: falsification or submission of false information, 22 trials (39%); problems with adverse events reporting, 14 trials (25%); protocol violations, 42 trials (74%); inadequate or inaccurate recordkeeping, 35 trials (61%); failure to protect the safety of patients and/or issues with oversight or informed consent, 30 trials (53%); and violations not otherwise categorized, 20 trials (35%). Only 3 of the 78 publications (4%) that resulted from trials in which the FDA found significant violations mentioned the objectionable conditions or practices found during the inspection. No corrections, retractions, expressions of concern, or other comments acknowledging the key issues identified by the inspection were subsequently published.
Conclusions and Relevance When the FDA finds significant departures from good clinical practice, those findings are seldom reflected in the peer-reviewed literature, even when there is evidence of data fabrication or other forms of research misconduct.
INTRODUCTION
As part of the drug approval process, the US Food and Drug Administration (FDA) regularly inspects clinical trial sites involved in FDA-regulated research to determine the degree to which these sites conform to regulations. The FDA regulations intend to ensure, among other things, that scientists adhere to good clinical practice and that they protect the rights of human participants. Such inspections often yield useful information about the reliability and quality of the clinical data produced at a clinical trial site.
An FDA inspection typically involves officials visiting a trial site and auditing the records kept at that site. During the course of several days, the inspectors verify that, among other things, the investigators adhered to the trial protocol, the participants had given informed consent, and the research had been duly approved by an institutional review board. The inspectors may also audit the data comparing, for example, an investigator’s progress notes in hospital records with data reported to the study sponsor to ensure that there are no irregularities.1
The FDA classifies its inspections in 1 of 3 ways, depending on the gravity of violations found. No action indicated indicates that there were no substantial violations. Voluntary action indicated means that inspectors have found violations of good clinical practice, but the nature and extent of those problems are not serious enough to require sanction. The most severe classification, official action indicated (OAI), is reserved for cases in which the inspection identified objectionable conditions or practices significant enough to warrant regulatory action.2 In the 2013 fiscal year, approximately 2% of the 644 inspections of trial sites carried out by the FDA’s Bioresearch Monitoring organization were classified as OAI.3 The nature and extent of the OAI violations, which include submission of false information and failure to report adverse events to the appropriate bodies, often raise questions about the validity and accuracy of the clinical trial site’s data. Consequently, the FDA typically excludes data from a site that received an OAI when judging the safety or efficacy of a new drug.
The goals of the present study were to identify publications describing clinical trials that the FDA had determined had an OAI violation, to describe the violations, and to determine whether the published article or any subsequent correction acknowledged the violation.
METHODS
A multipronged approach was used to identify clinical trials with an OAI violation (Figure). The process began by attempting to identify clinical trial sites and principal investigators who had received an OAI violation. Although there is no public canonical list of OAI inspections, the FDA maintains a database containing the results of some of its inspections.4 In July 2012, the database was searched for clinical investigators who had received an OAI. To obtain documents (form 483s and Establishment Inspection Reports) that provide details about a given inspection, Freedom of Information Act requests were made to the FDA. The request yielded documents related to 20 OAI-rated inspections, all dated before August 8, 2012, when the Freedom of Information Act request was submitted.
Relevant Clinical Trials
Identification of relevant clinical trials linked to specific violations described in an official action indicated (OAI)–rated inspection. Between October 7 and December 9, 2013, all warning letters that were issued to a clinical investigator after January 1, 1998, as well as all Notices of Disqualification Proceedings and Opportunity to Explain (NIDPOEs), Notices of Opportunity for a Hearing (NOOHs), and disbarment decisions that were on the US Food and Drug Administration’s website, were reviewed. FOIA indicates Freedom of Information Act.
To supplement the data obtained from the searches of the FDA database, Google searches of thehttp://www.FDA.gov domain were performed. The most effective searches used combinations of phrases and their variants that were contained in documents describing OAI-rated inspections of clinical sites (eg,classified as OAI, inspectionsummary, received an OAI, inspected, OAI classification, and inspection). This strategy yielded documents related to 21 OAI-rated inspections.
The best source of documentation of OAI-rated inspections came from instances in which the FDA took regulatory action against clinical investigators. Such actions occur only when the failure to adhere to research regulations is considered particularly grave. In such cases, the FDA often issues 1 or more documents that detail the problems found in an inspection: warning letter, Notice of Disqualification Proceedings and Opportunity to Explain, Notice of Opportunity for Hearing, and official notification of disbarment or sanctions. Between October 7 and December 9, 2013, all warning letters that were issued to a clinical investigator after January 1, 1998 (letters regarding 298 inspections), as well as all Notices of Disqualification Proceedings, Notices of Opportunity for Hearing, and disbarment decisions that were on the FDA’s website (documents concerning 82 inspections), were reviewed.
The 3 methods of search yielded 421 OAI-rated inspections. We then attempted to link the sites and investigators described in the related inspection documents to specific clinical trials. Heavy redactions in most of these documents prevented this linkage in most cases (eAppendix in the Supplement). However, whenever we were able to identify a clinical trial that received an OAI finding, we searched the peer-reviewed literature for any resultant publications. If such publications were found, they were independently reviewed by the author and by a second reader with the goal of identifying any written acknowledgment about the violations identified by the FDA. Agreement between the 2 reviewers was high (κ = 0.85). One article noted that data “were either missing, or were considered unreliable by the investigator due to problems collecting accurate data.”5(p3) The 2 reviewers disagreed about whether the unreliability might have been an oblique reference to problems found during an inspection. However, the inspection documents6 detailed failures to obtain informed consent, falsified information, misreporting the dosage of drugs for at least 7 patients, and failure to record data on 10 patients. After discussion, the reviewers concurred that the language in the article was not an acknowledgment of the inspection findings.
PubMed and Thomson-Reuters’ Web of Science were searched for any corrections, retractions, expressions of concern, or other comments in which those violations might have been aired after the article was published. Food and Drug Administration–related documents obtained in this investigation are available.7
RESULTS
General Findings
There were approximately 600 clinical trials mentioned in the documents we gathered; owing to redactions, most of these trials could not be identified. However, in some cases, key information was not redacted from the documents, allowing us to identify 101 trials in which at least one clinical trial site received an OAI grade on an inspection (Figure).
Of those 101 clinical trials, we identified 68 for which results had been published in the peer-reviewed literature, resulting in a total of 95 publications. For 11 of the clinical trials that had been published, the documents were not sufficiently detailed for us to prove that the violations described in the document were specific to the trial in question, so they were excluded from the primary analysis (Table 1).12– 16,18– 21,24– 26,28,29,33,34 For example, 1 warning letter8 and 1 Notice of Disqualification Proceedings and Opportunity to Explain9 detailed violations in 7 clinical trials of stem cell therapies, which then resulted in 4 publications.10,35– 37 Because of the redactions in those documents, there was ambiguity about which of the 7 trials was linked to which violation described in the documents. It was possible to tie specific violations to only 3 of the 4 published trials38– 40; the fourth trial41 was therefore excluded from analysis.
For each of the 57 remaining trials, 1 or more FDA inspections of a trial site had uncovered evidence of significant departures from good clinical practice, such as underreporting of adverse events, violations of protocol, violations of recruitment guidelines, and various forms of scientific misconduct.
In 22 of these trials (39%), the FDA cited researchers for falsification or submission of false information; in 14 (25%), for problems with adverse events reporting; in 42 (74%), for failure to follow the investigational plan or other violations of protocol; in 35 (61%), for inadequate or inaccurate recordkeeping; in 30 (53%), for failure to protect the safety, rights, and welfare of patients or issues with informed consent or institutional review board oversight; and in 20 (35%), for violations not otherwise categorized. Examples of uncategorized violations include cases in which the investigators used experimental compounds in patients not enrolled in trials, delegated tasks to unauthorized personnel, or otherwise failed to supervise clinical investigations properly.
The 57 clinical trials in our analysis resulted in 78 articles published in the peer-reviewed literature (Table 2). Of these 78 articles, only 3 publications (4%) included any mention of the FDA inspection violations despite the fact that for 59 of those 78 articles (76%), the inspection was completed at least 6 months before the article was published. Researchers are usually given a form 483 within a day of the inspection, with the form detailing any problems found by the inspector.
For the 3 articles that mentioned the inspection violations, 1 stated that 1 of the trial sites “was found to have allegedly entered fraudulent data and was dropped from participation.”121(p390) (References 76 through 184 are listed in the eReferences in the Supplement.) The research misconduct involved falsified laboratory test results in a phlebotomy trial. In the second instance, the article noted that the data from 1 clinical trial site were excluded owing to “protocol adherence and data quality issues.”111(p78) According to the FDA documents, the researcher apparently eliminated the blinding in a randomized protocol so she “could control drug treatment assignments”168(p7) of her patients; she was also cited for falsification of data in 2 other protocols. In the third instance, an article explained that data from several patients were excluded from the efficacy analysis because “site monitoring raised questions in regard to certain data at 1 study site.”65(p431) The FDA documents64 allege that none of the individuals enrolled at 1 study site had met the inclusion criteria and that the responsible researcher had fabricated chest radiographs of participants and committed other forms of misconduct.
In no other instance did we find acknowledgment of problems found during an FDA inspection. In addition, we were unable to identify any corrections, retractions, comments, or notifications of concern published after FDA identification of the violations.
Examples of Unreported Violations
To illustrate the importance of the unreported inspection violations, 4 cases cut examples are provided herein.
Case 1
A publication describing a stem cell trial in 26 patients with ischemic limbs stated that “all patients recognized and were aware of major clinical improvements in the treated (more ischemic) leg, despite no significant clinical changes in the control (less ischemic) leg.”37(p381) However, an FDA document169revealed that 1 patient had a foot amputated 2 weeks after administration of the stem cells. We found no correction or retraction.
Case 2
Eight of 16 FDA inspections of sites involved in a clinical trial of rivaroxaban,170 a novel anticoagulant, had been rated OAI. These inspections had uncovered evidence of various transgressions, such as “systemic discarding of medical records,”171(p3) unauthorized unblinding, falsification, and “concerns regarding improprieties in randomization.”172(p211) Consequently, the entire study, RECORD 4 (Regulation of Coagulation in Orthopedic Surgery to Prevent Deep-Venous Thrombosis and Pulmonary Embolism 4), was deemed unreliable by the FDA.171 These problems are not mentioned in the article describing the study’s results142 or in other publications associated with the trial.144,145
Case 3
A researcher was caught falsifying documents in a number of trials,173-176 in part because those falsifications led to the death of a patient undergoing treatment in a clinical trial comparing 2 chemotherapy regimens. The researcher had falsified laboratory test results to hide the patient’s impaired kidney and liver function, and the first dose of the treatment proved to be fatal. The researcher pleaded guilty to fraud and criminally negligent homicide and was sentenced to 71 months in prison. Although this episode is described in detail in FDA documents11,67 as well as court documents,177 none of the publications in the peer-reviewed literature associated with the chemotherapy study in which the patient died70– 72,178 have any mention of the falsification, fraud, or homicide. The publications associated with 2 of the 3 other studies for which the researcher falsified documents also do not report on the violations.68,73
Case 4
A clinical site in China taking part in a large trial of apixaban, a novel anticoagulant, had apparently altered patient records. If one were to exclude the data from the patients at that site, the claim of a statistically significant mortality benefit disappears.179 For this reason, among others, the FDA wrestled with whether it was appropriate to allow the manufacturer to claim a mortality benefit. None of this discussion appears in the literature. The claim for the mortality benefit, which has appeared in the literature since 2011,50,52,180consistently relies on the full data set, including data from the site at which the research misconduct allegedly occurred. This is true even for an article that was published52 nearly 18 months after the alleged research misconduct was discovered. In addition, the mortality benefit analysis of the FDA-approved drug label as of August 31, 2014, is also based on the full data set181 despite a recommendation from the FDA’s Office of Scientific Investigation that data from not just the problematic site but 23 additional suspect Chinese sites be excluded.182 Despite the fraudulent data, when all the suspect Chinese sites are excluded rather than just the one at which the evidence of alleged research misconduct was found, the mortality benefit becomes statistically significant at the P = .05 level once again.182 One FDA analyst, commenting on the “data quality issues” in this clinical trial, complained about the agency’s lack of transparency and poor handling of evidence of problems with trial data: Some of the responsibility for the data quality issues rests with us, the FDA: We have approved drugs ignoring similar data quality issues, granting superiority claims, and not discussing in the labels the data quality issues. We must stop doing this.182(p19)
DISCUSSION
Our study has some limitations. The data are descriptive rather than quantitative. We do not know how many publications derive from trials that received an OAI finding or whether a full sample of such publications would show a higher or lower rate of acknowledging inspection violations. Our search strategy was limited by the information publicly available. For example, the FDA database of clinical inspections is infrequently updated. In addition, documents from certain time periods and certain regions of the country were harder to locate than others, indicating that our search was biased. Moreover, the records that the FDA makes available are incomplete and often heavily redacted. The nature of the redactions—and thus, our likelihood of linking a given document to a specific clinical trial—also varied depending on which FDA officer was performing the redaction and the year in which the redactions were performed. All of these limitations prevent generalization of our findings to the entire population of clinical trials. Finally, problems uncovered during inspections of clinical trial sites represent only a fraction of the departures from good clinical practice of which the FDA becomes aware. For example, the FDA sometimes learns of departures from good clinical practice through communications with and inspections of organizations sponsoring and responsible for conducting clinical trials; these instances were not part of our investigation.
Even though several inspection documents reviewed here described major violations of good clinical practice, including allegations of fabrication and other forms of research misconduct, it was rare that objectionable conditions or practices uncovered by the FDA were reflected in the peer-reviewed literature.
Of course, not all violations are of equal severity. When a clinical trial site receives an OAI, it does not mean that the violations need be acknowledged in an article or, if discovered after publication of the study, warrant a correction. Even in the case of data fabrication, there is occasional ambiguity. For example, in a clinical trial183 of a drug administered via intravitreal injection, a researcher apparently fabricated images of patients’ retinas. Although one might argue that an article in which those images were used as data128might require a correction, it is unclear whether another article that addresses the study’s infection rates associated with intravitreal injections,129 without relying on the retinal images to support the findings, would be similarly affected. Furthermore, data are sometimes excluded from peer-reviewed publications, occasionally without explanation. Consequently, in some of the articles (Table 2), tainted data might be handled properly, even if not explicitly remarked upon in the publication; it was not possible in the present study to determine how often this occurred.
CONCLUSIONS
The findings presented in this study should give us pause. This investigation has found numerous studies for which the FDA determined there was significant evidence of fraudulent or otherwise problematic data. Such issues raise questions about the integrity of a clinical trial, and mention of these problems is missing from the relevant peer-reviewed literature. The FDA does not typically notify journals when a site participating in a published clinical trial receives an OAI inspection, nor does it generally make any announcement intended to alert the public about the research misconduct that it finds. The documents the agency discloses tend to be heavily redacted. As a result, it is usually very difficult, or even impossible, to determine which published clinical trials are implicated by the FDA’s allegations of research misconduct.
The FDA has legal as well as ethical responsibilities regarding the scientific misconduct it finds during its inspections. When the agency withholds the identity of a clinical trial affected by scientific misconduct, it does so because it considers the identity to be confidential commercial information, which it feels bound to protect.184 However, failing to notify the medical or scientific communities about allegations of serious research misconduct in clinical trials is incompatible with the FDA’s mission to protect the public health. Such allegations are relevant to include in the peer-reviewed literature on which physicians and other medical researchers rely to help them choose treatments that they offer to patients and other research participants.
To better serve the public health, the FDA should make unredacted information about its findings of research misconduct more readily available. The agency should make sure that any substantial evidence of misconduct is available to editors and readers of the scientific literature. One possible mechanism for this would be to use the national clinical trials database: any OAI inspection affecting a trial site should be promptly noted at http://www.clinicaltrials.gov. The FDA should also create a website or a publicly available database that lists all OAI-rated inspections of clinical sites and provides links to copies of the relevant, unredacted, inspection-related documents.
The FDA should be more transparent about its findings of research misconduct; however, most of the burden for ensuring the integrity of the research in the peer-reviewed literature falls to the authors of the articles submitted to peer-reviewed journals. Currently, there is no formal requirement for authors seeking to publish clinical trial data to disclose any adverse findings noted during FDA inspections. Journals should require that any such findings be disclosed. Voluntary disclosures are never foolproof, but, as with conflict-of-interest statements, requiring authors and journals to be forthcoming about significant departures from good clinical practice will help raise the standard for the reporting of research toward greater transparency.
ARTICLE INFORMATION
Accepted for Publication: December 3, 2014.
Published Online: February 9, 2015. doi:10.1001/jamainternmed.2014.7774.
Conflict of Interest Disclosures: None reported.
Funding/Support: All financial and material support for this study was provided by New York University.
Role of the Funder/Sponsor: New York University had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: Graduate students from New York University’s Science, Health, and Environmental Reporting and Business and Economic Reporting programs assisted with data collection and analysis: Lydia Anderson, MA, Rebecca Cudmore, MA, Kathryn Free, MA, Chelsea Harvey, MA, Joshua Krisch, MA, Jamie Lee, MA, Sarah Lewin, MA, Claire Maldarelli, MA, Elizabeth Newbern, MA, Hannah Newman, MA, Amy Nordrum, MA, Alexandra Ossola, MA, Neel Patel, MA, Krystnell Storr, MA, and Manasi Vaidya MA. Rob Garver and Tom Detzel of ProPublica contributed to useful discussions. Ms Anderson was compensated from the author’s research account for her role as the second reader of medical articles. No other individuals were compensated for their contributions.
REFERENCES
Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article
PubMed | Link to Article