Redefining the ‘E’ in EBM

The philosopher of science, Thomas Kuhn,1 would probably have called our reliance on biomedical journal trial evidence a paradigm. It has served us well, allowing the building of the philosophical2 and practical3 backbone of evidence-based healthcare. However, like all paradigms sooner or later it has begun to creak. Our reliance on journal articles needs a redefinition, if not a shift. In the last decade,4 evidence has accumulated,5 across a spectrum of different interventions,6 that journal publications7 cannot be trusted. Article reports of clinical trials suffer from a grave illness which is curable, but needs a concerted approach to prevent the growing threat of reporting bias.8 When some of us started looking at the alternative sources of evidence for our Cochrane review of neuraminidase inhibitors9 for influenza nearly a decade ago, we discovered that below every 10-page trial report lies a far deeper and more complex web of data and information needing attention. That is, if the trial was published in the first place.

The first problem is sheer bulk. For every page of journal article, there may be up to 8000 pages of regulatory data on the same clinical trial.10 We call it a compression factor.

The next problem is bias. We reasoned that even the most faithful servant of evidence would not be able to publish a 10-pager based on a regulatory report without a radical selection of information and data. As we have no idea what the criteria for choosing which plum to publish are, this introduces unfathomable bias. Sometimes the bias is so bad that it distorts single trial reports, but it also distorts the findings of systematic reviews, as our neuraminidase inhibitor story shows.9

Evidence of distortion in the results of research is now overwhelming, and it mainly comes from studies comparing journal articles with other sources of information.11 These sources include register entries and different types of regulatory data now on offer, from regulators’ reports to clinical study reports (the regulatory equivalent of a journal publication), to overviews of whole trial programmes. Secreted and confidential up to a few years ago, clinical study reports are now coming to light from regulators and industry sources with seeming unstoppable momentum. Latest to look at releasing clinical study reports is the mighty Food and Drug Administration.12 The catalyst to this change was the Nordic Cochrane Centre’s dogged insistence of access and the European Union Ombudsman’s support,13 which ultimately led the European Medicines Agency to change its policies.14 15

The thing about clinical study reports is not just that they provide missing information on such pedestrian details like what’s in a placebo or even what it looks like.9 They also provide sufficient data for carrying out stratified analyses, and more often report patient-relevant outcomes lay bare the shabby way in which we currently look at and analyse harms.6 16 17

So, should we ignore evidence from journal articles? If steps are not taken urgently to address the situation, then ‘probably’ would be our answer. By the law of Garbage In Garbage Out, whatever we produce in our reviews will be systematically assembled and synthesised garbage with a nice Cochrane logo on it. One major problem is our ignorance of the presence of garbage, as its invisibility makes its distortions credible and impossible to check. This is how some of us happily signed off a Cochrane review with findings which had been completely and invisibly subverted18 by reporting bias.

Garbage, however, is often neutral, but some articles are not neutral. They can be carefully contrived pieces of marketing,19 part of a global jigsaw. We can only guess at what their purpose is and what the true results are. We need to stop producing reviews based on articles (or at least solely on articles) and seriously and urgently look at drawing from data sources which allow alternative explanations and conclusions from the data, because the data set is detailed and near-complete.

How do we redefine the ‘E’ paradigm? We recently published an index of all prospective comparative studies on human papillomavirus vaccines we could find.20 The index is made of study IDs and (where possible) a description of their content. Predictably, only 62% of the completed studies had been published and none of the non-industry-sponsored studies’ underlying documentation was available. Ninety-five per cent of studies were listed on regulatory or industry trial registers or journal publication databases, but only 48% of completed studies listed on ClinicalTrials.gov had study results posted. It took us 3 months to complete the index, starting from correspondence with regulators and adding studies by identifying them by cross-referencing from several other sources: industry, registers and other regulatory documents. Not as straightforward as an electronic database publication search. But that’s the point. It is more difficult and complex because you are getting close to what really happened in the trials and how they were really run. Your labour is rewarded with a near-complete overview of the development of an important intervention often given to millions of healthy folk or patients around the world. Clinical study reports are still commercial documents, but they are written for the wise and cannot, and should not, omit anything (although they may have internal inconsistencies21). If there are distortions, they may be approved or overlooked by regulators.

Indexing, at least for now, is more resource-intensive than electronic database searches. The two can be run together and perhaps should be combined. But looking for regulatory data and compiling an index will give us a very good idea of what we are missing and what the limits of our reviews are.

Ethics and Evidence both begin with ‘E’.