It seems to have become a popular sport to try to exaggerate problems with disclosure of clinical trials, and to pretend that the problem of “secret hidden trials” is far worse than it really is. Perhaps the most prominent example of this is the All Trials campaign’s favourite statistic that “only half of all clinical trials have ever been published”, which I’ve debunked before. But a new paper was published last month which has given fresh material to the conspiracy theorists.

The paper in question was published in BMJ Open by Jennifer Miller and colleagues. They looked at 15 of the 48 drugs approved by the FDA in 2012. It’s not entirely clear to me why they focused on this particular subgroup: they state that they focused on large companies because they represented the majority of new drug applications. Now I’m no mathematician, but I have picked up some of the basics of maths in my career as a statistician, and I’m pretty sure that 15 out of 48 isn’t a majority. Remember that we are dealing with a subgroup analysis here: I think it might be important, and I’ll come back to it later.

Anyway, for each of those 15 drugs, Miller et al looked at the trials that had been used for the drug application, and then determined whether the trials had been registered and whether the results had been disclosed. They found that a median (per drug) of 65% of trials had been disclosed and 57% had been registered.

This study drew the kinds of responses you might expect from the usual suspects, describing the results as “inexcusable” and “appalling”.

(Note that both of those tweets imply that only 15 drugs were approved by the FDA in 2012, and don’t mention that it was a subgroup analysis from the 48 drugs that were really approved that year.)

The story was picked up in the media as well. “How pharma keeps a trove of drug trials out of public view” was how the Washington Post covered it. The Scientist obviously decided that even 65% disclosure wasn’t sensational enough, and reported “just one-third of the clinical trials that ought to have been reported by the trial sponsors were indeed published”.

But as you have probably guessed by now, when you start to look below the surface, some of these figures are not quite as they seem.

Let’s start with the figures for trial registration (the practice of making the design a trial publicly available before it starts, which makes it harder to hide negative results or pretend that secondary outcomes were really primary). Trial registration is a fairly recent phenomenon. It only really came into being in the early 2000s, and did not become mandatory until 2007. Bear in mind that drugs take many years to develop, so some of the early trials done for drugs that were licensed in 2012 would have been done many years earlier, perhaps before the investigators had even heard of trial registration, and certainly before it was mandatory. So it’s not surprising that such old studies had not been prospectively registered.

Happily, Miller et al reported a separate analysis of those trials that were subject to mandatory registration. In that analysis, the median percentage of registered trials increased from 57% to 100%.

So I think a reasonable conclusion might be that mandatory trial registration has been successful in ensuring that trials are now being registered. I wouldn’t call that “inexcusable” or “appalling”. I’d call that a splendid sign of progress in making research more transparent.

So what about the statistic that only 65% of the trials disclosed results? That’s still bad, right?

Again, it’s a bit more complicated than that.

First, it’s quite important to look at how the results break down by phase of trial. It is noteworthy that the vast majority of the unpublished studies were phase I studies. These are typically small scale trials in healthy volunteers which are done to determine whether it is worth developing the drug further in clinical trials in patients. While I do not dispute for a minute that phase I trials should be disclosed, they are actually of rather little relevance to prescribers. If we are going to make the argument that clinical trials should be disclosed so that prescribers can see the evidence on what those drugs do to patients, then the important thing is that trials in patients should be published. Trials in healthy volunteers, while they should also be published in an ideal world, are a lower priority.

So what about the phase III trials? Phase III trials are the important ones, usually randomised controlled trials in large numbers of patients, which tell you whether the drug works and what its side effects are like. Miller et al report that 20% of drugs had at least 1 undisclosed phase III trial. That’s an interesting way of framing it. Another way of putting is is that 80% of the drugs had every single one of their phase III trials in the public domain. I think that suggests that trial disclosure is working rather well, don’t you? Unfortunately, the way Miller et al present their data doesn’t allow the overall percentage disclosure of phase III trials to be determined, and my request to the authors to share their data has so far gone unheeded (of which more below), but it is clearly substantially higher than 80%. Obviously anything less than 100% still has room for improvement, but the scare stories about a third of trials being hidden clearly don’t stack up.

And talking of trials being “hidden”, that is rather emotive language to describe what may simply be small delays in publication. Miller et al applied a cutoff date of 1 February 2014 in their analysis, and if results were not disclosed by that date then they considered them to be not disclosed. Now of course results should be disclosed promptly, and if it takes a bit longer, then that is a problem, but it is really not the same thing as claiming that results are being “kept secret”. Just out of interest, I checked on one of the drugs that seemed to have a particularly low rate of disclosure. According to Miller et al, the application for Perjeta was based on 12 trials, and only 8% had results reported on clinicaltrials.gov. That means they considered only one of them to have been reported. According to the FDA’s medical review (see page 29), 17 trials were submitted, not 12, which makes you wonder how thorough Miller et al’s quality control was. Of those 17 trials, 14 had been disclosed on clinicaltrials.gov when I looked. So had Miller et al used a different cut-off date, they would have found 82% of trials with results posted, not 8%.

I would like to be able to tell you more about the lower disclosure rates for phase I trials. Phase I trials are done early in a drug’s development, and so the phase I trials included in this study would typically have been done many years ago. It is possible that the lower publication rate for phase I trials is because phase I trials are intrinsically less likely to be published than trials in patients, but it is also possible that it is simply a function of when they were done. We know that publication rates have been improving over recent years, and it is possible that the publication rate for phase I trials done a decade or more ago is not representative of the situation today.

Sadly, I can’t tell you more about that. To distinguish between those possibilities, I would need to see Miller et al’s raw data. I did email them to ask for their raw data, and they emailed back to say how much they support transparency and data sharing, but haven’t actually sent me their data. It’s not entirely clear to me whether that’s because they have simply been too busy to send it or whether they are only in favour of transparency if other people have to do it, but if they do send the data subsequently I’ll be sure to post an update.

The other problem here is that, as I mentioned earlier, we are looking at a subgroup analysis. I think this may be important, as another study that looked at disclosure of drugs approved in 2012 found very different results. Rawal and Deane looked at drugs approved by the EMA in 2012, and found that 92% of the relevant trials had been disclosed. Again, it’s less than 100%, and so not good enough, but it certainly shows that things are moving in the right direction. And it’s a lot higher than the 65% that Miller et al found.

Why might these studies have come to such different results? Well, they are not looking at the same drugs. Not all of the drugs approved by the FDA in 2012 were approved by the EMA the same year. 48 drugs were approved by the FDA, and 23 by the EMA. Only 11 drugs were common to both agencies, and only 3 of those 11 drugs were included in Miller et al’s analysis. Perhaps the 15 drugs selected by Miller et al were not a representative sample of all 48 drugs approved by the FDA. It would be interesting to repeat Miller et al’s analysis with all 48 of the drugs approved by the FDA to see if the findings were similar, although I doubt that anyone will ever do that.

But personally, I would probably consider a study that looked at all eligible trials more reliable than one that chose an arbitrary subset, so I suspect that 92% is a more accurate figure for trial disclosure for drugs approved in 2012 than 65%.

Are 100% of clinical trials being disclosed? No, and this study confirms that. But it also shows that we are getting pretty close, at least for the trials most relevant for prescribers. Until 100% of trials are disclosed, there is still work to do, but things are not nearly as bad as the doom-mongers would have you believe. Transparency of clinical trial reporting is vastly better than it used to be, and don’t let anyone tell you otherwise.

Update 23 January 2016:

I have still not received the raw data for this study, more than 2 months after I asked for it. I think it is safe to assume that I’m not going to get it now. That’s disappointing, especially from authors who write in support of transparency.