On transparency in science: lessons from the Pandemrix vaccine and 'partial' evidence-based practice

Minus any hype or sweeping generalisation, I want to draw your attention to a feature article recently published by Peter Doshi [1] titled: "Pandemrix vaccine: why was the public not told of early warning signs?" complete with some media interest (see here).

The feature is based around the vaccines developed to counter the 2009 flu pandemic that starred the H1N1 influenza virus which was sweeping across the globe at the time. The 2009 flu pandemic was potentially serious insofar as its infection pattern and fatalities linked to infection although in reality did not turn out to be quite the 'global killer' that was prophesied. The rush to bring to market a vaccine to protect lives and counter the rise of H1N1 also brought with it controversy from several perspectives [2]; some later talking points seemingly stemming from the inclusion of a specific adjuvant - AS03 - in some vaccine preparations, and data suggestive of a possible correlation between onset of narcolepsy and use of certain flu vaccines containing the AS03 adjuvant (see here). Narcolepsy - a sleep disorder - was one potential adverse side-effect talked about, but other labels/conditions were also the source of some inquiry too (see here). On the basis of the emerging published data, court cases ensued, some of which have apparently unearthed some "internal reports" suggesting that some manufacturers "were aware of other serious adverse events logged" in relation to certain H1N1 vaccines...

The Doshi feature goes through some of these 'revelations' that did not seem to have been made public at the time. He talks about an 'undisclosed problem' whereby one vaccine in particular, Pandemrix, seemed to have a greater frequency of various adverse events reported according to those internal company reports. Most worrying are the reports of 'fatal outcome' across different vaccine preparations, particularly Pandemrix. He also writes: "And the raw numbers of adverse events were not small" highlighting how in one of the latest internal reports dated March 2010 seen by himself (or at least, the British Medical Journal) over 5000 reports of "serious adverse events for Pandemrix" had been received by the company behind the vaccine.

It's timely that the Doshi feature appears now at the time of writing this post, just days after the Peter Gøtzsche - Cochrane drama (see here) and at the same time as other relevant pieces have been published (see here and see here [3]). The similarities around questions of transparency in science (particularly industry-derived science that has a public health aspect attached to it) are very noticeable. As is also the question of 'how much detail should the public be provided with?' when it comes to science in general and then specifically, science and science communication linked to the potential cost-benefit ratio of particular medicines or medical preparations. It's interesting too that Doshi also references some past debate in Germany [4] about the 2009 flu pandemic where it was apparently reported that "the German interior ministry rejected accusations that the ministry had ordered a less risky vaccine for top officials" in light of some newspaper headlines published at the time. 'All animals are equal, but some animals are more equal than others' perhaps springs to mind...

I don't think the issue of less than full transparency in science and the question 'how much detail should the public be provided with' are going to go away any time soon. Such issues strike to the heart of the matter of how peer-reviewed published science is seemingly only one part of a much bigger picture when it comes to the search for a complete truth. One wonders for example, how many other 'internal reports' lie in [research] file cabinets around the world that could conceivably influence evidence-based decisions on the effectiveness and safety of various medicines and other interventions indicated for various conditions? Without such documents ever seeing the cold light of day, how can science ever truly be said to be complete, transparent or indeed settled?

Such a 'transparency' situation also makes initiatives like OpenTrials - calling for all research to be more 'discover-able' and 'traceable' - all the more vital and important. Issues such as the requirement for pre-registration of all studies via databases such as ClinicalTrials.gov and importantly, the timely release and publication of results (see here) are pressing features. Study pre-registration (especially prospective registration) in particular, should perhaps be pushed a lot more than it currently is, and shouldn't just be reserved for big studies of drugs and medicines either.

Science is great and is getting better all the time. But it still has lots of issues and problems to overcome. A lack of research transparency seems to be a particularly important issue that, whilst starting to receive some much required attention, will if present, always mean that evidence-based practice really should be re-branded as 'partial evidence-based practice'. And that has lots of implications...

To close, given the subject matter included in the Doshi paper, I want to end with a reminder about not making sweeping generalisations. At the same time I'd also champion the words included in another recent article by Jørgensen and colleagues [5] (yes, also including Peter Gøtzsche as an author) responding to the idea that public confidence in public health initiatives such as vaccination may be "undermined" by even having such discussions in public. To quote: "Debates over sources of evidence must take place in public, especially when public health interventions are at stake." This is again, with the idea that science needs transparency - even better in the peer-reviewed domain - and closed door discussions rarely aid such transparency.

----------

[1] Doshi P. Pandemrix vaccine: why was the public not told of early warning signs? BMJ 2018;362:k3948

[2] Doshi P. Influenza: marketing vaccine by marketing disease. BMJ 2013;346:f3037.

[3] Jørgensen L. et al. Challenges of independent assessment of potential harms of HPV vaccines. BMJ 2018;362:k3694

[4] Stafford N. Only 12% of Germans say they will have H1N1 vaccine after row blows up over safety of adjuvants. BMJ 2009;339:b4335.

[5] Jørgensen L. et al. Response to Cochrane editors: Jørgensen, Gøtzsche and Jefferson. BMJ EBM Spotlight. 2018. Sept 23.

----------

Exercise may protect against the development of depression

"Available evidence supports the notion that physical activity can confer protection against the emergence of depression regardless of age and geographical region."

So said the findings reported by Felipe Schuch and colleagues [1] who undertook a review of the existing peer-reviewed research literature looking at whether physical activity a.k.a exercise *might* provide some important protection against the development of depression. Such a review took place in the context that NHS Choices, the go-to place for health and medical information here in Blighty, already has an entry called 'exercise for depression'.

The authors - many of whom are quite recognisable names in the area of science looking at the physical and the mental coexisting together - located almost 50 studies labelled as prospective cohort studies including over a quarter of a million participants residing across the globe. They undertook a meta-analysis - combining all the data from the various studies together and coming up with a sort of consensus finding - and concluded that yes, physical activity does seem to have a positive effect on reducing the risk of depression. The level of the decrease in risk of depression from participation in physical activity seemed to vary slightly according to age, amount of exercise and geographical region, but the trend was most definitely in the protective direction across such variables and remained even after potentially important variables such as body mass index (BMI) and tobacco smoking were taken into account. The authors also reported that: "Although significant publication bias was found, adjusting for this did not change the magnitude of the associations" indicating that although there was a tendency for results 'friendly' to the 'physical activity might reduce depression' hypothesis to be published over less positive results, this did not seemingly affect the findings in any particularly meaningful way.

Accepting that there is a further scheme of work required on this topic in terms of things like what physical activities might be 'best' for depression or depressive symptoms (see here), how to measure physical activity more objectively (ahem, actigraphy) and what the mechanism(s) of effect might be, these are important results. I can think of many, many different areas that the Schuch results could be pertinent to; several already covered on this blog (see here and see here). Many of those areas / conditions / labels reflect states where either physical activity seems to be reduced by choice (see here) or through necessity (see here) but the net result could be the same in terms of elevated risk of depression.

Still, the general message emerging from the Schuch paper is: if you can, move more, and perhaps not just for the physical benefits that accompany physical activity.

And just in case you need some more reading material on this topic, you could do a lot worse than the paper by Brett Gordon and colleagues [2] who concluded that: "Resistance exercise training significantly reduced depressive symptoms among adults regardless of health status, total prescribed volume of RET, or significant improvements in strength."

----------

[1] Schuch FB. et al. Physical Activity and Incident Depression: A Meta-Analysis of Prospective Cohort Studies. Am J Psychiatry. 2018 Apr 25:appiajp201817111194.

[2] Gordon BR. et al. Association of Efficacy of Resistance Exercise Training With Depressive Symptoms. JAMA Psychiatry. 2018. May 9.

----------

Publication bias and autism research

The title of the paper by Konstantin Mechler and colleagues - "Defining the hidden evidence in autism research. Forty per cent of rigorously designed clinical trials remain unpublished - a cross-sectional analysis" [1] - provides some discussion today. Drawing on the ideas that publication bias and/or the so-called 'file-drawer problem' - terms that refer to the non-publication of study results potentially skewing the collected scientific opinion in a particular area - might also extend into autism research too, Mechler et al detail results according to their analysis of a particular trial registration database called: ClinicalTrials.gov.

ClinicalTrials.gov is one of the premier 'tell everyone about your trial' databases designed to provide a bit of transparency to science. The idea is that you register (pre-register hopefully) your research study, register what you are going to do and how, and importantly, provide some details about what your are going to be assessing (outcomes). Some of the research that I've been involved with has a mention in this database (see here) with more to come in future times. Once your entry is in ClinicalTrials.gov it kinda stops you from making any major 'alterations' to your study potentially based on the results you get, whilst at the same time also getting quite a few prods to post things like your raw study results for everyone to see and analyse as they wish. And believe me, they are quite a persistent bunch over at ClinicalTrials.gov!

So Mechler and colleagues "searched for all completed randomized controlled clinical trials investigating interventions in ASD [autism spectrum disorder] and their results made public." They looked at how many trials had been submitted and how many reported results. Where no results were available on ClinicalTrials.gov or on other 'scientific databases' or after "enquiries of the responsible parties or sponsors listed", the authors listed the trial as 'not published'.

The good news: 60% of trials (n=30) were published in the peer-reviewed domain. The not-so-good news was that reported in the opening sentence of this post as some 40% of trials (n=20) fell into that not published category. Authors also mention that some 1600 participants were included in those not published trials, inferring that data on quite a few people diagnosed with an ASD and agreeing (themselves or by proxy) to participate in research remain 'missing' in a research sense.

"The results emphasize the serious issue of publication bias. The large proportion of unpublished results precludes valuable information and has the potential to distort evidence for treatment approaches in ASD." This an important point in any area of science but perhaps more so when you have a label like autism and a whole host of 'unknowns' about the aetiology, nature and course of presentation. I would like to think that there were some rational reasons why so many trials were missing results (studies having to be halted/stopped, collaborations breaking down, resources being exhausted, etc) but there is always going to suspicion around the non-publication of such results particularly the idea that 'researchers or funders did not get the results they hoped for'. Other discussions on this topic outside of autism (see here) point to the failure to publish being tantamount to an ethical breach. Strong words indeed.

Although Mechler et al focused on data from the ClinicalTrials.gov initiative I'd perhaps suggest that the issues they discuss probably go a lot deeper as a function of there being quite a few other databases where such trial information is held and listed. Perhaps also just as concerning is the fact that many other pieces of research are not listed anywhere when it comes to trial registration and therefore have even less 'motivation' to publish results or are perhaps more likely to be subject to 'alteration' in terms of methodology or outcome(s). All of which contributes to the possible tarnishing of science and the reporting of said science and paves the way for criticism.

Music, it's not Mother's Day or anything like that but do give her a call from time-to-time and treat your mother right...

----------

[1] Mechler K. et al. Defining the hidden evidence in autism research. Forty per cent of rigorously designed clinical trials remain unpublished - a cross-sectional analysis. International Journal of Methods in Psychiatric Research. 2016. Nov 9.

----------

Mechler K, Hoffmann GF, Dittmann RW, & Ries M (2016). Defining the hidden evidence in autism research. Forty per cent of rigorously designed clinical trials remain unpublished - a cross-sectional analysis. International journal of methods in psychiatric research PMID: 27862603