On transparency in science: lessons from the Pandemrix vaccine and 'partial' evidence-based practice

Minus any hype or sweeping generalisation, I want to draw your attention to a feature article recently published by Peter Doshi [1] titled: "Pandemrix vaccine: why was the public not told of early warning signs?" complete with some media interest (see here).

The feature is based around the vaccines developed to counter the 2009 flu pandemic that starred the H1N1 influenza virus which was sweeping across the globe at the time. The 2009 flu pandemic was potentially serious insofar as its infection pattern and fatalities linked to infection although in reality did not turn out to be quite the 'global killer' that was prophesied. The rush to bring to market a vaccine to protect lives and counter the rise of H1N1 also brought with it controversy from several perspectives [2]; some later talking points seemingly stemming from the inclusion of a specific adjuvant - AS03 - in some vaccine preparations, and data suggestive of a possible correlation between onset of narcolepsy and use of certain flu vaccines containing the AS03 adjuvant (see here). Narcolepsy - a sleep disorder - was one potential adverse side-effect talked about, but other labels/conditions were also the source of some inquiry too (see here). On the basis of the emerging published data, court cases ensued, some of which have apparently unearthed some "internal reports" suggesting that some manufacturers "were aware of other serious adverse events logged" in relation to certain H1N1 vaccines...

The Doshi feature goes through some of these 'revelations' that did not seem to have been made public at the time. He talks about an 'undisclosed problem' whereby one vaccine in particular, Pandemrix, seemed to have a greater frequency of various adverse events reported according to those internal company reports. Most worrying are the reports of 'fatal outcome' across different vaccine preparations, particularly Pandemrix. He also writes: "And the raw numbers of adverse events were not small" highlighting how in one of the latest internal reports dated March 2010 seen by himself (or at least, the British Medical Journal) over 5000 reports of "serious adverse events for Pandemrix" had been received by the company behind the vaccine.

It's timely that the Doshi feature appears now at the time of writing this post, just days after the Peter Gøtzsche - Cochrane drama (see here) and at the same time as other relevant pieces have been published (see here and see here [3]). The similarities around questions of transparency in science (particularly industry-derived science that has a public health aspect attached to it) are very noticeable. As is also the question of 'how much detail should the public be provided with?' when it comes to science in general and then specifically, science and science communication linked to the potential cost-benefit ratio of particular medicines or medical preparations. It's interesting too that Doshi also references some past debate in Germany [4] about the 2009 flu pandemic where it was apparently reported that "the German interior ministry rejected accusations that the ministry had ordered a less risky vaccine for top officials" in light of some newspaper headlines published at the time. 'All animals are equal, but some animals are more equal than others' perhaps springs to mind...

I don't think the issue of less than full transparency in science and the question 'how much detail should the public be provided with' are going to go away any time soon. Such issues strike to the heart of the matter of how peer-reviewed published science is seemingly only one part of a much bigger picture when it comes to the search for a complete truth. One wonders for example, how many other 'internal reports' lie in [research] file cabinets around the world that could conceivably influence evidence-based decisions on the effectiveness and safety of various medicines and other interventions indicated for various conditions? Without such documents ever seeing the cold light of day, how can science ever truly be said to be complete, transparent or indeed settled?

Such a 'transparency' situation also makes initiatives like OpenTrials - calling for all research to be more 'discover-able' and 'traceable' - all the more vital and important. Issues such as the requirement for pre-registration of all studies via databases such as ClinicalTrials.gov and importantly, the timely release and publication of results (see here) are pressing features. Study pre-registration (especially prospective registration) in particular, should perhaps be pushed a lot more than it currently is, and shouldn't just be reserved for big studies of drugs and medicines either.

Science is great and is getting better all the time. But it still has lots of issues and problems to overcome. A lack of research transparency seems to be a particularly important issue that, whilst starting to receive some much required attention, will if present, always mean that evidence-based practice really should be re-branded as 'partial evidence-based practice'. And that has lots of implications...

To close, given the subject matter included in the Doshi paper, I want to end with a reminder about not making sweeping generalisations. At the same time I'd also champion the words included in another recent article by Jørgensen and colleagues [5] (yes, also including Peter Gøtzsche as an author) responding to the idea that public confidence in public health initiatives such as vaccination may be "undermined" by even having such discussions in public. To quote: "Debates over sources of evidence must take place in public, especially when public health interventions are at stake." This is again, with the idea that science needs transparency - even better in the peer-reviewed domain - and closed door discussions rarely aid such transparency.

----------

[1] Doshi P. Pandemrix vaccine: why was the public not told of early warning signs? BMJ 2018;362:k3948

[2] Doshi P. Influenza: marketing vaccine by marketing disease. BMJ 2013;346:f3037.

[3] Jørgensen L. et al. Challenges of independent assessment of potential harms of HPV vaccines. BMJ 2018;362:k3694

[4] Stafford N. Only 12% of Germans say they will have H1N1 vaccine after row blows up over safety of adjuvants. BMJ 2009;339:b4335.

[5] Jørgensen L. et al. Response to Cochrane editors: Jørgensen, Gøtzsche and Jefferson. BMJ EBM Spotlight. 2018. Sept 23.

----------

On Cochrane and 'facts' and 'politics' in evidence-based medicine

The controversy surrounding the reported expulsion of Peter Gøtzsche from the Cochrane Collaboration (or should that just be Cochrane?) is something that I've been following for a few days now (see here). Cochrane, under the heading "Trusted evidence. Informed decisions. Better health", represents one of the premier go-to sources for evidence-based healthcare advice on a range of topics. Some of those topics have been previous fodder for this blog too (see here and see here for examples).

I don't profess to have any unique insight into the various goings-on leading to the reported expulsion of Gøtzsche and resignations of fellow members beyond that which has been discussed in various sections of the science media (see here and see here and see here). From what I gather, things look like they've been 'brewing' for a while with regards to Cochrane and the views and opinions expressed by some of those who are seemingly departing. Such a public spat however, is unlikely to be good for science or evidence-based medicine, and indeed may have some wider implications for some fundamentals of science and science communication...

One of the possible [late] precipitating events mentioned around the Gøtzsche saga was the publication of a quite scathing article by Lars Jørgensen and colleagues [1] (including Gøtzsche as an author) questioning the published results of a recent Cochrane review [2] titled: "Prophylactic vaccination against human papillomaviruses [HPV] to prevent cervical cancer and its precursors". The original review by Marc Arbyn et al garnered media headlines when published (see here) as per conclusions such as: "There is high‐certainty evidence that HPV vaccines protect against cervical precancer in adolescent girls and young women aged 15 to 26." A comforting finding by all accounts. Accompanying such efficacy data were other statements made by the authors on the basis of the evidence reviewed that: "The vaccines do not increase the risk of serious adverse events, miscarriage or pregnancy termination."

Jørgensen and colleagues however put forward their [peer-reviewed] view that the Arbyn paper fell short of the expected standards from Cochrane: "We do not find the Cochrane HPV vaccine review to be ‘Trusted evidence’, as it was influenced by reporting bias [3] and biased trial designs." They highlighted several 'issues' with the original review stretching from trial selection for the review, to the assessment of "serious and systemic adverse events" to potential "conflicts of interest." Similar sentiments had been voiced about other Cochrane reviews too that were subsequently pulled from the scientific literature. Feathers were inevitably ruffled (see here) by the Jørgensen paper, even as far as prompting a response from the journal that published the paper [4] about the peer-review process leading to publication of the critique. Things are getting serious when a journal has to defend its publication of a paper.

Without wishing to reduce this saga to any one event, I don't think it would be unreasonable to assume that the Jørgensen paper might have influenced matters quite considerably; perhaps even bringing them to a head. As per involvement on the Boesen paper [5] Gøtzsche is no stranger to calling out Cochrane reviews that seemingly don't make the grade, alongside also voicing opinions on various other matters down the years. To quote: "... in another book, [he] likened the pharmaceutical industry to "organized crime""; such forthright statements stretching back some years are unlikely to have made too many friends in certain circles.

As per the title of this post mentioning the words 'facts' and 'politics', one particular write-up of this saga I think hits the nail on the head. The opinion piece from Trish Greenhalgh [6] presents the two sides to this 'dispute': on the one hand is that the “crisis” is "philosophical (relating to the nature of facts)" and on the other, "political (relating to organisational governance)." The philosophical side of things is pretty evident as per the publication of the Jørgensen paper as a counter to the Arbyn paper. Greenhalgh mentions about how "Gøtzsche might be classified as an evidence-based medicine purist" given his views and sizeable contribution to various "statements on how to undertake and publish research." In this respect, his published views on the original Arbyn paper (and similarly in other reviews) seem to detail scientific standards not being met, or at least not being met to his and his co-authors standards. And certainly on points such as access to trial results and data, he's seemingly not alone in his concern (see here).

On the political side of things, well, at the time of writing we just don't know enough to reasonably comment. Greenhalgh mentions that: "The political explanation for Cochrane’s crisis relates to the tension between governing an organisation and respecting individual members’ academic freedom to express dissent." The fact that such dissent has over the years covered various important public health topics - "cast doubts about the safety of a vaccine against human papillomavirus (HPV), a cause of cervical cancer, and says psychiatry has “gone astray” by coercing patients into taking medication, such as antidepressants, they don’t want to use and that cause “brain damage” over the long run" - is likely to have really stretched those organisational relationships with Gøtzsche. Not least because immunisation and antidepressant use reflect important pillars of modern public healthcare and, historically, uptake of such medicines is very, very susceptible to differences in scientific views, opinions and beyond.

As to the idea mentioned by Greenhalgh that: "We should cut it [Cochrane] some slack while it gets its house in order", I'm not exactly sure how it's going to approach this 'house in order' requirement and what this means for the future credibility of Cochrane. Based solely on the 'facts' side of this saga, it strikes me that organisations like Cochrane actually need people like Peter Gøtzsche and their "evidence-based medicine purist" beliefs. They need them in order to critically (really critically) boil down the ever-growing research literature into scientifically sound statements for public and policy consumption without fear or favour. Minus such voices, it's more likely that evidence-based messages originating from such initiatives are perhaps going to be weakened, which in turn means that population healthcare is potentially going to suffer. Moreover, I assume also that just because Gøtzsche is reportedly not part of Cochrane any more does not mean that he won't be heard from again in the peer-reviewed domain...

----------

[1] Jørgensen L. et al. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine. 2018. July 27.

[2] Arbyn M. et al. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev. 2018 May 9;5:CD009069.

[3] Jørgensen L. et al. Index of the human papillomavirus (HPV) vaccine industry clinical study programmes and non-industry funded studies: a necessary basis to address reporting bias in a systematic review. Syst Rev. 2018 Jan 18;7(1):8.

[4] Heneghan C. & Onakpoya I. Editors’ response to concerns over the publication of the Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine. 2018. Sept 12.

[5] Boesen K. et al. The Cochrane Collaboration withdraws a review on methylphenidate for adults with attention deficit hyperactivity disorder. Evid Based Med. 2017 Aug;22(4):143-147.

[6] Greenhalgh T. The Cochrane Collaboration—what crisis? BMJ. 2018. Sept 17.

----------

HBOT and autism systematically reviewed again (and the same results?)

"To date, there is no evidence that hyperbaric oxygen therapy improves core symptoms and associated symptoms of ASD [autism spectrum disorder]."

So said the results of the review by Xiong and colleagues [1] (open-access available here) completed under the auspices of the Cochrane Collaboration, leaders in the science and publication of systematic reviews (see here for another example).

Looking at the collected peer-reviewed science on the topic of hyperbaric oxygen therapy (HBOT) for autism - where "the patient breathes near 100% oxygen intermittently while inside a hyperbaric chamber pressurized to greater than sea level pressure", authors looked to determine several factors. Whether "treatment with hyperbaric oxygen: 1. improves core symptoms of ASD, including social communication problems and stereotypical and repetitive behaviors; 2. improves noncore symptoms of ASD, such as challenging behaviors; 3. improves comorbid states, such as depression and anxiety; and 4. causes adverse effects."

The results kinda mirrored what has already been previously described in the existing research review literature (see here), that based on one trial only [2] reaching their inclusion criteria, there is little evidence at the moment to say that HBOT is blanket indicated for autism.

Obviously one has to be a little careful that 'absence of science' is not construed as 'absence of evidence'. Indeed, I'm a little annoyed that the authors start suggesting that HBOT "may not be appropriate" for further study with autism in mind in light of the lack of studies in this area based on "the absence of a persuasive theory of change from experimental and clinical studies, the unknown long-term safety of the treatment, and the financial and opportunity costs of not participating in other proven therapies." No, I'm not defending HBOT as an intervention for autism but I am defending the idea that research reviews based on one study should really be trying to clarify the science rather than block future research efforts. Indeed, I'm inclined to direct you to the paper by Rossignol and colleagues [3] potentially answering questions such as mode of action and how one might want to look at potential best-responders to this type of intervention (autisms people, autisms). That and what the Sampanthavivat study included in the Xiong review actually said about 'safety' during their trial: "interventions were safe and well tolerated with minimal side effect from middle ear barotraumas."

To close, and in keeping with the date, a spooky song (and a rather spooky singer it has to be said...)

----------

[1] Xiong T. et al. Hyperbaric oxygen therapy for people with autism spectrum disorder (ASD). Cochrane Database Syst Rev. 2016 Oct 13;10:CD010922.

[2] Sampanthavivat M. et al. Hyperbaric oxygen in the treatment of childhood autism: a randomised controlled trial. Diving Hyperb Med. 2012 Sep;42(3):128-33.

[3] Rossignol DA et al. Hyperbaric oxygen treatment in autism spectrum disorders. Medical Gas Research. 2012; 2: 16.

----------

Xiong T, Chen H, Luo R, & Mu D (2016). Hyperbaric oxygen therapy for people with autism spectrum disorder (ASD). The Cochrane database of systematic reviews, 10 PMID: 27737490

1000 blog entries or thereabouts

Although perhaps being a little self-indulgent, I'm taking a little time out today to reflect on a few milestones reached by this blog and some advice to be imparted.

Not so long ago I posted the 1000th entry on Questioning Answers and with it over 4 years of riding the roller coaster that is peer-reviewed science with an autism research slant. Allied to a page view count rising above 1.1 million (hopefully not all junk feeds!), I'd like to think that my science writing skills are being honed as well as there being some appetite for my various musings. Certainly, I have a lot more appreciation for the huge wealth of information that is out there in science land when it comes to the autism spectrum and beyond as a result...

Reflecting on the research blogging experience so far and in particular, remembering the fateful evening when it all started (see here), I'm minded to offer a few observations about the whole thing particularly with a view to encouraging more people to put on-line pen to on-line paper specifically with autism research in mind:

• Where does the time go? One question that has come up more than most about my blogging experience is 'where do you get the time to write?' The simple reply is: 'how long do you think it takes?' Depending on the size of the chosen paper and topic for blogging, it can take anywhere between 1-2 hours to get a post right. This includes reading said paper(s) and some light reading around the topic. Often there is some light tinkering to a post after a few hours break but this doesn't take long. Precisely when this writing gets done is very much dependent on the day and my mood. Aside from a day job, quite a big family and an exercise routine to maintain among other things, I tend to fit in bits of writing as and when I can. Early mornings and late nights have been previously experienced but timing very much depends on me. I've learned to live without watching old re-runs of Buck Rogers or endless news programmes in favour of doing something a little more intellectually stimulating...
• Picking a paper/topic. I don't have any strict criteria for what (or who) to blog about. There is a constant stream of peer-reviewed science published daily that never seems to dry up, always with something eye-catching to me. I am a great believer in evidence-based medicine and its application to the autism spectrum but for me, such evidence does not just have to include randomised-controlled trials or systematic reviews / meta-analyses. The humble case report and N=1 is often as revealing as 'big data' particularly given the heterogeneity of autism and its important 'add-ons'. That's why I will blog about many types of study based on different degrees of evidence. There are some topics/papers that are more appealing to me than others and I might prioritise them above others. That being said, I'd like to think I cover quite a broad spectrum of autism research on the blog.
• Social media strikes. Much like blogging, I've also come to embrace social media as part of my professional life. No longer just the place for 'cat videos' or the like, platforms like Twitter, Facebook and Google+ are veritable gold mines of information when it comes to science. Outside of some interesting discussions down the years, social media has also been a great way of promoting my blog to a large audience. Given that I'm a stand alone sort of blogger, social media has been instrumental in getting my musings out to a wider audience.
• Research qualifications? Research, as it currently is, was a part of my professional life before blogging, eventually leading up to the receipt of a PhD. Some of the most valuable lessons I learned during my PhD (aside from how to handle the term 'major corrections') were how to think and write with a critical eye, not to take anything at face value and to understand that science is by no means an absolute process or concept in terms of the data produced. Science is only as good as the people who do it and the techniques they use to do it, and just because something gets through peer-review, does not mean that it should be taken as Gospel. Having gone through the process of a PhD (alongside other post graduate academic experiences) has held me in good stead all these years but I wouldn't want anyone to think that it is a pre-requisite for science blogging. It's not. Some of the cleverest and most astute people I've met over the years of my research and blogging career have never been anywhere near a University lab or carried out something like a structured assessment. They are logical thinkers; many of them also quickly realising that science is all about probability, and that absolutes are few and far between. In other words, science blog as a critical thinker.

I'm gonna leave it at that with the self-indulgence and fatherly advice about blogging. I'd very much like to thank all those who've interacted with me and my blog over the various platforms of debate and years of doing it. I very much intend to keep going with the blog and hopefully provide a similar update when 2000 blog entries comes up. Hopefully at that time, I'll also be able to link to lots and lots of new autism research blogs too...

And in the grand tradition of this blog, here is a song by a favourite band: Frankly Mr Shankly by The Smiths ("a flatulent pain in the ass").

Beware the inflated science related press release!

I'm not normally minded to post on a Sunday (day of rest and all that) but I did want to bring your attention to the results presented by Petroc Sumner and colleagues [1] (open-access) concluding that: "Exaggeration in news is strongly associated with exaggeration in press releases" when it comes to the media reporting of [some] health-related science news.

The idea behind this particular study - which has been summarised pretty well in some of the accompanying media and in an editorial in the publishing journal [2] - was to look at the contribution of the press release (the 'look at me' part of some science publishing) to those sometimes 'inflated' health science headlines which we all encounter on a day-to-day basis. The results suggested that whilst [some] journalists and editors might need a refresher course on some of the basics of science (including the concept of probability and risk), academic press offices and even the very academics behind said research might also have to shoulder some responsibility (bearing in mind correlation is not the same as causation!) So when for example, a mouse study correlating the presence of compound X with improvement in the [mouse] condition Y is interpreted as meaning that compound X is the elixir of life for Homo sapiens and that we should instantly rush out and buy as much of it as we can afford, look to the press release before blaming the newspaper. I exaggerate of course with that example.

So as not to keep you from your Sunday breakfast or that magnificent Sunday lunch you no doubt have planned/are eating, I'm not going to say much more about this topic aside from making two additional points:

(a) Post-publication peer-review is mentioned somewhere in one of the texts. Although this is traditionally meant to imply that researchers who have a beef about some study or interpretation of results send a letter to the publishing (or other) journal outlining their issues, there are quite a few other mediums these days which seem to get the job done just as well. You are reading one of those mediums - yes, bloggers of the world unite - and how on occasion, blog entries have talked about press releases and peer-reviewed results not quite tallying together. I dare say someone somewhere might eventually start a blog titled something like: 'pressing ahead: press release vs. actual results' if it hasn't already been done. Other social media might also play an important role in highlighting discrepancies.

(b) Although interesting, the Sumner results perhaps leave out one very important variable in this scientific producer - consumer relationship: the consumer. You might well scoff that anyone without a PhD or related qualification 'doesn't understand science' but I would say that you are wrong. Although there are people out there who actually believe the term 'scientifically proven', as if God himself endorsed the product, I'd suggest that there are enough people who don't believe every headline they read and are vocal enough to say so. Search engines such as Google do a pretty good job of ensuring that anyone researching a scientific claim important to them (which most people do these days) quickly get both sides of the story. In that respect, an inflated press release (which are also generally detectable on the web) will probably not stay inflated for too long. Probably less so, after this and other research [3] starts to percolate through the web...

Without further ado, I'll leave you to finish of your 'super-food' packed Sunday lunch. And just in case anyone is interested, a few things to bear in mind when reading and interpreting science...

----------

[1] Sumner P. et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ 2014; 349: g7015.

[2] Goldacre B. Preventing bad reporting on health research. BMJ 2014; 349: g7465.

[3] Woloshin S. et al. Press releases by academic medical centers: not so academic? Ann Intern Med. 2009 May 5;150(9):613-8.

----------

Sumner, P., Vivian-Griffiths, S., Boivin, J., Williams, A., Venetis, C., Davies, A., Ogden, J., Whelan, L., Hughes, B., Dalton, B., Boy, F., & Chambers, C. (2014). The association between exaggeration in health related science news and academic press releases: retrospective observational study BMJ, 349 (dec09 7) DOI: 10.1136/bmj.g7015