“There is no evidence that jumping out of a plane with a parachute improves outcome”

“If you go to PubMed, you will find no publications that breathing air is good for you”

“There’ll never be a trial, we are beyond that”

Have you ever heard these statements made when someone discusses the evidence about a particular new (or old) therapy? The statements might be true but are they useful? Do they advance an argument? What do they mean?

A paper from a Christmas British Medical Journal in 2018 found no evidence of benefit of parachute use when people jumping out of an aeroplane (which happened to be stationary at ground level) were randomised to wearing either a parachute or an empty North Face rucksack. This evidence built on a 2003 systematic review which found that there was no randomised evidence on the usefulness of parachutes for high altitude exits. Both these articles have a somewhat tongue-in-cheek style, but make the point that “…under exceptional circumstances, common sense must be applied when considering the potential risks and benefits of interventions”.

It is self evident that wearing a parachute when jumping out of a plane in flight, or of being in an atmosphere with enough air to breathe, is good for you. When people quote arguments about parachutes or air (or similar) in response to a query about a lack of evidence for a particular intervention, they are implying that the intervention they are discussing is similarly self evidently safe, effective or cost effective, that common sense must be applied. 

The issue is that the benefits of most medical interventions are clearly not in this category. To give some examples from my own field, it is not self evident that dosimetric methods will improve the outcomes of selective internal radiation therapy sufficiently to make a difference to trial outcomes, that endovascular intervention for acute deep vein thrombosis improves long term outcomes compared with anticoagulation, that for complex aneurysms endovascular aneurysm repair is better than open surgery or conservative management… I could go on.

And here we come to the crux of the matter, which is that such comments add nothing to a discussion about an intervention’s evidence base. Rather, their effect is to stifle debate into a confused silence. Whether this is done intentionally or out of embarrassment is irrelevant, the effect is the same: intellectual curiosity is suppressed and questioning is discouraged. This is the opposite of the Empiricism that underpins the whole of Western scientific thought. Before people asked questions, it was self evident that the Earth was flat, that it was the centre of the universe and that it was orbited by the sun. That was just common sense.

A strategy related to appeals to common sense is the weaponisation of the weight of collective opinion. Clinical trial design is dependent on equipoise, meaning clinicians do not know which of several options is better. Equipoise is dependent on opinion, and opinion is swayed by much more than evidence. Medical professionals are just as receptive to marketing, advertising, fashion and halo bias as anyone. Nihilistic statements denying a trial is possible (or even desirable) on the grounds that an intervention has become too popular or culturally embedded are only true if we allow them to be. The role of senior ‘key opinion leaders’ is critical here: they have a responsibility to openly question the status quo, to use their experience to identify and highlight the holes in the evidence, to point out the ‘elephant in the room’. But too often these leaders (supported in some cases by professional bodies and societies) become a mouthpiece for industry and vested interest, promoting dubious evidence, suppressing debate and inhibiting intellectual curiosity. There are notable examples of trials overcoming the hurdle of entrenched clinical practice and assessing deeply embedded cultural norms. This requires committed leaders who create a culture where doubt, equipoise and enquiry can flourish.

Given the rapid pace of technological development in modern healthcare, it is not unreasonable to have an opinion about an intervention that is not backed by the evidence of multiple congruent randomised controlled trials. But this opinion must be bounded by a realistic uncertainty. A better word for this state of mind is a reckoning. To reckon allows for doubt. Instead, when an opinion becomes a belief, doubt is squeezed out. ‘Can this be true?’ becomes ‘I want this to be true’, then ‘it is true’ and ultimately ‘it is self evidently true’. Belief becomes orthodoxy, and questioning becomes heresy and is actively (or passive aggressively) suppressed.

Karl Popper’s theory of empirical falsification states that a theory is only scientifically valid if it is falsifiable. In his book on assessing often incomplete espionage intelligence, David Omand (former head of the UK electronic intelligence, security and cyber agency, GCHQ) comments that the best theory is the one with least evidence against it. A powerful question therefore is not “what evidence do I need to demonstrate that this view of the world is right?” but its opposite: “what evidence would I need to demonstrate that this view of the world is wrong?”. Before the second Gulf War in 2002-3 an important question was whether Iraq had an ongoing chemical weapons programme. As we all know, no evidence was found (before or after the invasion). The theory with the least evidence against it is that Iraq had, indeed, destroyed its chemical weapons stockpile. More prosaically, that all swans are white is self evident until you observe a single black one. 

If someone is so sure that an intervention is self evidently effective, pointing out an experimental design to test this should be welcomed, not seen as a threat. But belief (as opposed to a reckoning) is tied up in identity, self worth and professional pride. What then does an impassioned advocate of a particular technique have to gain from an honest answer to the question “what evidence would it take for you to abandon this intervention as ineffective?” if that evidence is then produced.

Research is hard. Even before the tricky task of patient recruitment begins, a team with complimentary skills in trial design, statistics, decision making, patient involvement, data science and many more must be assembled. Funding and time must be identified. Colleagues must be persuaded that the research question is important enough to be prioritised amongst their other commitments. This process is time consuming, expensive and often results in failure, as my fruitless attempts at getting funding from the National Institute for Healthcare Research for studies on abdominal aortic aneurysm attest. But this is not say that we should not try. We are lucky in medicine that many of the research questions we face are solvable by the tools we have at our disposal if only we could deploy them rapidly and at scale. Unlike climate scientists we can design experiments to test our hypotheses. We do not have to rely on observational data alone.

The 2018 study on parachute use is often cited as a criticism of evidence based medicine. That a trial can produce such a bizarre result is extrapolated to infer that all trials are flawed (especially if they do not produce the desired result). My reading of the paper is that the authors have little sympathy for these arguments. After discussing the criticisms levelled at randomised trials they write with masterly understatement “It will be up to the reader to determine the relevance of these findings in the real world” and that the “…accurate interpretation [of a trial] requires more than a cursory reading of the abstract.”

As I wander around device manufacturers at medical conferences I wonder: what if more of the resource used to fund the glossy stands, baristas and masseuses were channelled into rigorous and independent research, then generating the evidence to support what we do would be so much easier. And I wonder why we tolerate a professional culture that so embraces orthodoxy, finds excuses to not undertake rigorous assessments of the new (and less new) interventions we undertake and is happy to allow glib statements about trial desirability, feasibility and generalisability, about parachutes and air, to go unchallenged.