Values, guidance, NICE and the ESVS.

This is a transcript of a 7 minute talk I was invited to give at the Cardiovascular and Interventional Society of Europe’s [CIRSE] annual conference in Barcelona, as part of a session on “Controversies in Standard Endovascular Aneurysm Repair [EVAR] within IFU” [indications for use].

This talk: “NICE guidelines best inform clinical practice”, was one side of a debate: my opponent’s title was “European Vascular Society [ESVS] guidelines should be the standard of care”.

If you have on-demand access to CIRSE2022 content, you can view a recording of the session here.

Barcelona. Spain. 13^th September 2022. 15:00h

Thanks. My name is Chris Hammond, and I’m Clinical Director for radiology in Leeds. I was on the NICE AAA guideline development committee from 2015-2019.

I have no declarations, financial or otherwise. We’ll come onto that in a bit more detail later.

This talk is not going to be about data. I hope we are all familiar with the published evidence about AAA repair. No. This talk is about values. Specifically, the values that NICE brings to bear in its analysis and processes to create recommendations and why these values mean NICE guidelines best inform clinical practice. What are those values?

Rigour, diversity, context.

Let’s unpick those a little.

NICE’s is known for academic rigour. Before any development happens, the questions that need answering are clearly and precisely identified in a scoping exercise. A PICO question is created, the outcomes of interest defined, and the types of evidence we are prepared to accept are stipulated in advance.

The scope and research questions are then published and sent out for consultation – another vital step.

After the technical teams have done their work, their results are referred explicitly back to the scope. Conclusions and recommendations unrelated to the scope are not allowed.

This process is transparent and documented and it means committee members cannot change their mind on the importance of a subject if they do not like the evidence eventually produced.

It’s impossible to tell from the ESVS document what their guideline development process was. A few paragraphs at the beginning of the document are all we get. ESVS do not publish their scope, research questions, search strategies or results. How can we be assured therefore that their conclusions are not biased by hindsight, reinterpreting or de-emphasizing outcomes that are not expedient?

We can’t.

For example, data on cost effectiveness and outcomes for people unsuitable for open repair are inconvenient for EVAR enthusiasts. I’ll let you decide the extent to which these data are highlighted in the ESVS document.

More, in failing to define the acceptable levels of evidence for specific questions ESVS ends up making recommendations based on weak data. Recommendations are made based on the European Society of Cardiology criteria which conflate evidence and opinion. Which is it? Evidence or opinion?

Opinions may be widely held and still be wrong. The sun does not orbit the earth. An opinion formulated into a guideline gives the opinion illegitimate validity.

Finally, there is the rigour in dealing with potential conflicts of interest. These are the ESVS committee declarations – which I had to ask for. The NICE declarations are in the public domain on the NICE website. Financial conflicts of interest are not unexpected though one might argue that the extensive and financially substantial relationships with industry of some of the ESVS guideline authorship do raise an eyebrow.

The question though is what to do about them. NICE has a written policy on how to deal with a conflict, including exclusion of an individual from part of the guidance development where a conflict may be substantial. This occurred during NICE’s guideline development.

The ESVS has no such policy. I know because I have asked to see it. Which makes one wonder: why collect the declarations in the first place.

How can we then be assured these conflicts of interest did not influence guideline development, consciously or subconsciously.

We can’t

What about diversity?

This is the author list of the ESVS guideline. All 15 of the authors, all 13 of the document reviewers and all 10 of the guideline committee are medically qualified vascular specialists. They are likely to all have had similar training, attended similar conferences and educational events and have broadly similar perspectives. It’s a monoculture.

Where are the patients in this? The ESVS asked for patient review of the plain English summaries it wrote to support its document, but patients were not involved in the development of scoping criteria, outcomes of importance or in the drafting of the guideline itself.

Where is the diversity of clinical opinion? Where are the care of the elderly specialists to provide a holistic view? Where is anaesthesia? Primary care? Nursing?

Where is the representation of the people who pay for vascular services: infrastructure, salaries, devices? And who indirectly pay for all this, maybe for your meal out last night, for the cappuccino you’ve just drunk? Where is their perspective when they also have to fund the panoply of modern healthcare?

NICE committees have representation of all these groups, and their input into the development of the AAA guidance was pivotal. The NICE guidance was very controversial, but the consistency of arguments advanced by diverse committee members with no professional vested interest was persuasive.

Finally, we come to context.

An understanding of the ethical and social context underpinning a guideline is essential.

We cannot divorce the treatments we offer from the societal context in which we operate. We live in a society which emphasises individual freedom and choice and are comfortable with some people having more choices than others, usually based on wealth. Does this apply equally in healthcare? In aneurysm care? What if offering expensive choices for aneurysm repair means we don’t spend money on social care, nursing homes, cataracts or claudicants.

To what extent should guidelines interfere with the doctor-patient relationship? Limit it or the choices on offer? What is the cost of clinical freedom and who bears it?

NICE makes very clear the social context in which it makes its recommendations. It takes a society-wide perspective, and its social values and principles are explicit. You can find them on the NICE website. Even if you don’t agree with its philosophical approach, you know what it is.

We don’t know any of this for the ESVS guideline. We don’t know how ESVS values choice over cost, the individual over the collective. Healthcare over health. This means that the ESVS guideline ends up being a technical document, written by technicians for technicians, devoid of context and wider social relevance.

The ESVS guideline is not an independent dispassionate analysis, and it never could be, because its development within an organisation so financially reliant on funding from the medical devices industry was not openly and transparently underpinned by NICE’s values of rigour, diversity and context.

Rigour. Diversity. Context

That’s why NICE guidelines best inform clinical practice.

Thanks for your attention.

Some personal reflections on AAA guideline development with NICE

The development of the NICE guidance on the management Abdominal Aortic Aneurysm [AAA] has been a long, drawn out and difficult process. Publication of the final guidance (in Spring 2020) was overshadowed by the immediate (and ongoing) crisis created by the COVID pandemic which meant the revised recommendations did not get the scrutiny they deserved. While editorial comment was made about the surprising nature of NICE’s U-turn, the opportunity for wider debate and discussion has necessarily been lost.

My colleagues on the NICE AAA Guideline Development Committee [GDC] have recently published our (rejected) final recommendations on the repair of unruptured AAA. These were revised from the draft guideline after taking into account stakeholder feedback and NICE’s view on implementation but they were unacceptable to NICE. The basis for this was not made clear and the process by which NICE derived the recommendations it eventually published remains remarkably opaque.

The purpose of this blog post is to provide a personal perspective on my involvement in the guidance development process and to offer some suggestions for a way forward.

Since I was appointed to the AAA GDC, and particularly since the publication of the draft guidance in 2018, my involvement with NICE has been a significant professional and personal challenge. By and large (and with a couple of notable exceptions) I have not been subjected to the opprobrium that some of my colleagues on the committee have had to deal with but the behaviour of NICE’s senior management has left me with a deep sense of frustration, even anger at work taken for granted or ignored and for opportunities missed.

The gulf between the evidence base for the elective repair of unruptured AAA and current UK (and international) practice created a problem for NICE which was always going to result in difficult guideline implementation and professional acceptance. The complete proscription of standard EVAR in the draft guidance was a substantial shock to the vascular surgery and vascular radiology community and resulted in much tension and professional anxiety. This was evident in the stakeholder feedback received, which prompted a thoroughgoing review of the draft recommendations.

The health economic argument against EVAR for people fit for open repair is undeniable and I was (and remain) entirely content with the recommendation that EVAR not be offered to this group (whether you agree depends largely on whether you think cost-effectiveness is a reasonable basis for limiting access to an intervention). However, for patients unfit for open repair, my personal view was that there were cogent arguments for changes to the draft. In particular I had concerns about the generalisability of the randomised data to the whole of an almost certainly heterogenous population, and the difficulty applying population data to individuals in whom the alternative was no-repair. However, actually formulating revised recommendations incorporating these arguments was extremely difficult and I was unable to persuade anyone else in the GDC that my clumsy suggestions for rewording were an improvement. Ultimately, my GDC colleagues convinced me that minor revisions to the draft we made were reasonable and I was happy to accept cabinet responsibility for them.

However, NICE were unwilling to publish the revised recommendations on repair of unruptured AAA, and an impasse was reached in spring 2019. To resolve this, over the summer of 2019, the GDC made considerable further efforts to revise them into a format acceptable to NICE’s senior management, and incorporating the themes raised by stakeholders and NICE’s implementation concerns. These are the recommendations the GDC has recently published and I think they are excellent. They have the unanimous support of the whole GDC, with no dissenting voices.

During this time however (summer 2019 onwards), NICE abandoned and then (apparently deliberately) sidelined its GDC. One can argue the extent to which this behaviour failed the public in producing suboptimal recommendations on elective AAA repair. But NICE certainly failed in its duty of care to the committee members who received no support and only cursory communication and explanation. NICE simply asserted its right to editorial control over the recommendations it publishes – it has never previously exercised this right. It is astonishing NICE did not make more effort to engage with its GDC over 2019 to find a mutually acceptable set of recommendations. I am left with the sense that I, and the other GDC members, were deemed irresponsible and uncompromising absolutists when nothing could be further from the truth: despite NICE’s disengagement, we made huge efforts to create recommendations that were consistent with the evidence base, stakeholder comment and the published philosophical and ethical frameworks within which NICE requires its guidance to be developed.

I am also left with a sense that there has been a missed opportunity to influence repair strategies for unruptured AAA. The final guidance NICE published on this is bland and anodyne to the point of being meaningless. The cynic in me thinks this is deliberate: recommendations that don’t recommend anything allow practice to continue without amendment or cultural shift. Perhaps this is a convenient outcome for those to whom NICE seems to have turned, once it abandoned its GDC.

For example, NICE’s final (published) recommendation about repair for people unfit for open surgery:

1.5.5 Consider EVAR or conservative management for people with unruptured AAAs meeting the criteria in recommendation 1.5.1 who have anaesthetic risks and/or medical comorbidities that would contraindicate open surgical repair.

How should a clinician consider this? What evidence and perspective should they bear in mind when making a decision with the patient? What information should the patient be offered? Does this recommendation contain anything that will help vascular specialists amend their practice to mitigate the marked regional variation in the management of unruptured AAA in the UK (variation that surely cannot be explained by case-mix)? While there is some evidence that the draft guidance has resulted in a small increase in the proportion of AAAs repaired with open surgery, whether this endures remains to be seen and there is nothing in the guideline to lock it in.

These concerns were not just mine. Ultimately the whole GDC was in agreement (again, with no dissenting voices) that NICE’s published recommendations about repair of unruptured AAAs neither reflect the evidence nor (even) NICE’s own narrative accompanying the recommendations. Given the GDC’s professional and lay diversity, this unanimity is striking. We all agonised over the decisions we made, but all independently reached the same conclusion. This was not some kind of bunkered groupthink.

Despite all this, there are some significant positives to be taken from my experience with NICE. even if I remain dissatisfied with the process and aspects of the eventual outcome. I enjoyed meeting colleagues I would otherwise not have met, and in particular I enjoyed the careful and academic consideration of the evidence base and the challenging of some of my preconceptions. NICE’s technical teams and information specialists are impressive and the clarity and precision they brought to committee discussions was very enlightening. Obviously I now have a valuable understanding of how NICE develops guidance and some of the compromises involved.

My experience also prompted a hitherto unknown interest in health economics and in particular the ethics associated with using this in decision making about health and healthcare interventions. What do we mean by need? How do we balance choice, affordability, cost effectiveness and equity? When demand in healthcare is substantially supplier driven, how do we prevent market failure? What value judgements do we make or need to make when allocating resource? These are big questions and set against them, and the huge challenges of the COVID pandemic, a spat about elective repair of unruptured AAA seems insignificant. But the themes raised when thinking about provision of AAA repair are an illustrative worked example in microcosm. Recovery from the pandemic urgently requires that, across the entire NHS, we allocate resource where it is most effective, rather than on the basis of special pleading or the misguided notion that doing nothing represents a medical or moral failure. I’m aware this is not everyones cup of tea!

Where do we go from here?

I think we need to consider carefully the language we use about AAAs and their repair: language has a powerful effect on thinking and constructs meaning. Elective AAA repair is predominantly an exercise in risk factor management in a multimorbid population, not a life saving intervention. While there has been a (thankful) move away from ‘ticking time bomb’, words like ‘threshold’ and ‘turndown’ imply a necessity for repair that is unsupported by the evidence base. They create a psychological momentum toward intervention that takes conscious effort to halt. ‘Likely (or unlikely) to benefit from repair’ seems more appropriate, though it does not trip off the tongue.

More importantly, we need to understand more about AAA as a disease, not just about EVAR or open repair as ways of treating it. This is an essential shift in emphasis. Until we know more about the contemporary prognosis of people with AAA we will be unable to make decisions with them about whether repair is worthwhile (by whatever criteria we choose: clinical effectiveness, cost effectiveness, patient satisfaction or something else). Focussing solely on research into the outcomes after repair (with any technique) will always fail to provide an answer to the first question that we should ask: will this person benefit from having their AAA repaired? The paradox is that it is impossible to investigate the natural history of AAA if we continue to repair nearly all of them. And it is also possible that once we know the prognosis for people with AAA, fewer procedures will be undertaken: a professional problem for vascular specialists who enjoy the technical challenge of AAA repair or have built a career on it, and a financial one for the medical technology industry. Will we like what we find?

The future of AAA repair is in our hands. We can choose to focus on AAA repair technique or on the AAA and the person with it. So next time you go to a conference, or attend a seminar, a webinar or a course ask yourself what is being addressed: technique or disease? When key opinion leaders speak at tentpole events like CIRSE, LINC, Cx, SIR and BSIR ask them: “Who should not get repair and who should?”, “Does cost effectiveness matter?”. Do their answers convince you? Are these questions they are interested in? Only by challenging can we shift the frame of reference.

As for NICE, I still believe it is of immense value as an organisation. But if you do get involved (and despite everything I would probably recommend it) be prepared to have some of your idealism and enthusiasm tarnished by political expediency. I wonder whether the medical profession and wider society are ready for some of the conclusions that flow inevitably from a consistent implementation of NICE’s principles.

Further reading:

A response to Balancing evidence in guidelines – an essay (BMJ)

NICE’s AAA guideline – an unexplained U-turn (BMJ opinion)

Two responses (from NICE and from the VSGBI) to the BJS editorial referenced in the first paragraph

Cost-effectiveness, art and science in medicine

I’m consulting with a man in his mid 70s: a retired teacher in an inner city secondary school. He used to smoke, but quit several years ago. He’s otherwise pretty fit: he has never learned to drive so walks everywhere and uses public transport. He is married, and his wife and he are still independent. They have grown up children who live a long way away. They go out to the theatre when they can and enjoy going to the local pub together and with friends. He’s got high blood pressure which is well controlled. He also has an abdominal aortic aneurysm (an AAA). It’s large. And he is worried about it. We talk about the options for repair. He is anxious about a major abdominal operation. He’s not dead set against it, but is concerned about the recovery, the impact it will have on his wife. On balance he is minded to have an endovascular repair, for which the AAA is anatomically suitable. We discuss the long term outcomes of open and endovascular repair. The need for secondary procedures and surveillance. He leaves the consultation undecided and plans to discuss it with his family.

I have spent the last 5 years on the NICE Guideline Development Committee, developing a guideline for the management of people with AAA. The results of several large trials comparing open and endovascular repair of AAA are consistent that while endovascular repair and open repair are safe (meaning very few people die from them), endovascular repair is safer by a small margin and gets people out of hospital and back to normal substantially more quickly than open surgery. However the longer term outcomes are not as good and beyond about 7 years, more people are dead after endovascular repair than after open repair. Because of this, endovascular repair is not cost effective, meaning the opportunity cost of providing it is too great and at a population level offering endovascular repair causes harm. Putting aside arguments about the contemporary relevance or methodological detail of the evidence, the only possible conclusion is that endovascular repair should not be undertaken if someone can have an open operation.

There are several problems with accepting this analysis. One is that it requires a belief that savings made in one place in the healthcare economy will be realised in a benefit elsewhere: something called Pareto efficiency. This might be an intellectual leap too far for some in the NHS of 2020. But perhaps the biggest issue that it is a cold joyless analysis – a faceless functional accounting, the reduction of individual encounters to marks on a Kaplan-Meier chart. It is the science in medicine, but medicine is more than science. It’s an art. For me some of the joy of practicing medicine is in this art: the ability to synthesise the evidence into a narrative a patient can engage with, helping them work out the best option. What do I do then, when having done this, my patient’s preferred option is not cost effective, when the art and the science collide? How do I decide between the interests of a real person sitting before me, vulnerable, perhaps anxious, who trusts me to make decisions with them, for them and in their best interests and the interests an unknown person (or group) with whom I have no relationship – the people who will theoretically benefit if I choose the science over the art. How do I decide between my patient and society?

Since the publication of the draft NICE guidance on AAA, which recommended against endovascular repair partly on the basis of cost effectiveness arguments, I have defended and explained the decisions the guideline committee made at conferences, in conversations and in print. I have posed questions to conference panellists who were challenging the guideline about whether they think cost effectiveness is an important consideration when deciding a therapy and received frequently unconvincing responses. But in the back of my mind, I’m questioning: are they right? Are the unseen consequences of my clinical decisions (for unknown people) my responsibility or are these too distant from me to consider.

Within the contexts by which NICE asks its committees to make decisions, the answer is clear: they are not too distant, and must be taken into account. But while it may be the right thing to do, to follow the science and deny treatments that are not cost effective, it can feel wrong. It’s a depressing analysis to frame the future for individual people in terms of population outcomes. Individual characteristics, ambitions, concerns and expectations, love, beauty and hope are subordinated to the inevitable logic of the data. This feels like a betrayal of the doctor-patient relationship which is ultimately a personal one: and guidelines are implemented at a personal level, patient by patient.

This is not a new dilemma, either in medicine or in economics in general. It is a version of ‘the tragedy of the commons’. To some extent patients, the public and physicians understand it, in the general acceptance of triage in emergency departments, or waiting lists for elective treatment. But the idea of not offering treatments at all, solely on the basis of a cost effectiveness calculation, seems to be too much for many people in their role as clinicians or patients, even if it makes sense in their role as taxpayers. In fact it is probably impossible (and certainly not desirable) for clinicians to make bedside judgements on the basis of cost effectiveness, not only because of how it makes them feel, but also because it would undermine the trust central to the relationship with their patient. Solutions to the dilemma attempt to constrain the options available to clinicians either by incentivising certain therapies, or by limiting choice to cost effective options only (by either not funding the alternatives or by creating guidelines). In order to allow these solutions, clinicians need to accept that some decisions are taken out of their hands. But in doing this, it can seem as if the art in their practice is reduced to a near irrelevance. Perhaps this, then, is one of the reasons for the criticism, even resentment, of some of NICE’s draft AAA guidance.

Where does this leave us? We are caught in the cleft stick of increasingly costly technological advances in healthcare, wanting to offer them to our patients where appropriate, but understanding that ultimately resources are finite. We will need to face this as a society and as individual clinicians sooner or later unless healthcare costs are to escalate uncontrollably. We surely need to understand that the technical advances that interest us may not be in the best interests of society at large, and accept this where it is the case. We need to allow trusted organisations to make these decisions for us, within an open and transparent process, and we need to allow that the art in medicine (and our joy in practising it) is retained but refocussed to help our patients navigate their therapy choices within the constraints imposed on us by those trusted organisations. NICE’s failure to stand firm on its principles in respect of this aspect of the AAA draft guidance sets an unfortunate precedent and makes these issues more difficult, not less.

My patient returns to clinic. He has spoken with his family and his wife. He wants an endovascular repair. Cognitive dissonance rages within me. I take a deep breath in, and begin….