January 25, 2013

expert reaction to story on QALY system

A European Commission-funded study claimed the ‘quality-adjusted life years’ (QALY) system, used by NICE to recommend which drugs should be funded, is flawed.

Prof Sheila Bird, Programme Leader at the MRC Biostatistics Unit, University of Cambridge, said:

“The UK’s decisions on cost-effectiveness, for example by NICE, are indeed informed by valuations on quality and length of life.  These valuations were drawn up through a carefully-designed survey in 1993 of 3,395 interviewees (response-rate of 64%) from representatively sampled 5,324 UK addresses.

“This was a major undertaking by health economists at York University, and is relied upon still. Other nations, such as Canada, have copied the UK’s methodology to develop their own national valuations.

“The UK is already considering the need to update its valuations as they may have changed in the 20 years since they were first examined.  Whether they have changed enough to warrant the considerable research costs to re-examine them is a matter of judgement.

“However, the contrast is stark between how well these valuations were originally obtained and the description of how eccentrically 1,300 respondents – across several nations – were approached: not to elicit valuations, except for mobility, but to inquire about how risk averse they are. 

“It is plain common sense to know that individuals vary in how risk-prone/averse they are!  Indeed, that variation is precisely WHY national-applicable utilities, such as the UK’s, must properly represent the national spectrum and be averaged across it.  It needs good applicable science that has stood the test of time, and a robust methodology that has since been adopted by other nations; and that’s exactly what we have.

“This new project, apparently unpublished, seems to have a number of other specific flaws.

“It assumes that the system of valuations which grades different conditions cannot be said to reflect how real people view these conditions.  This is wrong.  The population-based valuations are precisely what we need to properly reflect the UK public’s view of the value to be placed on life in 243 different states of well-being/disability.

“It finds that most people are either risk averse or willing to take risk, and that it was rare to find someone neutral.  This is right but irrelevant.  Of course people vary, which is exactly why population-based valuations are applied and why the original York study used a representative sample of households.

“The project’s leader, Ariel Beresniak, proposes that valuations be made differently for each disease, and claims in his defence:  ‘The pro-QALY lobby will respond that such a system doesn’t compare different treatments for different diseases…But we never need to do this in real life medical situations.’  But, this comparison across diseases is precisely what NICE has to do!  And it is also what NHS commissioners will need to do with NICE’s help.

“All countries have to make difficult decisions about affordable treatments. The valuations underlying QALYs help us to do so on an objective basis, but judgment matters too.”

Prof David Spiegelhalter, Winton Professor of The Public Understanding of Risk, University of Cambridge, said:

“These type of criticisms are not new and do not invalidate what is done by NICE.  Of course the QALY approach is not perfect, but some mechanism is needed to provide consistent comparisons across different medical interventions, based on aggregate benefit and cost.  Otherwise the money could go to those with the most appealing emotional argument.”

Dr Adam Jacobs, Director, Dianthus Medical Limited, said:

“There are indeed a number of assumptions underlying the use of QALYs in health economic analyses. That is not a new revelation: all health economists are very well aware of those assumptions and the limitations they place upon conclusions drawn from QALYs. A good health economics analysis should make the assumptions explicit, and will also look at sensitivity analyses to find out what the effects would be if some of the assumptions turned out to be inaccurate.

“While QALYs are undoubtedly an imperfect methodology for assessing cost effectiveness of interventions, they are still useful, and I am not aware of any alternative methodologies that have been shown to perform any better.”

Prof John Cairns, Professor of Health Economics, London School of Hygiene and Tropical Medicine, & a member of the NICE Appraisal Committee for ten years, said:

“Given limited budgets we do need to compare different treatments for different diseases because agreeing to spend more on a particular treatment for a particular disease implies that there will be less available to spend on other diseases.  We need to be able to compare what we gain by spending in one area with what we lose by not spending in another area.  This new project’s suggested approach of using measures of outcome specific to particular diseases will not allow us to do that.

“QALYs are certainly not perfect and we should be looking for better ways of informing decision making but getting rid of an imperfect system without replacing it with a better one is not the way forward.”

Dr Andrew Walker, economist at the University of Glasgow, with ten years’ experience of reviewing new medicines at the Scottish Medicines Consortium (SMC), said:

“I am amazed it has taken these authors 3 years and EUR1m to establish what we already know, that QALYs are not perfect.  Anyone who makes decisions using QALYs and who cannot think of at least three issues with them is not thinking hard enough.  But as I explain QALYs to doctors and some patient groups around the country they can see they have a valuable role to play.  No measure is perfect.

“They give one example of where one organization has made a questionable decision: as someone who is pro-QALY I can easily say ‘I think NICE got that wrong’ without in any way believing we should scrap QALYs.  If the UK government is found to have got its figures wrong on debt reduction, do we try to scrap all statistics?  Of course not.

“As an alternative they propose cost per remission in arthritis, but I ask them to tell me how they define remission, how long remission lasts and how much we are willing to pay for one remission. What’s more they have set out four conditions the QALY must meet in full to be valid in their eyes – have they tested their ‘cost per remission’ approach against these same criteria?

“I can only speak for Scotland and the UK and with a limited health budget we do have to make decisions across diseases.  If we want to spend more on cancer medicines it has to come from somewhere.  The researchers speak as though there were no budget limits.”

ECH Outcomes Project.