select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
The Lancet have published an expression of concern for a study previously published into hydroxychloroquine and chloroquine in COVID-19 patients.
Prof Martin Landray, Professor of Medicine & Epidemiology, Nuffield Department of Population Health, University of Oxford, said:
“I welcome the statement of The Lancet to investigate the integrity of this paper and await the conclusions with interest. However, there is an even bigger issue at large. Even with perfect data and thorough, reproducible analyses, these kind of observational studies are a wholly inadequate method for making important decisions about the benefits and harms of possible treatments for COVID-19. Patients who happen to get a particular drug (in this case hydroxychloroquine) are different from those who did not. Some of those differences are obvious and can be taken into account. But many are either not obvious, not measured, or not measured well, meaning that any conclusions are inherently unreliable.
“For all such studies we are left with the conclusion, ‘We need evidence from well-conducted randomised controlled trials.’
“And yet the reaction of many, including some regulators and renowned clinical commentators, was exactly the opposite – instructing trials to pause recruitment or dissuading doctors to enrol patients. We simply don’t know whether hydroxychloroquine will turn out to be an effective treatment for COVID-19 or not. This is a drug that is very widely used for a disease that kills hundreds of thousands of people, but on the basis of no good evidence. The sooner we get answers from randomised controlled trials the better. If it turns out hydroxychloroquine is effective for COVID-19, then let’s use it; if not, let’s abandon it. But this is not a time for speculation and certainly not a time to hold up the very trials that will give us the knowledge we need.”
Prof Peter Horby, Professor of Emerging Infectious Diseases and Global Health in the Nuffield Department of Medicine, University of Oxford, said:
“I welcome the statement from the Lancet, which follows a similar statement by the NEJM regarding a study by the same group on cardiovascular drugs and COVID-19. The very serious concerns being raised about the validity of the papers by Mehra et al need to be recognised and actioned urgently, and ought to bring about serious reflection on whether the quality of editorial and peer review during the pandemic has been adequate. Scientific publication must above all be rigorous and honest. In an emergency, these values are needed more than ever.
“The Lancet publication by Mehra et al has had major adverse impacts, resulting in the suspension of numerous well-designed clinical trials. This is completely unjustified. Even if the results were correct, observational data such as this, with its inherent weaknesses, should not be used to stop trials which will provide definitive and actionable answers.”
Dr Stephen Griffin, Associate Professor in the School of Medicine, University of Leeds, said:
“Serious concerns have been raised by both the Lancet and NEJM regarding a corporate provider of medical data, Surgisphere, which provided data used in two influential COVID19 studies; I will focus here upon the Lancet study. This paper had profound impact in leading the WHO to remove treatment arms from its own trials, and to recommend the cessation of off-label use for the investigational drugs chloroquine and hydroxychloroquine. These are often combined with macrolide antibiotics such as azithromycin in SARS-CoV2 infected patients, and concerns arose from the Lancet study based upon the reported increased mortality associated with these treatments, related to cardiac arrhythmia. In addition to journal-led investigations, authors not affiliated with Surgisphere are undertaking an independent data audit for the Lancet study.
“This development poses grave issues on several fronts. First, doubts regarding data integrity severely undermine the peer review process and pose a fundamental threat to the advancement of scientific and clinical understanding amidst this devastating pandemic. Second, the withdrawal of these drugs from various clinical settings based upon what may be dubious mortality data could deprive COVID patients of a potentially effective therapy. Lastly, we must ask is it feasible, or indeed fair, for individual peer reviewers to assess such studies, as they necessarily depend upon intrinsic trust and cannot logistically validate such enormous data sets.
“However, such concerns may also serve to invalidate what may otherwise be an important COVID19 study. Whilst this was not an RCT, the size of this retrospective study leant considerable weight to the findings. Nevertheless, the increase in mortality within treatment groups was unexpected, despite these drugs being known to affect QT intervals in a proportion of patients. Notably, recruitment was halted in a trial comparing high and low dose chloroquine in COVID patients due to adverse effects in those receiving the higher dose, including QT elongations and increased mortality; only ~80 from more than 400 enlisted patients had been treated at this point (doi:10.1001/jamanetworkopen.2020.8857). By contrast, the majority of studies have not reported increased mortality, albeit usually in small numbers of patients.
“Previous to the Lancet study, another retrospective study of ~1400 patients in New York State failed to detect increased mortality associated with hydroxychloroquine therapy (DOI: 10.1056/NEJMoa2012410). However, the main finding of this smaller study was that no therapeutic benefit was apparent, in agreement with the Lancet paper. Thus, the question of efficacy when using these drugs remains unanswered, with the balance of evidence possibly indicating relatively little effect. It will require placebo-controlled RCTs to finally establish both the safety and effectiveness of these drugs, such as the ongoing RECOVERY trial, which naturally places major emphasis upon patient safety. Until these data are available and the outcomes of Surgisphere investigations are known, the true clinical usefulness of these repurposed drugs remains unknown. It is therefore prudent that they currently remain assessed only within the context of RCTs and that their off-label use is restricted.”
Prof Babak Javid, Principal Investigator, Tsinghua University School of Medicine, Beijing, and Consultant in Infectious Diseases at Cambridge University Hospitals, said:
“On 22nd May The Lancet published an article describing a very large retrospective observational cohort of moderately to severely ill Covid-19 patients, a substantial proportion of whom had been treated with the investigational Covid drugs hydroxychloroquine or chloroquine. The study, which was not of randomised controlled trials, suggested that either agent was associated with a large (about 30%) increased risk of death in this context. The study was impressive in that it claimed to draw on a database of almost 100 thousand Covid patients from many hospitals from all over the world. Therefore, even though it was a retrospective analysis, the results carried a lot of weight. In fact, on the basis of this study’s findings, several high profile clinical trials of hydroxychloroquine for Covid were halted. However, in the last few days, there has been mounting concern that question the validity of the data, which were gathered from a database provided by a private company, Surgisphere. For example, the number of Covid cases that were supposed to be from a subset of Australian hospitals was actually greater than the sum total of cases in Australia reported at the time. Several other inconsistencies have also emerged. The raw data have not been made available for general scrutiny. Now, the Lancet journal has released an “expression of concern” with regards to the study. A sister study looking at heart medications and Covid also using Surgisphere data, published in the equally high profile New England Journal of Medicine has also been called into question in recent days. An independent audit of the data is now ongoing to determine their validity.
“In many ways, the harm has already been done: no high quality trials of hydroxychloroquine for Covid have yet reported, and some may now be unable to recruit sufficient patients to arrive at an answer. It should be noted that the expression of concern does not, in itself, mean, at this time, that the findings of the study are incorrect. However, the lack of data transparency does call into question how expert reviewers can be expected to fully assess the validity of this and similar studies.”
Prof Stephen Evans, Professor of Pharmacoepidemiology, The London School of Hygiene and Tropical Medicine, said:
“Doubts have been cast on the integrity of the data from this study. A correction has been published, but there were concerns that went beyond this. An expression of concern has now been published by The Lancet Editors who are awaiting the results of an “independent audit of the provenance and validity of the data has been commissioned by the authors not affiliated with Surgisphere”.
“A previous publication in The New England Journal of Medicine by the same first author with two other authors in common with The Lancet paper has had the following: “Expression of Concern: Mehra MR et al. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19. N Engl J Med. DOI: 10.1056/NEJMoa2007621.” The NEJM Editor-in-Chief says “Recently, substantive concerns have been raised about the quality of the information in that database.”
“The NEJM paper looked at cardiovascular risk factors and drug treatments, noting especially that ACE inhibitors had a dramatically protective effect on mortality while ARBs had a null effect. Some previous commentators had suggested that both classes of drugs could be harmful, but the overall evidence of their harm (or benefit) is limited.
“It is clearly vital that the audit of the Surgisphere data is truly independent and the auditors should be approved by the relevant journals and possibly also the FDA in the US which has extensive experience of auditing clinical data.
“There are definite doubts about the integrity of The Lancet paper. In retrospect many readers and decision makers may well have placed too much reliance on that paper.
“There were queries raised in previous Science Media Centre comments1 about the methods of analysis and some of the results did not fit with what is known about other risk factors, but the subsequent scrutiny of the paper by others has presumably led to this expression of concern.
“The adverse effects of both hydroxychloroquine and macrolide antibiotics on QT interval, and hence on ventricular arrhythmias, claimed in the paper was expected. The overall adverse effect on mortality was stronger than might be expected. Even if the data in The Lancet paper are proved to be unreliable, it remains true that the overall picture in relation to benefits and harms of hydroxychloroquine with or without macrolide antibiotics remains uncertain.
“There is no doubt that it is the randomised trial data that will provide more reliable answers. It may still be reasonable to carry out an interim analysis across as many trials as possible of the potential benefits and harms of these drugs, but it is possible that some of the trial investigators will not release data until their own data monitoring committees have formed a view of their data.
“It remains the case that it is the randomised data that will elucidate the questions.
“The Annals of Internal Medicine have an ongoing published, but regularly updated (most recently on 27 May), review: “Hydroxychloroquine or Chloroquine for Treatment or Prophylaxis of COVID-19: A Living Systematic Review”, https://www.acpjournals.org/doi/10.7326/M20-2496
“This has summarised data from randomised trials and from observational studies.
“They did not include The Lancet paper in their review and currently conclude “there is insufficient and often conflicting evidence on the benefits and harms of using hydroxychloroquine or chloroquine to treat COVID-19. As such, it is impossible to determine the balance of benefits to harms. There are no assessments of hydroxychloroquine or chloroquine for prophylaxis against COVID-19”.
“It is clear that the large randomised trials that are ongoing may be sufficient to provide reliable answers, but with the current falls in the numbers of cases of Covid-19 and deaths from it in many countries the full answers could still be delayed.”
https://www.thelancet.com/lancet/article/s0140673620312903
All our previous output on this subject can be seen at this weblink:
www.sciencemediacentre.org/tag/covid-19
Declared interests
Prof Martin Landray: “Co-chief investigator of the RECOVERY trial of potential treatments for COVID-19 (funded by UKRI and NIHR). Research funding to University of Oxford received from Novartis, Boehringer Ingelheim, and Merck Sharp & Dohme.
Infrastructure and core funding received from Health Data Research UK, NIHR Oxford Biomedical Research Centre, UK Biobank Ltd, MRC Population Health Research Unit, and British Heart Foundation Centre for Research Excellence. Employee of University of Oxford with salary supported by Li Ka Shing Foundation, Health Data Research UK, NIHR Oxford Biomedical Research Centre, Wellcome Trust, and National Health Service.”
Prof Peter Horby: “Chief Investigator of the RECOVERY trial.”
Dr Stephen Griffin: “No conflicts.”
Prof Babak Javid: “None.”
Prof Stephen Evans: “No conflicts of interest. I am funded (1 day/week) by LSHTM. They get funding from various companies, including Astra Zeneca and GSK but I am not funded by them, I have no involvement in obtaining funding from them and I am not an investigator or any grants obtained from them. I am the statistician to the “meta-Data Safety and Monitoring Board” for CEPI [https://cepi.net/]. I will probably be paid for my attendance at meetings and expenses for travel.”