select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
There have been News reports that Public Health England (PHE) have said a new antibody test, developed by the healthcare company Roche, is “a very positive development” and that assessments at Porton Down, a PHE laboratory, found it was ‘highly specific’.
Prof Carl Heneghan, Director of the Centre for Evidence-Based Medicine, University of Oxford, and Dr Susannah Fleming, Senior Quantitative Researcher, Nuffield Department of Primary Care Health Sciences, University of Oxford, said:
“We have a piece that explains the results now online: COVID-19: Roche Antibody Test – 14th May, https://www.cebm.net/covid-19/covid-19-roche-antibody-test-14th-may/
“So, if we have 10,000 people in our population. With a prevalence of people who have had COVID-19 of 5%, that means 500 should have antibodies, and 9,500 shouldn’t.
“Being generous, and assuming Roche’s claim of 100% sensitivity, we know that all 500 of the people who should have antibodies will have positive tests. However, 0.2% of the people without antibodies (19 out of the 9,500) will also have a positive test.
“Assuming the 5% prevalence, of those people who have a positive test, 96.3% (500/519) will have a “true positive”. This is the positive predictive value.
“Because there are no false negatives, the negative predictive value is 100%.
“This isn’t as good as 99.8% and 100%, but it’s still pretty good. We can be pretty confident in both positive and negative tests. Most important in this case are positive tests since they might be used to decide if someone is safe to return to work or school, or to come out of shielding. However, around 4% (1 in 25) of positive tests may be false positives. It is important that people receiving the tests are aware of this.
“What we still don’t know is whether a positive antibody test is associated with protection from future COVID-19 infection. And because we haven’t seen the study methods yet we can’t tell you whether these set of results are reliable or not.”
Dr Colin Butter, Associate Professor and Programme Leader in Bioveterinary Science, University of Lincoln, said:
“The reported news that the new antibody test from Roche has appropriate sensitivity and specificity is very welcome news and finally adds a critical tool to the armoury of COVID testing that has relevance and value as we move to a lower level of lockdown. Potentially this will enable the identification of individuals that have been infected and may be immune. There are, however, some cautions and limitations that need to be understood.
“The assay is a variant of a standard laboratory technique known by its acronym “ELISA”. In this case the enzyme-linked immunosorbent assay is a sophistication of standard methodology and allows the detection of COVID-specific antibody. The specificity of the test is given as over 99.8%, meaning that only 0.2% of samples would produce a ‘false positive” result. Importantly, the testing of this included the use of samples from patients recovered from other coronavirus infections, including those that may result in the common cold: these did not react in the test.
“At 14 days after infection the sensitivity of the assay is given as 100%, meaning that no samples produce a “false negative”, that is to say every patient who has been infected with COVID will be positive for the test. Roche do report a confidence interval for these data, leaving open the possibility that a very few samples will produce an inappropriate result. The technique is less technically involved than that used to detect the virus itself (the “have I got it” test) and much faster to produce results.
“A limitation of the technique is that it is laboratory based, meaning that blood samples must be sent to a specialist facility, and that it relies on equipment that is proprietary to Roche. It will be important to understand how widespread this equipment is and how testing could be organised regionally and locally to provide a rapid and high throughput system.
“The relationship between antibody levels and immunity to COVID infection is not conclusively demonstrated, though evidence so far supports this relationship. The level of antibody may be important and the Roche test, unlike the lateral flow devices that have so far failed to meet PHE’s specifications, does allow for assessment of this.
“Provided that sufficient capacity is available, and that the logistics of testing can be well managed, this development could be important to a structured return to work.”
Dr Robert Shorten, Chair, Microbiology Professional Committee, Association for Clinical Biochemistry and Laboratory Medicine and Alexandra Yates, Director of Scientific Affairs, Association for Clinical Biochemistry and Laboratory Medicine, said:
Is this good news?
“This is potentially very good news.
What does ‘high specificity’ mean?
“Specificity is a measure of how certain we are that we can believe a positive result. If a test is 99% specific and we test 100 patients without a disease, the test will give a negative result 99 times out of 100. The remaining one is a false positive. A non-specific test could give false positive results due to cross-reactivity with other related viruses, such as normally circulating coronavirus that cause common colds. False positive results could make us believe that an infection has occurred when it hasn’t and could overestimate the prevalence of COVID-19 in a population.
What could a reliable antibody test help with? Do we know enough yet about whether the presence of antibodies definitely means a period of immunity?
“A reliable test can help us to understand how many people in a population have been infected. This could be helpful in choosing how to develop strategies for the next phases of the outbreak. We do not know enough about COVID-19 to say if detectable antibodies are sufficient to give immunity against future infection, and if there is immunity, for how long it will last.
Is this a home test or a lab test?
“This test will be performed in a laboratory using a sample of blood taken from patients by a healthcare professional. This test has the potential to be analysed rapidly in large numbers in existing NHS Hospital Laboratories across the UK, by registered Healthcare Scientists.”
Prof Matthew Baylis, Chair of Veterinary Epidemiology, University of Liverpool, said:
“The news of a highly accurate antibody test, developed by Roche and now endorsed by Public Health England, is very welcome. Reliable tests will be an essential part of the armoury required to fight COVID-19. The antibody test is designed to tell if an individual has previously had the disease – the presence of antibodies to the virus is a marker for past infection. However, there appears to be some mis-reporting of the test’s accuracy, with the media reporting it as 100% specific, while the Roche website describes it as 100% sensitive and only ‘over 99.8%‘ specific. Quoted statements by John Newton at PHE also imply they found it to be 100% sensitive, although he used the term “highly specific” which may explain the confusion. This distinction is extremely important. If the test were indeed to be 100% specific, then a positive test result would be entirely reliable – the individual would be confident they have recovered from COVID-19. Roche’s reported test specificity of 99.8% still sounds impressive, but it is less useful for a disease of low prevalence, as COVID-19 may be. If the prevalence of people in the UK who have had COVID-19 is 0.34% (i.e. about 229,000 cases out of a population of 66.5 million) then a randomly chosen person who tests positive has a more than one-in-three chance of being a false positive – in reality, they have not got antibodies to COVID-19, and remain susceptible to the disease.
“This one-in-three comes about as follows. Suppose we test 10,000 randomly selected UK samples for COVID-19 antibody using a test that is 100% sensitive and 99.8% specific. If the prevalence is as low as 0.34%, then we expect there to be 34 positive samples among the 10,000; and as the test is 100% sensitive, all of these will give positive results. Now, because the prevalence is low, most of the samples (an expected 9,966) are negative. When these are tested with a test with specificity of 99.8%, most will show up as negative (an expected 9,946) but 20 of them incorrectly give positive results. So, out of the 10,000 samples tested, we expect to get 54 positive results in all, of which only 34 are true positives, and 20 are false positives. There is more than a one in three chance that a positive test result is from a negative sample.
“The critical question here is what is the prevalence of antibody in the UK population? If very low, such as under 0.5%, then positive test results would be fairly unreliable because there will also be a number of false positives. If, however, the prevalence of antibody in the population is higher, say 5% – and this is the sort of value that some models give – then the situation would be very different. With 5% antibody prevalence, true positive results swamp the false positive results: over 96% of positive test results would now be expected to be from positive samples, indicating much greater reliability of positive results. If the antibody prevalence is even higher, say 10%, then now over 98% of positive results will be from positive samples. This illustrates how interpreting the results of the new antibody test depends critically on how many people have already developed antibody to COVID-19.
“The lesson here is that it is important to not be bamboozled by the high accuracy levels given for diagnostics tests. They need to be considered from the perspective of the user who does not know in advance if the sample is positive or negative (and for whom the question is, what does a positive or negative test result mean?). This is affected by the prevalence of the disease in the population. Epidemiologists use measures called ‘predictive values’ to make these assessments.
“It is debatable whether a test for which one in three positive results are unreliable is as much of a game changer as is being reported. Combining information from test results with evidence of a person having had clinical signs of COVID-19 will be better. Also, the reliability of a positive test result will be greater if the true prevalence of people who have had COVID-19 is actually higher than we think; and of course, this prevalence will inevitably increase over the coming weeks, increasing the reliability of a positive test result as time goes on.”
Prof Daniel Altmann, Professor of Immunology, Imperial College London, said:
“This represents a massive step forward in the push to narrate the natural history of this infection. Without good mapping of how many are immune, it’s virtually impossible to understand how much the virus has spread, what is the story linking exposure, immunity and chances of protection against reinfection, and thus, progress to genuine, protective immunity. The fact that the search for this test seems to have been such an unexpectedly long haul to reach this point simply seems to speak to the fact that some things that would once have taken years of tweaking and optimising can zoom forward quickly given the backing and the will to do it, while some things in research simply take time to optimise from a standing start.”
Prof Lawrence Young, Professor of Molecular Oncology, University of Warwick, said:
“This antibody test is a significant development. It will allow us to identify who has been infected in the past rather than the current swab test which detects individuals who are currently infected and shedding the virus. Evidence from other coronavirus infections in humans and animals suggests that these antibodies are protective against reinfection but we don’t know this for sure in relation to the COVID-19 virus. A number of studies have identified antibodies in infected patients which block the ability of the virus to infect supporting a protective role for these antibodies. Clinical trials using convalescent plasma, that’s blood from patients who have recovered from COVID-19 and have antibodies against the virus, have shown some exciting early results in COVID-19 patients. A better understanding of the immune response to the SARS-CoV-2 virus will help us to develop effective vaccines. This new antibody test will be an important weapon in fighting the virus. The hope is that this antibody testing, which is currently hospital based, could be developed into a simple home test using a finger prick of blood. This would allow more widespread testing in the community and provide us, for the first time, with a true indication of who has been infected.”
Prof Ravi Gupta, Professor of Microbiology at the University of Cambridge, said:
“The news of a reliable antibody test is welcome. The Roche test is unlikely to give false results based on the information available, but it needs to be done in a dedicated hospital or central laboratory on specialised machines and therefore turnaround time for results is likely to be at least a day or more. It is therefore less useful in real time diagnosis in hospital for patients in the immune phase of disease. However, the test will be useful for measuring levels of antibodies in the general population and in health care workers. It will be important in our efforts to understand whether previous SARS-CoV-2 exposure confers protective immunity.”
Prof Richard Tedder, Visiting Professor in Medical Virology, Imperial College London, said:
“A number of manufacturers already have large platforms capable of detecting antibody and in this respect the development by Roche, is neither surprising nor remarkable. Just how sensitive and just how specific it is remains to be seen when this platform is used routinely. I find it surprising that PHE appear to have given this preferential treatment and publicity over and above the existing platforms by other companies. That said any assay, whether it be on a big platform such as this system or on the more conventional and widely used microtitre plates in the large number of virology laboratories who run such immunoassays, has the potential to provide a method for determining the prevalence of antibody and therefore perhaps a degree of resistance to reinfection in the population of this country. It will be interesting to see if this assay by Roche identifies accurately those individuals whose antibody is likely to neutralise the virus infectivity.”
Prof Carl Heneghan, Director of the Centre for Evidence-Based Medicine, University of Oxford, said:
“On the 13th of March, the Roche statement said “Negative results do not preclude SARS-CoV-2 infection and should not be used as the sole basis for patient management decisions. Negative results must be combined with clinical observations, patient history, and epidemiological information.”: https://www.roche.com/media/releases/med-cor-2020-03-13.htm
“Which has gone from this to the 3rd of May statement that “the serology test has a specificity greater than 99.8% and sensitivity of 100% (14 Days post-PCR confirmation).”: https://www.roche.com/dam/jcr:409cf75c-a33a-48f8-8a48-c7d9a71fcfab/en/03052020-rochemediareleae-elecsys-covid19.pdf
“Without seeing the study methods and the data it’s impossible to verify these claims of accuracy.”
Dr Alexander Edwards, Associate Professor in Biomedical Technology, Reading School of Pharmacy, University of Reading, said:
“This antibody test from Roche makes use of their standard laboratory immunoanalysers. These are large and sophisticated instruments that automate the whole process from sample through to result. Because automatic immunoanalysers have been used for so many years to run large numbers of different clinically important blood tests, it is not surprising to see this product coming to the market. We would expect to see a range of such ‘blue chip’ diagnostics products being approved in the coming weeks.
“The specificity of the test is really important because if you test very large numbers of individuals, you don’t want to get false positive test results because those individuals may erroneously be informed they have already been infected. At the moment we don’t know if having antibody will fully protect from infection and for how long, so a specific test still doesn’t answer all the questions. But false positives are still always a concern so it’s vital to know that such laboratory antibody tests can achieve this accuracy.
“The next challenge is to scale up not only the numbers of these antibody tests that are run in the labs equipped with the analyser instrument, but a full system that makes use of the tests to survey populations and accurately map where the virus has spread. We must not forget these instruments are already being used for many different essential blood tests – if we switch to running COVID-19 tests we must make sure other patients don’t suffer from not having essential tests.
“Also, these surveys using this lab test will require blood samples to be taken from large populations, which is perhaps even harder than taking swabs. This is why alternative methods of taking blood are being explored, such as people taking a fingertip blood spot onto a card and posting the dry blood back to the laboratory could be useful. Even if the lab test is very accurate, we still need to ensure that such home sampling is equally accurate.”
Prof Matt Keeling, Professor of Populations and Disease, University of Warwick, said:
“This could be a complete game-changer. The key is the very high sensitivity and specificity.
High sensitivity means that if you’ve previously had Coronavirus the test will correctly identify this; this likely means that such individuals are at lower risk of catching the virus again.
“High specificity (and a figure of 100% is being quoted) means that if you haven’t had Coronavirus the test will correctly identify this. This is also important, as any “false positives” (given a positive reading when they haven’t had the virus) could incorrectly assume they have had the virus and therefore are a lower risk.
“Previous tests had much lower sensitivity, which meant that while population level results could be scaled to give a national picture, they were not accurate enough for individual patients.
“The key question remains about how protective antibodies (which this test detects) are in the long and short term – but now we have an accurate test we can start measuring this. If a positive test implies immunity, then this potentially offers a way out of lock-down or at least a way of sheltering susceptible individuals from the most risky situations – e.g. deploying test positive (and hence immune) health care workers to look after COVID-19 patients.”
Prof Sheila Bird, Formerly Programme Leader, MRC Biostatistics Unit, University of Cambridge, said:
“Highly specific antibody testing opens is important for point-of-care and a bonus for surveillance.
“However, details matter. Hence, for the independent evaluation at Porton Down, may we please know the study-design, including how masking of the tested sera was achieved, cross-checking against other tests (if any) and the number of sera tested from persons who had i) never been exposed to coronavirus-2, ii) been hospitalized with COVID-19 disease and had recovered, iii) tested antigen positive for coronavirus-2 but experienced only mild symptoms, iii) tested antigen positive for coronavirus-2 but remained asymptomatic, iv) been exposed to a different coronavirus.
“Meta-analysis of the results from evaluations conducted by Roche and independently at Porton Down should give consistent and precise overview of specificity and sensitivity of the antibody test, both with 95% confidence intervals. Results should be reported publicly in a statistically appropriate manner for this important breakthrough.”
Prof Arne Akbar, President of the British Society for Immunology, said:
“Antibody tests are difficult to develop and news that Public Health England have approved the test from Roche is definitely a positive step forward. A good antibody test needs to be extremely accurate to make sure that it both correctly identifies all people who have these antibodies and that it doesn’t give ‘false positives’ or incorrectly identify someone as having been exposed to COVID-19. Public Health England will have carried out extensive examination on this test to look at these factors and it’s reassuring to see them give this the green light.
“However, developing a reliable antibody test is only the first step on the road to us being able to understand how any immunity against COVID-19 works and being able to use this knowledge to manage the spread of the disease. A positive result to a reliable antibody test will tell us if that person has encountered the virus in the past and mounted an immune response. However, it does not tell us if these antibodies will stop you getting sick from COVID-19 in the future or how long any protection generated might last – these are both factors that we do not yet know about COVID-19. Additionally, the immune system is extremely complex and there are lots of ways that it can generate immunity, antibodies being only one. We need to fully understand what ‘effective immunity’ against COVID-19 comprises to be able to better plan our response to managing the spread of this disease.”
All our previous output on this subject can be seen at this weblink:
www.sciencemediacentre.org/tag/covid-19
Declared interests
Prof Arne Akbar: “Professor Akbar receives grant funding from the BBSRC, MRC and Dermatrust.”
None others received.