select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
Public Health England (PHE) an Oxford University have conducted an evaluation of four commercially available antibody tests for COVID-19.
Prof Sheila Bird, Formerly Programme Leader, MRC Biostatistics Unit, University of Cambridge, said:
“The four immunoassays were each evaluated on a like-for-like basis using, identically, 994 known negative samples from 994 individuals (to assess specificity) and on 536 known positive samples from 536 individuals (to assess sensitivity). The evaluation was conducted in two sites (Abbott and DiaSorin tests at the John Radcliffe Hospital Clinical Biochemistry and Microbiology laboratories in Oxford; Roche and Siemens tests at PHE Porton Down) during the same 3-week period (May/June 2020). There is a common reporting standard (tables and plots, with 95% confidence intervals) for the primary results.
“Primary results are interpreted in the light of the UK Medicines and Healthcare products Regulatory Agency’s (MHRA) recently-released “Target Product Profile” for enzyme immunoassays, specifying i) clinical sensitivity of >=98% (95% CI: 96-100%) in confirmed SARS-CoV-2 positive cases (defined by RT-PCR) >= 20 days after the appearance of first symptoms; and clinical specificity of >=98% (95% CI: 96-100%) on samples collected more than six months before the first identified cases of SARS-CoV-2 infection. The Siemens SARS-CoV-2 Total (COV2T) met both criteria.
“Interestingly, seven known positive samples tested negative by all four assays. Antibody responses rose over the first 3-4 weeks from symptom onset, which the team evidenced because additional samples were available for testing at < 20 days (figure 5).
“Other secondary analyses together with the claim that “Antibody responses were sustained up to 73 days post symptom onset and up to 82 days post a positive PCR result” warrant a statistical alert.
“First, persistence of antibody response is more powerfully analysed within-person rather than between people. Larger between-person standard deviation tends to obscure changes.
“Within-person trajectories require a study design that set out to acquire and analyse serial samples from a sufficient number of known positive individuals. There is, however, merit in the between-person comparison too as surveillance studies may deploy these antibody tests in a series of monthly surveillance studies, each recruiting a different cohort of eligible participants.
“The second alert concerns the analysts’ exploration of adjusting the manufacturer’s assay threshold so that MHRA’s criterion for clinical specificity is just met – rather than comfortably met – with traded-off advantage in terms of clinical sensitivity. Any such data-led alternative needs to be validated independently in new data, as the next data-set may not “just” deliver!
“Analysts also explored SARS-CoV-2 antibody tests’ clinical sensitivity in confirmed positive cases >= 30 days after the appearance of first symptoms rather than MHRA’s >=20 days. The exploration was inspired by having observed that antibody responses rose over the first 3-4 weeks after symptom-onset.
“These statistical caveats apart in respect of secondary analyses, this is a much needed and sound piece of scientific work which will have important impact and sets a commendable standard.
“PS: Five tests were evaluated. The fifth was non-commercial Oxford Immunoassay. Four of the seven known positive samples which tested negative in all four commercial assays were positive by the Oxford immunoassay, see
Prof Sanjeev Krishna, Professor of Molecular Parasitology and Medicine, St George’s, University of London, said:
“The results of this evaluation are interesting but raise several issues, mostly concerning the length of time it has taken to produce the results for these tests. These evaluations could and should have been carried out more quickly.
“There is also the question of the standards used to assess these diagnostics, which were suggested as being unrealistic in terms of sensitivity and specificity when they came out. The tests assessed fall short of MHRA standards for false positives and false negatives, and yet are widely used in assays testing for infection, meaning there could be regulatory issues that need to be resolved.
“These findings highlight how important it is to have independent evaluations, carried out urgently and with transparent procedures. The tests can be improved in some cases, for example, by adjusting the cut-off values for positive and negative results, but these changes would also need to be evaluated.
“Amidst these often-confusing messages about antibody tests, there needs to be clarity about their best use. Are they most effective for screening the population, identifying those eligible for vaccine or plasma donation studies, or simply for diagnosing patients in clinical care? The requirements placed on each test and the evaluations needed will vary depending on their intended use.”
Prof Richard Tedder, Visiting Professor in Medical Virology, Imperial College London, said:
“I think this is a well conducted and very comprehensive study and I have no criticism of the presentation. It is without doubt fully comprehensive and the data is well presented, in addition the samples under which the assays have been scrutinised is appropriate and definitive.
“My concern is that users who read this must realise that the format of the four assays are based on different antigens. In the absence of a ‘gold standard’ assay (a single assay that does everything; almost all gold standards turnout at best to be tarnished brass), you cannot necessarily expect that an assay based on antibody to the nucleoprotein (Roche and Abbott) will compare absolutely favourably with an assay based on the envelope try spike (Siemens) or to an assay based on S1 and S2 (Diasorin) and the reciprocal applies. To expect absolute comparability against these three different proteins is naïve in the extreme. Each antigen and thus each antibody will have its own attributes. In addition antibody to nucleoprotein will have a different spectrum of genuine sero-positivity (Sensitivity) and false reactivity (Specificity) to those attributes displayed by antibody against the tri Spike (the full envelope protein) and against the S1 and S2 components of the envelope protein.
“The user, i.e. the purchaser and indeed in this case Her majesty’s government, needed to bear this in mind at the time decisions were made. Only two of the four assays (Siemens and DiaSorin) will detect antibody which is likely to neutralise the virus infectivity and which is likely in turn to confer a degree of protection against reinfection. This caveat must be borne in mind by those laboratories that are reporting nucleoprotein antibody alone – in such situations one can only infer that because a person who has nucleoprotein antibody in serum has indeed been infected by SARS CoV that their serum may have neutralising activity as well.
“Antibody to nucleoprotein is a good marker for previous exposure and infection by SARS CoV 2. It will have a degree of non-specificity because of reactivity with seasonal coronavirus serology, it will not be of use to define whether you have a good vaccine response. To look for a vaccine protective response, and by analogy to look for an immune ‘protective’ response you would not use nucleoprotein antibody testing but you would use spike or RBD antibody testing. This is neither better no worse for either assay – it is just you have to use the appropriate assay for the purpose intended.”
All our previous output on this subject can be seen at this weblink:
www.sciencemediacentre.org/tag/covid-19
Declared interests
None received.