select search filters
briefings
roundups & rapid reactions
Fiona fox's blog
The Office for National Statistics (ONS) have released the latest report from the COVID-19 infection survey, looking at characteristics of people testing positive for COVID-19.
Prof Kevin McConway, Emeritus Professor of Applied Statistics, The Open University, said:
“These statistics are really not easy to interpret, for a combination of reasons. The most obvious, looking at the diagrams in the ONS bulletin, is that there’s much more statistical uncertainty in the more recent estimates than there was earlier, judging by the confidence intervals in Figures 1 and 2 in the bulletin and the numbers in the accompanying spreadsheet. That’s shown by the bars around the estimates, representing the confidence intervals or margins of error, getting considerably longer. There’s a good statistical reason why that happens – it’s because the number of people testing positive for the virus, in the ONS Covid-19 Infection Survey (CIS), has been falling pretty rapidly over time. When the numbers testing positive are low, there’s simply not as much data available to estimate the quantities involved, so the estimates are subject to more potential statistical inaccuracy. As an example, the data tables on the spreadsheet for people who travel to work by train or bus show that the number testing positive was 90 in October, then generally rose as infection levels rose, to 175 in January, but then fell again so that in the most recent 28-day period there were only 10 positive cases in people travelling by train or bus. You just can’t get very precise estimates from numbers that small. It’s true that the number of people in the CIS sample who were travelling by train or bus is higher for the most recent 28-day period than for the one before, but it’s still lower than it was in December (when there wasn’t such a strict lockdown).
“This level of statistical uncertainty means that what look like pretty large differences might be within the range that could occur just by chance. (In the jargon, they aren’t statistically significant.) That applies, for example, to what looks like a pretty huge fall in the risk of testing positive for people who travel by train or bus, in Figure 1 of the bulletin, comparing the latest 28-day interval ending on 16 April, with the one before that ending on 19 March. The confidence intervals overlap quite a lot. The data are compatible with the odds ratio for the later period being anywhere between about 10% higher than for the earlier period, or being about 80% lower, or anything between. It’s more likely to be somewhere in the middle of that range, so that it’s more likely that the risk has gone down than up, but really this range is so wide that you can’t say much at all. The same goes for comparisons between the risks of different means of transport. The data are compatible with the risk (odds ratio) of travelling by foot/bike/other being anywhere between over four times as high as for travelling by train or bus, or being more than a quarter less that the train/bus risk. Really the recent data tell us very little about that. I could make similar points about the comparisons by ease of social distancing in Figure 2 of the bulletin.
“The other reason for the interpretation being difficult is more subtle, but in the end more important. It’s because the data are observational, and observational data make it very hard to work out what causes what. There are many differences between people who travel to work, say, by train or bus, and people who travel by foot, bike, or other means, apart from just their means of getting to work. They may work in different places, in different types of jobs, live in different kinds of households, and so on. So any differences in the rate of testing positive, between people who use different transport means, may be caused by some combination of these other differences, and could even not be caused by the way they travel to work at all. In other words, if someone didn’t change anything in their life (same house, same family, same job) but changed the way they travel to work, it might make no difference at all to their risk of testing positive – or it might make a big difference. There’s just no way of telling from an observational study like this. Now of course ONS know this perfectly well. It’s possible to make statistical adjustments to try to account for some of the other differences between people who travel in different ways, and ONS did that by using a statistical model that included people’s age, gender, the region where they live, their ethnicity, the size of their household, a measure of the level of deprivation in their neighbourhood, and more. But this didn’t account for everything – in particular, it didn’t make an adjustment for the type of job people work in, it can’t allow for the different jobs that others in their household might do, and it can’t account for what they do when they aren’t at work or travelling to work. The adjustment makes it a bit clearer what might be going on, but it’s still true that these results can’t actually establish what causes what. That’s why the ONS bulletin repeatedly, and rightly, says things like “It is important to note that a broad range of factors including the general contact individuals have with others both in and out of work will contribute to their likelihood of testing positive. Transmission is therefore complex, and caution must be applied in over-simplifying the findings.” Normally the pattern of changes in risks over time can also give more clue on what’s happening, but the previous issue about the huge statistical uncertainty, when overall risks are low, gets in the way of that too.”
All our previous output on this subject can be seen at this weblink:
www.sciencemediacentre.org/tag/covid-19
Declared interests
Prof Kevin McConway: “I am a Trustee of the SMC and a member of its Advisory Committee. I am also a member of the Public Data Advisory Group, which provides expert advice to the Cabinet Office on aspects of public understanding of data during the pandemic. My quote above is in my capacity as an independent professional statistician.”