This article is reproduced by kind permission from Drowning in Data: https://drowningindatadotblog.wordpress.com/2021/01/24/covid-19-deaths-comparing-age-groups/
A feature of all medical tests is that they aren’t 100% accurate. This can be described using the sensitivity and specificity of each test. This means that sometimes a test will give a positive result when it should have given a negative result (a false positive) or a negative result when it should have given a positive result (a false negative). False negatives can be more dangerous as that means people will go out into the community thinking they don’t have an infection (after all, the test was negative) and carry on spreading it. However, if there are lots of false positives compared to true positives, we end up spending lots of money treating people who don’t need treatment and it causes stress and worry to more people.
There’s an excellent paper on how to interpret a COVID-19 test result published by the British Medical Journal. The paper shows in an interactive way the effect of changing sensitivity and specificity on the results.
One other factor that determines how many true and false positives are produced is the prevalence. This is the percentage of people in a population who truly have the condition that is being tested for.
The Office for National Statistics (ONS) publishes prevalence data based on random sampling of several thousand Scottish households. Each person is tested several times and the data is used to model the likely prevalence in the rest of the community. This modelling is considered valid only for the spread of COVID-19 in the community, as care homes and hospitals are likely to have different prevalence rates. The modelled community prevalence results are reported from 13 December 2020 onwards and are graphed below.
Prevalence is around about 1% in the community in Scotland, and has been since the start of the ONS surveys in early December. The large grey area shows the 95% confidence interval – the true prevalence value is highly likely to be within the grey area on the graph.
So we have the prevalence for community (Pillar 2) testing in Scotland. How do we then calculate the number of false positives and true positives?
RT-PCR (Reverse-Transcription Polymerase Chain Reaction) testing is used across the UK as a screening test for COVID-19. RT-PCR was extensively studied by Public Health England to calculate its likely sensitivity and specificity. Several PCR machines from different manufacturers were compared to give a range of sensitivity values from 90-100% and specificity values from 97-100%. Note that a specificity of 100% means that no false positives would be identified by the test. This is highly unlikely in practice – it would be interesting to know what kinds of benchmarking tests are used on the PCR machines used across the country. (If anyone can tell me if benchmarking studies are published on the PCR tests used in Pillar 2 testing, please get in touch!)
To estimate the number of false positives and true positives, I did the following:
- I took the range of ONS prevalence values for each day (using the lower, median and upper values)
- I calculated the number of false positives based on a specificity of 98.5% (this was the middle of the range of specificities reported by the Public Health England group) using the British Medical Journal calculations shown above
- I calculated the number of true positives based on a sensitivity of 95% and the number of tests carried out each day from the NHS OpenData website using the same calculations.
- I then graphed these values on a stacked bar graph (shown below).
These values are estimates based on the ONS prevalence data. The ONS data has a range of possible values based on their modelling and experimental sampling. The graph shows that for many of the days in the period 13 December 2020 until 23 January 2020 there were between 1.5 and 2x more false positives than true positives. Here’s the multiple of false positives to true positives graphed for the same range of dates.
So for a specificity of 98.5% (which is really high) and prevalence between around 0.75 and 1.1% there are sometimes up to twice as many false positives as true positives. The real number of false positives could be higher (because the performance of PCR testing across the country isn’t as high as the machines tested in one of the best labs in the UK) or lower (because the true prevalence is higher than that estimated by the ONS). However, these calculations show that even with a high performing screening test, if there is very low prevalence, there will be many more false positives than true positives.
Figures for COVID-19 cases reported in the media are often reported as absolutes. However, as I have shown here, there is always uncertainty with any measurement. And as prevalence has been so low, the number of false positives is larger than the number of true positives.
What does this mean for community COVID-19 screening in Scotland?
What is an acceptable level of false positives? Would the Government make the same decisions and restrictions if they knew that there were so many false positives?
How would you feel knowing that you’re more likely to have a false positive result than a true positive if you participate in Pillar 2 testing?
For more articles and data sets please visit Drowning in Data: https://drowningindatadotblog.wordpress.com/