Coronavirus: country comparisons are pointless unless we account for these biases in testing

Norman Fenton, Queen Mary University of London; Magda Osman, Queen Mary University of London; Martin Neil, Queen Mary University of London, and Scott McLachlan, Queen Mary University of London

Suppose we wanted to estimate how many car owners there are in the UK and how many of those own a Ford Fiesta, but we only have data on those people who visited Ford car showrooms in the last year. If 10% of the showroom visitors owned a Fiesta, then, because of the bias in the sample, this would certainly overestimate the proportion of Ford Fiesta owners in the country.

Estimating death rates for people with COVID-19 is currently undertaken largely along the same lines. In the UK, for example, almost all testing of COVID-19 is performed on people already hospitalised with COVID-19 symptoms. At the time of writing, there are 29,474 confirmed COVID-19 cases (analogous to car owners visiting a showroom) of whom 2,352 have died (Ford Fiesta owners who visited a showroom). But it misses out all the people with mild or no symptoms.

Read more:
COVID-19 tests: how they work and what’s in development

Concluding that the death rate from COVID-19 is on average 8% (2,352 out of 29,474) ignores the many people with COVID-19 who are not hospitalised and have not died (analogous to car owners who did not visit a Ford showroom and who do not own a Ford Fiesta). It is therefore equivalent to making the mistake of concluding that 10% of all car owners own a Fiesta.

There are many prominent examples of this sort of conclusion. The Oxford COVID-19 Evidence Service have undertaken a thorough statistical analysis. They acknowledge potential selection bias, and add confidence intervals showing how big the error may be for the (potentially highly misleading) proportion of deaths among confirmed COVID-19 patients.

They note various factors that can result in wide national differences – for example the UK’s 8% (mean) “death rate” is very high compared to Germany’s 0.74%. These factors include different demographics, for example the number of elderly in a population, as well as how deaths are reported. For example, in some countries everybody who dies after having been diagnosed with COVID-19 is recorded as a COVID-19 death, even if the disease was not the actual cause, while other people may die from the virus without actually having been diagnosed with COVID-19.

However, the models fail to incorporate explicit causal explanations in their modelling that might enable us to make more meaningful inferences from the available data, including data on virus testing.

We have developed an initial prototype “causal model” whose structure is shown in the figure above. The links between the named variables in a model like this show how they are dependent on each other. These links, along with other unknown variables, are captured as probabilities. As data are entered for specific, known variables, all of the unknown variable probabilities are updated using a method called Bayesian inference. The model shows that the COVID-19 death rate is as much a function of sampling methods, testing and reporting, as it is determined by the underlying rate of infection in a vulnerable population.

Therefore, different countries may appear to have different death rates, but only because they have applied different sampling and reporting policies. It is not necessarily because they are managing the virus any better or that the virus has infected fewer or more people.

With a causal model that explains the process by which the data is generated, we can better account for these differences between countries. We can also more accurately learn the underlying true population infection and death rates from the observed data. Such a model could be extended to include demographic factors, as well as social distancing and other prevention policies. We have developed such models for many similar problems and are currently gathering data required for populating the kind of model that we outline in the above figure.

Random testing

In the absence of community-wide testing, only random testing applied throughout the population will enable us to learn about the number of people with COVID-19 who are asymptomatic or have already recovered. Only when we know how many people don’t show symptoms, will we know the underlying infection and death rate. It will also enable us to learn about the accuracy of the tests (false positive and false negative rates).

Random testing therefore remains the most effective strategy to avoid selection bias and reduce the distortions in reported statistics. Ideally, this should be combined with a causal model.

Currently it seems there are no state-wide protocols in place in any country for randomised community testing of citizens for COVID-19. Spain did attempt it. But that involved purchasing large volumes of rapid COVID-19 tests, and they soon discovered that some Chinese-sourced tests had poor validity and reliability delivering only 30% accuracy – resulting in high numbers of false positives.

Read more:
COVID-19 tests: how they work and what’s in development

Countries like Norway have proposed introducing such tests, but there is uncertainty around how to legislatively compel citizens to test – and what might constitute an appropriate randomisation protocol. In Iceland, they have voluntary sampling which has covered 3% of the population, but this isn’t random. Some countries with large scale testing, like South Korea, might get closer to being random.

The reason it is so hard to achieve random testing is that you have to account for several practical and psychological factors. How does one collect samples randomly? Gathering samples from volunteers may not be sufficient as it does not prevent self-selection bias.

During the H1N1 influenza pandemic of 2009–2010, there was a lot of anxiety about the disease that created “mass psychogenic illness”. This is when hypersensitivity to particular symptoms leads to healthy people self-diagnosing as having a virus – meaning they would be highly incentivised to get tested. This could, in part, further contribute to false positive rates if the sensitivity and specificity of the tests are not fully understood.

While self-selection bias is not going to be eliminated, it could be reduced by running field tests. This could involve asking the public to volunteer samples in locations where, even in a lockdown state, they might be expected to attend and also from those in self-imposed isolation or quarantine.

In any event, when statistics are communicated at press conferences or in the media, their limitations should be explained and any relevance to the individual or population should be properly delineated. It is this which we contend is lacking in the current crisis.

Norman Fenton, Professor of Risk and Information Management, Queen Mary University of London; Magda Osman, Reader in Experimental Psychology, Queen Mary University of London; Martin Neil, Professor in Computer Science and Statistics, Queen Mary University of London, and Scott McLachlan, Postdoctoral Researcher in Computer Science, Queen Mary University of London

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Germany’s preparedness was helped in part by a recognition that coronavirus was likely to become a global problem. While most countries were slow in dusting off pandemic preparedness plans, in early January a scientist in Berlin, Olfert Landt, recognised the similarity to Sars and realised a test kit would be needed.

Lacking a gene sequence for the new coronavirus, Landt and his company designed their first test kit based on Sars and other known coronaviruses. The protocol was published by the WHO on 17 January, before the Chinese test. The British government passed on this test. By the end of February Landt had produced 4m kits and was making another 1.5m a week.

As well as having an effective test in mass production, Germany signed up politically to mass testing from the beginning, resulting in it being able to do 12,000 tests daily.

The UK government has promised it will dramatically increase coronavirus testing to 100,000 a day by the end of the month.

It has been criticised for not testing more people already, as countries like Germany have managed to reach 50,000 tests a day.

Daily coronavirus testing passed 10,000 people a day across the UK on Thursday 2 April – so how can capacity be increased tenfold in just four weeks?

What’s stopping the UK testing more people?

There are lots of different pieces to the puzzle when it comes to making large-scale testing work.

You need the laboratory space as well as enough, and the right kind of, machines.

You need the right reagents – highly specific substances used to extract the virus’s genetic material and to make it easier to study.

You need staff to take the swabs from patients’ noses or throats, and staff in labs to process the tests.

And you need the logistics in place to get samples from patients to labs.

We’re talking about diagnostic tests to find out if you have the virus here – ones that involve a nose or throat swab that has to be sent off to a lab.

Antibody tests use blood to look for evidence you’ve already had the virus – but these are unlikely to be available on a large enough scale in time to be part of the 100,000 target.

Image copyright Getty Images Image caption A drive-through testing centre for NHS staff has been set up in the car park of a branch of Ikea

Are these problems being addressed?

At first, only a small number of public health laboratories were being used to do coronavirus tests.

In the past fortnight, this was extended to a further 40 NHS labs around the UK.

Now the government is saying it will start to use the lab capacity of private companies to carry out coronavirus tests.

The UK has a large pharmaceutical and biomedical industry whose capacity could “easily” allow the country to do many more tests than it is now, according to Dr Rupert Beale, who has been involved in developing a diagnostic test for coronavirus at the Francis Crick Institute, which will be made available to NHS staff in the area. But so far, this industry hasn’t been tapped into, he said.

As well as labs, Health Secretary Matt Hancock said the government would now call on UK-based “pharmaceutical giants” GlaxoSmithKline and AstraZeneca to make the reagents necessary to carry out the tests.

The UK has faced problems getting hold of the relevant reagents, which have been in high global demand.

This has been a global problem, but some countries were in a better position than others.

Could the UK have increased testing sooner?

The decision was made to centralise the UK’s testing effort to a small number of public health labs, and this position only began to shift in the last couple of weeks.

Part of being able to scale up means “being willing to cede a little control over where, how and by whom the tests are conducted in order to increase capacity and decrease turnaround times”, according to Prof Eleanor Riley, an immunologist at the University of Edinburgh.

The two countries seen as the biggest testing success stories, Germany and South Korea, made many more labs available from an earlier stage – three times as many in Germany and two-thirds more in South Korea when adjusted for population size.

Image copyright Getty Images Image caption In South Korea, walk-in testing centres were quickly set up

There has also been a question of the availability of testing kits, including the reagents needed to study the virus once on a swab.

While some components have been in short supply worldwide, South Korea acted faster than the UK in developing a test, and was able to stockpile materials.

Germany benefited from home-grown diagnostics and manufacturing companies which gave it first access to raw materials and new technology.

Some of the disparities now are down to structural differences in countries’ testing capacity that pre-date the pandemic, according to Prof John Newton at Public Health England, who will oversee the government’s testing plan.

Massive virology labs were set up in South Korea in the wake of the 2003 Sars outbreak, he said, while Germany was already a large global testing base.

How have the government’s promises evolved?

At first, 10,000 tests were promised by the end of March, rising to 25,000 by the end of April.

Now the government has quadrupled its pledge to 100,000.

It didn’t quite meet its first pledge, breaking the 10,000 mark two days late on 2 April.

Germany still needs to increase its coronavirus testing rate dramatically if it wants to successfully manage the virus, scientists are advising the government.

The country has the capacity to carry out up to 500,000 tests a week, but must increase that to more than a million, or 200,000 tests a day, they say. This is despite other countries looking to Germany because of its comparatively high testing rate and relatively low death rate from Covid-19 of under 1%.

As the British government has struggled to tackle a testing shortfall, Germany has been repeatedly named as an example to follow in recent days, but a confidential scientific study commissioned by the government and recently leaked to German media showed that if it wanted to avoid a mass outbreak of the coronavirus, the country would have to greatly increase its testing capacity.

Ever since the government’s introduction of a strict lockdown, including closing the country’s borders and shutting down schools and nurseries, the scientists said around a million people in Germany – which has a population of just under 83 million – were likely to become infected, and around 12,000 would die. That outlook, however, was based on an increase in testing to 100,000 a day and within weeks to 200,000 a day.

The study’s authors said that in order to carry out testing in the most efficient and quickest way possible, it would also be necessary in the longer term to rely on the use of big data and location tracking via mobile phones, an issue that is controversial in Germany.

Lothar Wieler, the head of the government’s main public health advisory body, the Robert Koch Institute (RKI), said that while German testing was high compared with other countries, it would have to be speeded up. “We need to increase efficiency,” he said, adding that he welcomed the news that the use of quicker and simpler tests, in addition to the current standard PCR – polymerase chain reaction test, which directly detects the DNA or RNA of the pathogen – would soon be available.

Germany has a large network of more than 85 laboratories and a good history of detecting illness early. A nationwide network set up between doctors’ practices, normally used to monitor the flu, was also used for testing for Covid-19 from February onwards, and the fact the system only detected a case on 12 March indicates that the virus was not widespread before then..

There has been huge worldwide interest in why the death rate in Germany from Covid-19 has remained much lower than many other countries. Germany’s death rate is around 0.5%, compared with Italy’s 9.5%, and a worldwide rate of around 4.7%.

“The fact is that Germany started testing early on and has tested broadly. That way, many cases but also milder cases have been detected, and they did not generally include the elderly,” Wieler said.

It has, he said, in effect distorted the figures. He warns that the situation in Germany is likely to worsen, as many more of the elderly who have so far not been so affected, are likely to get sick. “We are still at the start of the epidemic, and it is completely unclear as to how it will develop,” Wieler said. “But of course the number of deaths in Germany will rise.”

He has insisted there is not, as has sometimes been suspected, an “under-reporting” of deaths from Covid-19 by hospitals. It would in theory be possible that some deaths were not counted as deaths from coronavirus, if the patient had an underlying health condition. But that is seldom likely to be the case, due to the sensibility towards the illness right now. Only if someone with coronavirus was killed by something unrelated to the virus, such as a car crash, would they not be included, medical authorities say.

Testing laboratories across Germany report that while they have been at pains to increase their capacity, they face many obstacles.

Germany’s federal system has contributed to an inefficient gathering of data – including an unclear picture of how much testing has been done – and delays in testing suspected cases, delivering results or informing contact people. Germany’s regional health bodies have recruited extra staff to help deal with the backlog, but say they are struggling to keep up. The RKI has recruited scouts – often medical students and trained doctors who arrived as refugees but are waiting to have their qualifications recognised – to support the system.

Accredited Laboratories in Medicine (ALM), the largest association representing laboratories in Germany working on the tests, has said that testing capacity has increased significantly since the first case emerged in the country at the end of January.

“We’re working around the clock, in a shift system, including at weekends,” Evangelos Kotsopoulos, head of ALM told Die Zeit newspaper. “Our main bottlenecks are staff – many of whom have young children and are affected by school and nursery closures – and supplies from industry. Around the world there are only five or six big suppliers for the materials that we need.”

Kotsopoulos said Germany should be far more focused on who it tested, to ensure those “who don’t have symptoms but just want to know whether they have it or not” are not tested, in favour of those such as at-risk patients for whom it is considered a necessity. “It is simply not practically possible right now to test everyone in a population of our size,” he said.

According to the RKI’s guideline, priority should be given to testing medical workers, people with underlying health conditions showing coronavirus symptoms and those in contact with those who are infected, so as to be able to place them under quarantine.

But many working on the testing frontline report concerns about a lack of materials and inefficiency, as well as a shortage of staff, forcing many people to work seven-day weeks.

“The reagents we need are produced by just three manufacturers in Germany and they have run dry,” Elisabeth Koerber-Kröll from the district medical faculty in Schwäbisch Hall, in Baden-Württemberg, south-west Germany, one of the hardest hit regions told broadcaster SWR.

Increasing emphasis is also being put on antibody tests, as well as antigen tests, as a more efficient way of managing the spread of the virus, in addition to self-testing, although many are sceptical that that this is reliable enough.