How reliable are the results of a rapid Covid test?

The problem

A person is feeling sick and decides to take a rapid Covid test. Unfortunately, the test comes back positive.

A positive test result

The test is positive, but how certain can the person be that they are truly infected with Covid, i.e., how certain can they be that the test result is accurate? The same question could be asked if the test were negative: How certain is it in this case that the person is truly not infected with Covid?

First, we take a look at the package insert of the rapid test in the hope of finding answers to our questions.

The package insert of a commercial Covid rapid test

Let’s try to decipher the information. Apparently, a study was conducted in which samples were taken from individuals. These samples were subjected to an RT-PCR test and a rapid test of the CLUNGENE brand via nasal swab, and the results of both tests were compared for each person.

In this study, we are only interested in the accuracy of the rapid test. To do this, we assume that the result of the RT-PCR test is exact and that we know whether a person is infected with Covid or not. We then check whether the rapid test correctly identifies the health status of the individuals or not. From here on, we will refer to the Covid rapid test simply as test.

When we evaluate the result of such a test, we can first determine that we are dealing with four possible outcomes:

• The person is sick and the test is positive.
• The person is sick and the test is negative.
• The person is healthy and the test is positive.
• The person is healthy and the test is negative.

This can be presented as a table:

This table representation is called a confusion matrix.

Below the confusion matrix in the package insert, we find two percentage values: PPA and NPA. These were calculated from the values in the table and are quality measures for medical tests:

PPA stands for positive percent agreement or true positive rate, or sensitivity. The value indicates how many percent of confirmed positive Covid cases are correctly identified as positive by the test. In our test, it is on average 91.4%.

NPA stands for negative percent agreement or true negative rate, or specificity. The value indicates how many percent of confirmed negative Covid cases are correctly identified as negative by the test. In our test, it is on average 99.4%.

Why do test accuracy, sensitivity, and specificity not answer our actual question, which we see in the summary at the top right? We will look at this on the following page.

Analysis

To formalize the problem, we represent the procedure in the study as a tree diagram, which will later help us answer the actual question. We know who is healthy and who is sick and check for each group of people how the test turns out:

Tree diagram with events

We will first define two events and their negation to help us do the number work. They correspond to the possibilities mentioned above.

• \(S\): Person sick,
• \(\overline S\): person healthy,
• \(T\): test positive,
• \(\overline T\): test negative.

We can transfer these events to a tree diagram, together with the associated probabilities \(P\):

Tree diagram with events and their probabilities

As formulated in the question, we want to know the probability that a person is actually sick if the test is positive. This can be formulated using the above-defined events as conditional probability – the probability that a person is sick, given that the test is positive: \(P(S \mid T)\). This probability is also called positive predictive value, but it does not appear in the above tree diagram. There, the events \(S\) and \(T\) are reversed as in \(P(T \mid S)\).

\(P(T \mid S)\) is called sensitivity and is a quality measure for a test, as we have already seen in the package insert. Sensitivity indicates how high the probability is that the test shows a positive result for a person who is actually sick and is therefore also called true positive rate.

To calculate the actually sought positive predictive value, i.e., the conditional probability \(P(S \mid T)\), we can use Bayes’ Theorem:

\[ P(S \mid T) = \frac{P(T \mid S) P(S)}{P(T)}. \]

So what we need to calculate \(P(S \mid T)\) are three probabilities: \(P(S)\), \(P(T \mid S)\), and \(P(T)\).

\(P(S)\) indicates how high the probability is generally to be infected with Covid at this time – completely independent of a test result. This value is also called prevalence. We can never know this probability exactly, but we can take an estimated value that corresponds to the proportion of currently infected people in Germany. Let’s first set \(P(S) = 0.2\%\). This is roughly the prevalence during the first wave of Covid in spring 2020 in Germany.

One might also come up with the idea of reading \(P(S)\) from the study in the package insert, but this is not sensible in most cases: the study that led to the results in the package insert precisely examines how well the test works in sick people, so that people with proven Covid infection were specifically tested. This cannot be spoken of as a cross-section of the population.

The probability \(P(T \mid S)\) is already given in the package insert as sensitivity.

Finally, we are missing \(P(T)\): the probability that a person – whether sick or healthy – receives a positive test result. Here, the law of total probability helps us, which allows us to add the probabilities for the two possibilities that lead to positive tests (person sick and person healthy), since they exclude each other:

Tree diagram with both possibilities that lead to positive tests

Thus, we get

\[ P(T) = P(T \mid S) P(S) + P(T \mid \overline S) P(\overline S), \]

and thus

\[ P(S \mid T) = \frac{P(T \mid S) P(S)}{P(T)} = \frac{P(T \mid S) P(S)}{P(T \mid S) P(S) + P(T \mid \overline S) P(\overline S)}. \]

However, this brings us two new problems: we need \(P(T \mid \overline S)\) and \(P(\overline S)\). Fortunately, we can calculate both via the probability of complementary events. Thus, \(P(\overline S)\) (the general probability of not being sick) is easy to calculate, because we already know \(P(S)\) (the general probability of being sick, estimated using prevalence): \(P(\overline S) = 100\% - P(S) = 99.8\%\).

Finally, we are still missing \(P(T \mid \overline S)\): the probability of having a positive test result when one is actually healthy (a false positive result). Here, another quality measure for tests helps us: the specificity, also called true negative rate. It gives us the probability that a person receives a negative test result when they are actually healthy, so \(P(\overline T | \overline S)\). The specificity is often printed in the package insert of rapid tests, as we have already seen. As we can see in the tree diagram, \(P(\overline T | \overline S)\) is complementary to \(P(T | \overline S)\) and thus \(P(T | \overline S) = 100\% - P(\overline T | \overline S)\).

die <- 1:6
___(die)

die <- 1:6
mean(die)

confmat <- matrix(c(139, 13, 3, 462), ncol = 2)
colnames(confmat) <- c("sick", "healthy")
rownames(confmat) <- c("positive", "negative")
confmat

confmat <- matrix(c(139, 13, 3, 462), ncol = 2)
colnames(confmat) <- c("sick", "healthy")
rownames(confmat) <- c("positive", "negative")
confmat2 <- addmargins(confmat)
confmat2

confmat <- matrix(c(139, 13, 3, 462), ncol = 2)
colnames(confmat) <- c("sick", "healthy")
rownames(confmat) <- c("positive", "negative")
confmat

confmat <- matrix(c(139, 13, 3, 462), ncol = 2)
colnames(confmat) <- c("sick", "healthy")
rownames(confmat) <- c("positive", "negative")
confmat[1, 1] / sum(confmat[,1])

The influence of prevalence and test accuracy

Let’s take another look at the calculation of the conditional probability \(P(S \mid T)\):

\[\begin{align} P(S \mid T) &= \frac{P(T \mid S) P(S)}{P(T \mid S) P(S) + P(T \mid \overline S) P(\overline S)} \\[10pt] &= \frac{0.914 \cdot 0.002}{0.914 \cdot 0.002 + 0.006 \cdot 0.998} \\[10pt] &= 0.2339. \end{align}\]

This equation clearly shows the significant influence of prevalence on the overall result: Even with a high sensitivity of the test, the numerator \(P(T \mid S) P(S)\) and thus the overall result become very small when the prevalence is very low. This also makes intuitive sense: If there are very few cases of a disease overall, the probability that a person is actually affected by it, even if the test is positive, is comparatively low. In contrast, the probability is very high that a person is affected by a disease with many cases (i.e., high prevalence) regardless of how well the test performs.

We can also visualize this by plotting \(P(S \mid T)\) as a function of prevalence for fixed sensitivity and specificity:

We see how steeply the curve rises, and starting at a prevalence of 10%, we can really trust a test result with the given sensitivity and specificity. For a lower prevalence, this is less justified, as shown here in more detail:

On the other hand, it also demonstrates how important accurate tests are. In the case of low prevalence, specificity plays a particularly important role, as the following graphics illustrate:

By using this interactive graphic, you can illustrate the positive predictive value in relation to the prevalence for self-chosen sensitivity and specificity values:

The specificity \(P(\overline T \mid \overline S)\) has a great influence, because as we have already shown

\[\begin{align} P(S \mid T) &= \frac{P(T \mid S) P(S)}{P(T \mid S) P(S) + P(T \mid \overline S) P(\overline S)} \\[10pt] &= \frac{P(T \mid S) P(S)}{P(T \mid S) P(S) + (1 - P(\overline T \mid \overline S)) P(\overline S)}, \end{align}\]

and since \(P(S)\) is very small, \(P(\overline S)\) is very large and thus the specificity \(P(\overline T \mid \overline S)\) has the “power” over the denominator through the term \((1 - P(\overline T \mid \overline S)) P(\overline S)\). Thus, a high specificity causes a small denominator and therefore a high positive predictive value.

So we see: good tests (notably with high specificity) can make a big difference here especially at the beginning of an outbreak. They allow infections to be detected quite reliably early on, even if the prevalence is still low.

Other parameters and circumstances aside from prevalence and test quality also have a significant influence on the result. In particular, it is important how thoroughly the swab was taken. It is also known that temperature affects the accuracy of Covid rapid tests. So you see that the calculations can become much more complex!

npv_param <- expand.grid(prev = ___,
                         sens = ___)
npv_param

npv_param <- expand.grid(prev = seq(0.1, 0.9, length.out = 9),
                         sens = c(0.8, 0.91, 0.99))
npv_param

spec <- 0.994
npv_param$nvw <- ___
npv_param

spec <- 0.994
npv_param$npv <- spec * (1-npv_param$prev) / (spec * (1-npv_param$prev) + (1-npv_param$sens) * npv_param$prev)
npv_param

ggplot(npv_param, aes(___)) +
    ___ +  # "geom_..." für Linienplot
    theme_minimal()

ggplot(npv_param, aes(prev, npv, color = sens, group = sens)) +
    geom_line() +
    theme_minimal()