Paradoxes in the reporting of Covid19 vaccine effectiveness

The full pdf version of the following article (which includes the Appendix) can be found here.

Paradoxes in the reporting of Covid19 vaccine effectiveness

Why current studies (for or against vaccination) cannot be trusted and what we can do about it

Norman Fenton, Martin Neil and Scott McLachlan

Risk Information and Management Research

School of Electronic Engineering and Computer Science,

Queen Mary University of London

15 Sept 2021

The randomized controlled trials (RCTs) to establish the safety and effectiveness of Covid19 vaccines produced impressive results (Polack et al., 2020) but were inevitably limited in the way they assessed safety (Folegatti et al., 2020)[1] and are effectively continuing (Ledford, Cyranoski, & Van Noorden, 2020; Singh et al., 2021) . Ultimately, the safety and effectiveness of these vaccines will be determined by real world observational data over the coming months and years.

However, data from observational studies on vaccine effectiveness can easily be misinterpreted leading to incorrect conclusions. For example, we previously noted[2] the Public Health England data shown in Figure 1 for Covid19 cases and deaths of vaccinated and unvaccinated people up to June 2021. Overall, the death rate was three times higher in the vaccinated group, leading many to conclude that vaccination increases the risk of death from Covid19. But this conclusion was wrong for this data because, in each of the different age categories (under 50 and 50+), the death rate was lower in the vaccinated group.

Figure 1 Data from Public Health England, June 2021

This is an example of Simpson’s paradox (Pearl & Mackenzie, 2018). It arises here because most vaccinated people were in the 50+ category where most deaths occur. Specifically: a) a much higher proportion of those aged 50+ were vaccinated compared to those aged <50; and b) those aged 50+ are much more likely to die.

So, as shown in Figure 2(a), ‘age’ is a confounding variable. While it is reasonable to assume that death is dependent on age, in a proper RCT to determine the effectiveness of the vaccine we would need to break the dependency of vaccination on age as shown in Figure 2(b), by ensuring the same proportion of people were vaccinated in each age category.

Figure 2 Causal model reflecting the observed data

The Appendix demonstrates how this causal model, and Bayesian inference, can both explain the paradox and avoid it (by simulating an RCT). Using the model in Figure 2 (b), which avoids the confounding effect of age, we conclude (based only on the data in this study) that the (relative) risk of death is four times higher in the unvaccinated (0.417%) than the vaccinated (0.104%), meaning the absolute increase in risk of death is 0.313% greater for the unvaccinated.

An excellent article by Jeffrey Morris[3] demonstrates the paradox in more detail using more recent data from Israel.

Clearly confounding factors like age (and also comorbidities) must, therefore, always be considered to avoid underestimating vaccine effectiveness data. However, the conclusions of these studies are also confounded by failing to consider non-Covid deaths, which will overestimate the safety of the vaccine if there were serious adverse reactions.

In fact, there are many other confounding factors that can compromise the results of any observational study into vaccine effectiveness (Krause et al., 2021). By ‘compromise’ we mean not just over- or under-estimate effectiveness, but - as in the example above - may completely reverse the results if we fail to adjust even for a single confounder (Fenton, Neil, & Constantinou, 2019).

In particular, the following usually ignored confounding factors will certainly overestimate vaccine effectiveness. These include:

  • The classification of Covid19 deaths and hospitalizations. For those classified as Covid19 cases who die (whether due to Covid19 or some other condition), there is the issue of whether the patient is classified as dying ‘with’ Covid19 or ‘from’ Covid19. There may be diff