The statistical illusion of better pregnancy outcomes for vaccinated women

How can a lower still birth rate in ‘vaccinated’ pregnant women mean that they are at higher risk of a still birth? It is all about what stage of their pregnancy they get vaccinated – there is ‘survivor bias’ in those vaccinated late in pregnancy.

The following analogy is based on one used by colleague Prof Retsef Levi. Suppose we want to find out whether runners in a 40 km (that’s 26-mile) marathon are more likely to finish the race if they are given a special vitamin drink during the race. To do this we set up a drinking station at 20 km where runners can pick up the drink if they wish. Suppose 200 runners start the race and we observe the following:

The first row represents the runners who do not take the drink and the second row represents the runners who do. So, there are 100 in each group; but, whereas 70 of those who take the drink complete the race (70%), only 48 of those who do not take the drink complete it (48%).

It seems clear that taking the drink increases the chances of completing the race. But this may not be the case.

The problem is that in marathons many runners drop out before the 20 km mark. That means that such runners not only fail to complete the race but also fail to reach the drink station (and so are recorded in the ‘no drink’ category). This means the summary data above may be ‘hiding’ something like the following data on such runners:

In other words, of those who get the option of taking the drink, the completion rate 70% (70 out of 100) of those who do so is lower than the completion rate 80% (48 out of 60) of those who do not. In this hypothetical example taking the drink therefore lowers the chances of completing the race, despite the ‘overall aggregate’ data seeming to point strongly to the opposite conclusion. It is the survivor bias of those who make it past 20km that was causing the original flawed conclusion.

This analogy is important because there has been much recent furore over conflicting government reports about whether the Covid vaccine is safe and effective for pregnant women and it turns out that similar statistical ‘illusions’ may be at play when comparing pregnancy outcomes of ‘vaccinated’ against ‘unvaccinated’ women. This is especially true of outcomes relating to whether the pregnancy resulted in the birth of a healthy baby. Ultimately what people really want to know is: if there are 100 pregnant women, how many end up with a healthy baby? Many of the studies over-complicate things and fail to answer that question. Imagine if we simply make the following replacements to the above marathon example:

• Starting marathon runners --> newly pregnant women

• Runners who complete the race --> those who delivered a healthy baby

• Vitamin drink at 20km --> vaccine at 20 weeks into the pregnancy

All the women reaching 20 weeks pregnancy must, by definition, have avoided early miscarriages** (which is when most pregnancy failures occur) and so the 'aggregate' results would suggest that vaccinated women are more likely to deliver a healthy baby than unvaccinated women. But again, the detailed result is that of those pregnant women who get to the option of vaccination, it is the unvaccinated who are more likely to deliver a healthy baby.

Of course, this is all hypothetical and over-simplified because pregnant women might get the option of vaccination at many different stages both before and during pregnancy and we have to take such information into account in order to arrive at a suitable risk assessment. But it turns out that this 'survivor bias' problem is still highly relevant to the real-world studies and data.

I have already explained in a previous article why the UKHSA’s regular vaccine surveillance reports are likely to be overestimating the safety of the vaccine because, instead of comparing the ‘never vaccinated’ with different categories of vaccinated women, they lump together the never vaccinated with those vaccinated pre-pregnancy into a single ‘no dose in pregnancy’ category and compare these with the ‘one or more doses in pregnancy' category. Here is an example of one of their graphs from the most recent report (and note this does not tell us anything about the chance a pregnant woman will give birth to a healthy baby since it deals only with still births** and not miscarriages in early pregnancy).

But things are much worse than I indicated in that article when it comes to comparing the overall chance of a successful birth per pregnancy, because the aggregated data ignore the massive confounding effect of women vaccinated late in pregnancy (i.e those who must have avoided early miscarriages** when most pregnancy failures occur).

Indeed, those claiming that the vaccine is safe and effective for pregnant women point to studies such as this which even seem to show the miscarriage and still birth rates of vaccinated women to be lower than that of unvaccinated women. But these studies do not properly factor in the stage during pregnancy when the vaccine was taken.

To understand the