How Wrong Was Ioannidis?

This video was censored on YouTube.

I am re-sharing it here so that I never forget that time the U.S. flirted with dystopia. John Ioannidis is a professor at Stanford who has made a career out of assessing evidence in medical research. Before the pandemic, he was glowingly covered by The Atlantic and The Economist, his research was written about extensively in The New York Times, and he was an invited speaker for the Talks at Google series. While credentials don’t grant infallibility, the fact that a scientist with his expertise was censored for the statements he made in this video — positions that were/are held by other experts — should be deeply unsettling to us all.

In this blog post, I’m going to evaluate the accuracy of the most controversial points made by Ioannidis in the video. As a brief summary, in the interview (which was originally posted on March 23rd), Ioannidis argues the fatality risk of COVID-19 is most likely much lower than the numbers that were being presented at the time (because these numbers were being calculated from bad data). Based on this observation, he suggests that a more targeted response of protecting the most vulnerable might be a less harmful policy than full lockdowns (a position also held by other health professionals including David Katz at Yale and the Swedish government).

On the COVID-19 Infection Fatality Rate

As some background, I’ll first define the “infection fatality rate” (IFR). The naive way to calculate the IFR is to divide the number of people who died from a disease by the number of people who were infected. However, the number that people are typically actually interested in is a population’s IFR, which is effectively the probability that a random person in the population will die if infected by the disease. To estimate a population’s IFR, researchers sometimes need to account for sub-populations being infected at different rates. This distinction is important because, as I elaborate on later, nursing homes appear to be infected with COVID-19 at higher rates than the general population in many regions.

Jumping in, here’s exactly what Ioannidis said regarding the COVID-19 IFR (the full interview transcript can be found here):

And information from settings where we have more complete information about that denominator suggest that the infection fatality rate is much, much lower than 3.4% it is actually probably much lower compared to the .9% that is the main figure that went into some influential calculations by a wonderful team of researchers at Imperial College which probably overestimated the exact infection fatality risk.

and later (referring to the Diamond Princess cruise ship):

If you try to adjust for the age difference between the passengers and the crew and the general population of a country like the US, the age and gender adjusted infection fatality rate in the US population would be much, much lower than 1%. it may be .1, .2%. We need to have a lot of uncertainty around that estimate because only 7 deaths were recorded, so it’s a very small sample size. And also, these people may not be the same as the general population in terms of what diseases they have, because we know that people who have severe diseases in their background have increased risk of serious outcomes, including mortality. But, allowing for these uncertainties, probably an estimate anywhere between 0.05 up to 1% may be more reasonable to consider, as opposed to, let’s say, 3.4%.

In his essay published on Stat, Ioannidis gave a more precise estimate of 0.3% for the COVID-19 IFR.

So, three months later, what does the research say? A recent serosurvey of COVID-19 antibodies in Geneva, Switzerland conducted by researchers from Johns Hopkins and the University of Geneva suggested an IFR of 0.64%. For those under 50, the researchers calculated an IFR of 0.005%. For children, the virus appears to be less lethal than the flu. However, the authors go on to discuss two reasons why these IFR estimates might actually be too high. The first reason is the fact that nursing home residents represent a large number of the COVID-19 deaths (which is common across countries) but are a very small percent of the total population. If individuals in nursing homes were infected at higher rates than the general population because of the disease’s transmission dynamics and/or poor policies (which appears to be the case), then the estimated population IFR will be too high. When the researchers took nursing home deaths out of their dataset, the IFR dropped by half to 0.32%. The second reason is that:

If mild infections have significantly lower and short-lived antibody responses, our estimates of IFR may be biased upwards.

The study they cited explains why this might be true. Intriguingly, after the cited study was published, researchers in Italy announced wastewater samples collected in the middle of December 2019 contained detectable levels of the COVID-19 virus — a full two months prior to Italy’s first reported case.

Other serosurveys have arrived at similar IFR estimates. For example, a serosurvey of a German community suggested an IFR of 0.36%, but the lead epidemiologist on the study believes the true IFR is even lower, around 0.24% to 0.26%. A nation-wide serosurvey in Spain suggested an IFR of ~1%. However, similar to the other serosurveys, the researchers observed an extremely steep risk gradient with age such that the IFR for those under 50 was only 0.03%. Further, 66% of COVID-19 deaths in Spain were nursing home residents (86.4% of Spain’s COVID-19 deaths were 70 or older, while 62.6% were 80 or older), which suggests their estimate needs to be corrected to get the true population IFR. Removing all the nursing home deaths would likely reduce their calculated IFR to somewhere in the 0.3% to 0.4% range. And, again, this is before performing any correction to account for the fact that serosurveys may be underdetecting the number of people who have actually been infected by COVID-19. Lastly, the CDC estimates that the IFR for COVID-19 is between 0.2% and 0.3%.

So, on the fatality risk of COVID-19, research strongly supports Ioannidis.

How do nursing home outbreaks bias the estimate of a population’s IFR for COVID-19? (skip this section if you don’t care)

Imagine that nursing home residents make up 0.5% of a hypothetical population (similar to the percent in the U.S.). Further, imagine that individuals in nursing homes have a 10% chance of dying from COVID-19 while the rest of the population has a 0.1% chance (i.e., those in nursing homes have a 100x increased risk of dying vs. those who are not). The true IFR for this population is:

0.995 * 0.001 + 0.005 * 0.1 = 0.001495 = 0.1495%

Now, imagine that 100% of the nursing home population becomes infected while only 10% of the rest of the population becomes infected. If we assume a population of one million individuals, that would mean there are 5,000 people living in nursing homes, which would translate to 500 deaths if 100% were infected. In the general population, there would be 0.1 * 995,000 = 99,500 infected, which translates to 99.5 deaths (obviously there can’t be half deaths, but we’ll roll with it). The naive IFR for this group of people would then be:

(500 + 99.5) / (5,000 + 99,500) = 0.0057 = 0.57%

i.e., nearly four times higher than the true population IFR! While a 100% infection rate in nursing homes is obviously not realistic for entire countries, in places like New York City, where nursing homes were required to take COVID-19 patients discharged from the hospital, and Connecticut, where 194 out of 216 nursing homes had at least one COVID-19 case, the exposure rate to COVID-19 for nursing home residents could have been very high.

On Lockdowns vs. Targeted Responses

Given the steep age gradient of COVID-19 risk (which was already apparent from data available at the time of the Ioannidis interview), it makes sense to consider different responses and their costs and benefits. This is what Ioannidis said about the potential costs of lockdowns:

If we shut everyone in their house, it is a solution. If we manage to even isolate everyone, not even being in touch with any other person, in theory, we are containing the spread of the virus. As you realize, this is very difficult to do. It has lots of consequences, and for a society like ours, it means that very soon you will start seeing a major impact on the economy. We already see that. If the economy is ruined, you have unemployment, you have poverty, you have bankruptcies, you have lots of diseases that are associated with this sort of social and economic disruption. We have strong evidence that that can lead to an increase in depression, in anxiety, in suicides, in heart attacks — in common things, in things that cumulatively could have a much higher impact on deaths compared to what SARS-CoV-2 can achieve on its own. So, there are some models that suggest that if you go down that path of basically lockdown, you may need to wait for 18 months.

Within the context of these potential costs (many of which have already come true, e.g., increased unemployment, closing of businesses [especially those owned by black people], and delayed cancer treatment, not to mention other possible consequences, e.g., poor children being educationally left behind, increased tuberculosis burden, and increased global hunger; for more, see my thread here), Ioannidis suggested an alternative strategy might lead to the least amount of overall harm:

Depending on what is the stage of the epidemic and how many people are infected, it may be that a solution of, indeed, isolating those who are at high risk and making sure that we protect them as fiercely as we can while letting the rest of the society continue their work and try to maintain the economy and also support these much fewer numbers of individuals who are at risk — it may make a lot of sense. And I think we need data to decide if this is the best course, versus a blind lockdown that would last forever with very uncertain outcomes.

Clearly, those in nursing homes are among the most vulnerable, yet multiple locked down states and countries seem to have actively endangered these individuals (e.g., New York, Italy, and Britain). Further, the outbreaks have brought to light some of the truly horrific conditions and unhygienic practices found in many long-term care facilities (like what the Canadian army discovered in Ontario). The utter lack of media attention given to the COVID-19 tragedy in nursing homes is a testament to how little concern we, as a society, actually have for our elderly (for a list of the limited articles that exist on COVID-19 and nursing homes, see my thread here).

Additionally, COVID-19 appears to have spread extensively within hospitals, which obviously contain many vulnerable people (see, e.g., Britain and Italy; there are also numerous studies at this point documenting widespread infection of healthcare workers in different locations).

So, regarding the costs of the lockdowns and the importance of protecting the most vulnerable, Ioannidis was correct.

On the Total Death Toll of COVID-19

Lastly, on the total number of COVID-19 deaths, Ioannidis had this to say:

As I said, we still have a lot of uncertainty on what would be the exact evolution of this epidemic. We still see that there’s growth in the number of cases and in the number of deaths, but let’s say that — and this is an entirely hypothetical scenario — that that new coronavirus was not detected, no one had noticed it, and no one had found that this is a new entity, and eventually it killed 10,000 people in the US based on this presentation that you have respiratory distress syndrome, acute respiratory distress syndrome.

This “prediction” of 10,000 deaths has been constantly misrepresented and taken out of context. Literally in that same response, Ioannidis goes on to state:

Now, this doesn’t mean that that’s going to be the final outcome of this pandemic. There’s a chance that we may end up with far worse outcomes or there is even a chance that we may end up with even less compared to 10,000 deaths in this country. This is why we need data. We need, urgently, data to be able to get some sense of where we are and where we are heading.

Clearly, Ioannidis acknowledged that more deaths were possible. In his original “prediction” (which it is not), Ioannidis is describing a hypothetical scenario in which COVID-19 killed a large number of people and the vast majority of the public did not notice (in contrast to inducing a panic). In his Stat essay, Ioannidis describes how he arrived at the 10,000 total deaths number:

If we assume that case fatality rate [note: Ioannidis means IFR here. CFR and IFR were being used interchangeably at the start of the pandemic.] among individuals infected by SARS-CoV-2 is 0.3% in the general population — a mid-range guess from my Diamond Princess analysis — and that 1% of the U.S. population gets infected (about 3.3 million people), this would translate to about 10,000 deaths.

It is now obvious that a (probably much) higher percent of the population has been infected, and, as previously discussed, nursing home residents appear to have been infected at an even higher rate in many regions. In Connecticut, COVID-19 death statistics suggest 12.2% of the state’s entire nursing home population has died of COVID-19, while, in Massachusetts, that number is 9.1% (notably, New York appears to be hiding its COVID-19 nursing home statistics).

His broader point about the number of flu deaths society has been willing to tolerate in the past is worth exploring further. To provide some context on the magnitude of the COVID-19 death toll, in the 2017–2018 flu season, the CDC estimates there were 61,000 flu deaths with a 95% confidence interval spanning from 46,404 to 94,987 deaths. For the flu pandemic of 1968/1969, the CDC states:

The estimated number of deaths was 1 million worldwide and about 100,000 in the United States. Most excess deaths were in people 65 years and older.

The U.S. population in 1968 was 201 million, so the equivalent number of deaths for the 2020 U.S. population would be 164,000, and that’s without accounting for the fact that the U.S. population skews older now (meaning the total number of deaths would be even higher when using the age-adjusted fatality rates). Deaths for the 1968 flu peaked in the winter of 1968/1969 and the U.S. responded by holding Woodstock that summer. Prior to COVID-19, I, like many other Americans, had never even heard of any of the other large flu epidemics that hit the U.S. outside of the 1918 flu pandemic.

Again, no one is suggesting these deaths are meaningless and that we should not take steps to minimize them, but the fact that we do not lock down society every flu season suggests we have collectively decided to accept a certain amount of risk in our daily lives in exchange for the benefits of a modern lifestyle. David Spiegelhalter, the Chair of the Winton Centre for Risk and Evidence Communication at the University of Cambridge, calculated the risk of becoming infected and dying of COVID-19 in the U.K. and found it to be equivalent to the normal risk of dying over a two week span for the non-elderly and the risk of dying over a month for the elderly.

The common rebuttal to the above is that the reason why COVID-19 deaths aren’t even higher is because of the lockdowns. Yet Sweden, which infamously did not lock down, has not fared worse than many other countries that did (by the way, Anders Tegnell did not say he regretted Sweden’s strategy). Many critics are quick to point out that Sweden has experienced more COVID-19 deaths per capita than their Nordic neighbors Norway (whose prime minister has expressed regret over using such strict lockdowns) and Finland, but the death toll of COVID-19 has been extremely heterogeneous even among neighboring regions with similar government responses. For example, Massachusetts currently has 12 times as many COVID-19 deaths per capita as neighboring Vermont, which suggests things like population density, tourism traffic, number of nursing home residents, and nursing home policies are all much more important factors in determining a region’s COVID-19 death toll than any particular government response (at least at the point in the pandemic when many governments started reacting). Further, like many other countries and U.S. states, the vast majority (70%) of COVID-19 deaths in Sweden were among those receiving elderly care and more than half were in care homes. Lastly, Sweden’s total mortality for this year does not appear excessive when compared to the total mortality they experienced during flu seasons in the past 30 years.

As another example, Florida, with its spring breakers, theme parks, “early” re-opening, and large elderly population, has 10 times fewer COVID-19 deaths per capita than similarly-sized and extremely locked down New York (and, no, Florida is not obfuscating their death numbers).

If, as Ioannidis advocated for, nursing homes had been adequately protected (instead of seemingly doing the opposite), the COVID-19 death toll could have been much lower. But even still, Ioannidis never suggested a larger death toll was out of the realm of possibility, and he explicitly made a plea for better data collection to enable more accurate projections and better informed responses.

So, on the possible death toll of COVID-19, Ioannidis was never misleading.

Conclusion

On close inspection, it appears Ioannidis was simply using his expertise to make reasonable and important arguments about the threat of COVID-19 and the costs and benefits of different responses. The video should never have been removed from YouTube, and we should be ashamed of ourselves for accepting such censorship. I hope we can all learn from this episode in history and that, the next time we face an unknown threat, we do not allow fear and panic to cloud our judgement, nor accept the silencing of alternative points of view.