Why India’s Response to WHO on Excess COVID-19 Deaths Doesn’t Hold Water

This article was originally published on The Wire Science, our website dedicated to science, health and environment reportage and analysis. Follow, read and share.

The WHO has reportedly estimated that India could have had four million excess deaths during the COVID-19 pandemic. This would make India the country with the highest number of pandemic deaths in the world by far – with around a quarter of the global total.

The Indian government has objected to the WHO’s estimates. According to the New York Times, the Indian government’s objections have stalled the publication of  the estimates, potentially delaying an important effort to understand the worldwide impact of the pandemic.

Let’s be clear that this is not a new controversy. The WHO’s estimate is in line with previous work, and the Indian government’s objections also follow a familiar pattern. Several preprint as well as peer-reviewed papers on pandemic mortality in India have elicited rebuttals from the health ministry in the form of statements entitled “COVID-19: Myth v. Facts”. These rebuttals have labelled the academic work variously as “ill informed”, “speculative”, “fallacious” and “completely inaccurate”. The work has also been accused of “creating panic in the community” and “misguiding people”.

The numbers in context

Until the WHO report is available, it is hard to say much about the details of their approach to estimating excess mortality – but there are some things we can say with confidence.

First, the WHO estimate is consistent with estimates in a number of published studies, technical papers and preprint papers. These come from several independent research groups, and use data from the civil registration system (CRS), a few different surveys and other government systems.

Most of this work has estimated excess deaths due to the pandemic to be between 3 and 5 million, with the lowest central estimate being 2.7 million and the highest being 6.3 million. Nowhere do we see estimates close to the official figure of around 0.5 million COVID-19 deaths.

Second, the WHO’s estimate is in keeping with the epidemiology of COVID-19. The numbers are consistent with India’s large but relatively young population, and the wide spread of disease, largely occurring before vaccination could have a major impact. Four million pandemic deaths is tragic but very plausible – unless we believe that Indians are somehow less vulnerable to the severe effects of COVID-19 than others.

Third, the WHO’s estimate implies that India has spectacularly failed to count its pandemic deaths, with around eight-times as many excess deaths as official deaths from the disease. This ratio is high, but believable if we read ground-level reports from journalists, which have shown how easy it is for official figures to miss the majority of deaths when disease sweeps  through marginalised populations.

For context, India is not alone in failing to track pandemic deaths: the ratio of excess deaths to official COVID-19 deaths is similar or higher in several countries, mostly in Asia and Africa. But it seems important to understand why some countries, such as Iran and South Africa, have done better, with much lower ratios of excess deaths to recorded deaths.

Estimating excess mortality

The WHO estimate is neither surprising nor implausible – but this does not place it above criticism. Before we can assess the Indian government’s objections, we need to understand what the literature actually attempts to estimate.

The number of pandemic excess deaths is the difference between two quantities: expected deaths during the pandemic and actual deaths in this period.

Estimating expected deaths means asking, “If the pandemic had not occurred, how many people would have died during the pandemic?” This is an example of a counterfactual – a question about a scenario that didn’t actually occur. To answer it, you necessarily need some modelling, even if there is good historical data.

Estimating actual deaths should be more straightforward. In countries with good vital registration, we simply use death registration data. In India, however, there is missing data, plus uncertainties around levels of registration. Thus, we need some modelling and/or extrapolation to fill the gaps.

All of this means putting a number on pandemic excess deaths in India is not straightforward, and that there is space for valid disagreements. So let’s examine the Indian government’s objections to see if they can be classified as “valid scientific debate”.

Government objections

One objection is that many of the studies use “unofficial” CRS data – obtained, for example, by journalists through Right to Information requests. The Union government could, of course, accelerate the release of “official” data and bypass this problem. But in any case, it is unclear what data is in fact unofficial and why this should affect the results.

Andhra Pradesh, for example, has freely available CRS data on an online portal, and this shows a huge surge of 2.5-3 lakh excess death registrations during the pandemic – compared to just 15,000 official COVID-19 deaths in the state. Is data on this portal “official” or “unofficial”? Are we to believe that most of these excess registrations are a consequence of errors?

A related objection is that the size and diversity of India makes filling the data gaps risky. There is some truth in this – but the objection is overstated. CRS data of reasonable quality is available for more than half of the national population, with more limited data covering much of the remainder. Moreover, there is no evidence that extrapolation leads to national estimates which are too high. In fact, some states such as Uttar Pradesh and Gujarat which are often left out of analyses for lack of good data, have quite possibly seen higher mortality than the national average.

Ultimately, data gaps are a problem, but:

• We can quantify and mitigate this problem in various well-known ways. For example, the authors of the WHO analysis stress that they carried out “extensive sensitivity… and cross-validation analyses” to address the uncertainties.

• Issues of data availability and quality can push estimates in either direction. The government has never demonstrated that they systematically push excess mortality estimates upwards.

In response to government criticisms about data and methodology we have to ask two questions: has the government demonstrated that the scale of the uncertainty is enough to make a critical difference to the results? And have they demonstrated evidence of systematic bias?

To both questions, the answer is a clear ‘no’.

Incoherent objections

While the objections about data quality and availability begin with a grain of truth, some others are contradictory or incoherent. For example, one statement dated July 27, 2021, claims that “all the births and deaths in the country get registered”. This is, of course, false, as government data plainly shows. But it also begs the question: if CRS data is so reliable, why does the government insist that the big surges in deaths in CRS data are a mirage?

Then there are logical absurdities, such as this from March 11, 2022 (emphasis in the original):

“For states where Civil Registration System was available, reported deaths during the pandemic has been compared with average reported deaths for the same period in the year 2018 and 2019 which doesn’t take into account multiple pandemic management efforts including lockdown, containment zones, testing and contact tracing, wider dissemination and implementation of clinical management protocols and world’s largest vaccination drive, which form the foundation of pandemic management in the country.“

The health ministry appears to be arguing, “We tried to manage the epidemic; so it is unfair to find high mortality in CRS data. You have not given us marks for effort.”

This appears to be a category error. We expect mitigation and vaccination to reduce mortality; but they do not change how we measure it!

The list of errors and flawed arguments goes on. Here are a few more examples.

The health ministry repeatedly reminds us of Indian Council of Medical Research (ICMR) guidance on death recording, in order to claim that “the likelihood of underreporting is less” – but there is no discussion of whether ICMR guidance was implemented. (It often was not.)

Monetary compensation for families of victims is mentioned, again to claim that underreporting is unlikely. This is problematic because many, many victims died without being tested. And in any case, compensation was only approved in October 2021, after most of the deaths had occurred.

Test positivity is brought in: “But, this variation in COVID-19 positivity rate within India was not considered for modeling purposes.” But test positivity has no clear relationship to mortality.

Conclusions

When all is said and done, it appears that the objections raised by the Indian government to estimates of COVID-19 mortality do not hold water. The objections appear to fall into two categories:

• Arguments which begin with a valid but general observation – gaps in the data, risks in extrapolation, etc. – but fail to demonstrate how these invalidate the conclusions or lead to systematic bias.
• Distractions, red herrings and logical errors.

Each published estimate has uncertainties and there are valid criticisms to be made of all the studies. But the bottom line is this: mortality data is now available from many different sources, and it has been analysed by many different groups. And even the most optimistic readings of this data point to India’s pandemic toll being many times the official COVID-19 toll.

It is almost a year since convincing data started to emerge showing that the great majority of pandemic deaths were going uncounted in India. Since then, there has been plenty of time for the Indian government to strengthen mortality surveillance. Even now, many uncertainties could be resolved by (i) accelerating the release of official CRS data for the pandemic period, and (ii) conducting a large-scale mortality survey to fill in gaps left by incomplete death registration.

Instead, government efforts seem focussed on rebutting scientific work. These rebuttals are weakly argued and show little awareness of the data or methodologies. They also seem to be in bad faith, always implying that published estimates of mortality are too high, but never trying to quantify the uncertainties or acknowledging that the estimates could be too low.

It is a sign of the times that a fair amount of coverage of the most recent dispute has framed the story as “India” standing up to a global body in an act of defiance. “No, we did not die in large numbers”, the government and supportive media appear to proclaim on behalf of Indians. COVID-19 brought tragedy into the lives of many millions of families in India, and it is an absurd version of patriotism that denies this.

Murad Banaji is a mathematician with an interest in disease modelling.