All Bets Are Off for Measuring Pandemic Preparedness

All Bets Are Off for Measuring Pandemic Preparedness

Three standard measures don't predict how countries fare in COVID-19—so how do we better prepare for the next pandemic?

Photo shows a pair of hands holding up the corners of two cards to reveal a pair of aces. A large stack of chips is in the background.
A player checks his cards during the finals of the Russian Masters Poker Cup at the Azov-City, Russia, gambling zone, south of Russia's southern city of Rostov-on-Don, on September 23, 2010. REUTERS/Vladimir Konstantinov

As the COVID-19 pandemic continues to evolve, advocates and policymakers are already developing policy and legislative proposals to invest further in global and national pandemic preparedness programs. In order to guide these critical investments intended to protect people against future epidemic threats, it is important to have reliable measures to track progress, identify countries lagging behind, and benchmark against high achievers. However, the COVID-19 pandemic has revealed that it remains unclear how best to measure countries’ capacity to respond effectively to severe pandemic threats.

How best to measure countries’ capacity to respond effectively to severe pandemic threats remains unclear

This measurement gap—the inability to accurately assess countries’ pandemic preparedness—is not for lack of trying. Following the 2014–2016 Ebola outbreak in West Africa, the World Health Organization (WHO) and regional partners developed the Joint External Evaluation (JEE) process to monitor countries’ adoption and implementation of the core capacities under the International Health Regulations. These evaluations are designed to help countries identify “the most critical gaps within their human and animal health systems in order to prioritize opportunities for enhanced preparedness and response,” according to the WHO. More than one hundred nations have undertaken voluntary Joint External Evaluations. Another metric, the 2019 Global Health Security (GHS) Index, includes important and relevant measures for the current pandemic that go beyond Joint External Evaluations. This index is an assessment and benchmarking tool that measures a country’s capacity to rapidly respond to and mitigate the spread of an epidemic, taking into account its health system’s ability to care for the sick and protect health workers.

The photo shows the teacher at the front of a room showing a placard to the children.
The Joint External Evaluation process emerged after the 2014–16 Ebola outbreak in West Africa. Here a teacher conducts an Ebola awareness drive in Gueupleu, Ivory Coast, on November 3, 2014. REUTERS/Thierry Gouegnon

The Joint External Evaluation and the Global Health Security Index have been essential advances in global health security, introducing measurement, furthering accountability, and drawing international attention to efforts aimed at building the capacity of nations to prevent, detect, and respond to epidemic threats.

In many ways, these metrics have been a catalyst for increased pandemic preparedness. Critically, however, six months into the current pandemic, these existing measures have not correlated with lower COVID-19 death rates. On the contrary, countries with higher Joint External Evaluation and Global Health Security Index scores seem to experience higher death rates, even when accounting for differences in countries’ population age structures. Death rates in the following figures show the cumulative, reported, age-standardized to COVID-19 deaths per hundred thousand people in the fifty days following the date of the first death in that country.

Joint External Evaluation Scores vs. COVID-19 Death Rates

(Authors' Note: In order to age-standardize these death rates, we used the same methods as outlined in Section 2 of Appendix B from IHME’s COVID-19 paper, using the average age-pattern of mortality based on data from countries where such data has been available.)

The disconnect that exists between Global Health Security Index overall scores and COVID-19 death rates also holds true when the analysis is limited to more specific indicators from the index, such as the “Rapid Response to and Mitigation of the Spread of an Epidemic.” In the interest of space, however, we only include the results from the comparison with Global Health Security Index overall scores.

Global Health Security Index Scores vs. COVID-19 Death Rates

Universal health coverage (UHC), a measure of access to quality health services, is another metric for which many have advocated as a potential means of improving global health security. However, it has so far fared no better as a predictor of nations’ success in coping with the COVID-19 pandemic.

Several countries that have acted quickly to successfully contain COVID-19, such as South Korea and Taiwan, have high levels of universal health coverage. However, the same is true for many of the countries that have struggled in this pandemic, including Italy and the United Kingdom.

Universal Health Coverage vs. COVID-19 Death Rates

Joint External Evaluations, the Global Health Security Index, and measures of universal health coverage are not meant to predict health outcomes but are intended to be tools for identifying gaps in capacity and mobilizing financial and political support to fill those gaps. Francis Fukuyama in Foreign Affairs has argued that a competent state institution is one of three essential ingredients to nations’ success in the current pandemic—effective leadership and a government that citizens trust and listen to must also be in the mix. This insight rings true with many people’s lived experience of COVID-19.


One of the more successful countries in containing the coronavirus

This pandemic has demonstrated, however, that we may not yet know how to judge the competence of government institutions. Ranking high on the Joint External Evaluations, Global Health Security Index, and universal health coverage has not only been insufficient for predicting success in this pandemic—it also appears to be unnecessary. Vietnam is ranked 50th in Global Health Security Index overall scores, 59th in Joint External Evaluations scores, and 104th in universal health coverage but so far has been one of the more successful countries in containing the coronavirus.

The photo shows a number of people on motorcycles wearing masks.
Laborers wearing masks wait for a ferry on the way home after work during the coronavirus pandemic in Hai Duong province, Vietnam, on April 7, 2020. REUTERS/ Nguyen Huy Kham

Other reports have previously observed the lack of correlation between current measures of pandemic preparedness and success in containing the spread of the coronavirus. This post seeks to better visualize that disconnect with regard to COVID-19 deaths, extends the analysis to universal health coverage, and accounts for national differences in population age-structure and the timing of the epidemic.

The results illustrate that we may not yet understand how to measure and monitor the degree to which countries are prepared to weather severe pandemics. As policymakers scale up investments in programs and funds designed to help countries prepare for future epidemics, we should ensure that we are able to accurately measure countries’ pandemic preparedness and response capabilities in order to establish goals and track nations’ progress over time.

Picture shows the girl standing next to her masterpiece.
A small snapshot of the pandemic: Nguyen Doi Chung Anh, a ten-year-old Vietnamese school girl, stands next to her painting about the coronavirus disease at home in Hanoi, Vietnam, on June 10, 2020. REUTERS/Nguyen Huy Kham

EDITOR'S NOTE: Foreign Affairs magazine, mentioned in the eighth paragraph, is published by the Council on Foreign Relations, which also publishes Think Global Health. Sawyer Cosby and Joseph Dieleman are employed by the University of Washington's Institute for Health Metrics and Evaluation, (IHME), a partner on Think Global Health. All statements and views expressed in this article are solely those of the individual authors and are not necessarily shared by their institution.

Sawyer Crosby is a data analyst on the Resource Tracking team at the University of Washington's Institute for Heath Metrics and Evaluations (IHME).

Joseph L. Dieleman is an associate professor at the University of Washington’s Institute for Health Metrics and Evaluation.

Samantha Kiernan is a Schwarzman Scholar at Tsinghua University in Beijing, China. 

Thomas J. Bollyky is director of the global health program and senior fellow for global health, economics, and development at the Council on Foreign Relations.

Most Popular