As the COVID-19 pandemic continues to evolve, advocates and policymakers are already developing policy and legislative proposals to invest further in global and national pandemic preparedness programs. In order to guide these critical investments intended to protect people against future epidemic threats, it is important to have reliable measures to track progress, identify countries lagging behind, and benchmark against high achievers. However, the COVID-19 pandemic has revealed that it remains unclear how best to measure countries’ capacity to respond effectively to severe pandemic threats.

How best to measure countries’ capacity to respond effectively to severe pandemic threats remains unclear

This measurement gap—the inability to accurately assess countries’ pandemic preparedness—is not for lack of trying. Following the 2014–2016 Ebola outbreak in West Africa, the World Health Organization (WHO) and regional partners developed the Joint External Evaluation (JEE) process to monitor countries’ adoption and implementation of the core capacities under the International Health Regulations. These evaluations are designed to help countries identify “the most critical gaps within their human and animal health systems in order to prioritize opportunities for enhanced preparedness and response,” according to the WHO. More than one hundred nations have undertaken voluntary Joint External Evaluations. Another metric, the 2019 Global Health Security (GHS) Index, includes important and relevant measures for the current pandemic that go beyond Joint External Evaluations. This index is an assessment and benchmarking tool that measures a country’s capacity to rapidly respond to and mitigate the spread of an epidemic, taking into account its health system’s ability to care for the sick and protect health workers.

The Joint External Evaluation and the Global Health Security Index have been essential advances in global health security, introducing measurement, furthering accountability, and drawing international attention to efforts aimed at building the capacity of nations to prevent, detect, and respond to epidemic threats.

In many ways, these metrics have been a catalyst for increased pandemic preparedness. Critically, however, six months into the current pandemic, these existing measures have not correlated with lower COVID-19 death rates. On the contrary, countries with higher Joint External Evaluation and Global Health Security Index scores seem to experience higher death rates, even when accounting for differences in countries’ population age structures. Death rates in the following figures show the cumulative, reported, age-standardized to COVID-19 deaths per hundred thousand people in the fifty days following the date of the first death in that country.

(Authors’ Note: In order to age-standardize these death rates, we used the same methods as outlined in Section 2 of Appendix B from IHME’s COVID-19 paper, using the average age-pattern of mortality based on data from countries where such data has been available.)

The disconnect that exists between Global Health Security Index overall scores and COVID-19 death rates also holds true when the analysis is limited to more specific indicators from the index, such as the “Rapid Response to and Mitigation of the Spread of an Epidemic.” In the interest of space, however, we only include the results from the comparison with Global Health Security Index overall scores.

Universal health coverage (UHC), a measure of access to quality health services, is another metric for which many have advocated as a potential means of improving global health security. However, it has so far fared no better as a predictor of nations’ success in coping with the COVID-19 pandemic.

Several countries that have acted quickly to successfully contain COVID-19, such as South Korea and Taiwan, have high levels of universal health coverage. However, the same is true for many of the countries that have struggled in this pandemic, including Italy and the United Kingdom.

Joint External Evaluations, the Global Health Security Index, and measures of universal health coverage are not meant to predict health outcomes but are intended to be tools for identifying gaps in capacity and mobilizing financial and political support to fill those gaps. Francis Fukuyama in Foreign Affairs has argued that a competent state institution is one of three essential ingredients to nations’ success in the current pandemic—effective leadership and a government that citizens trust and listen to must also be in the mix. This insight rings true with many people’s lived experience of COVID-19.

This pandemic has demonstrated, however, that we may not yet know how to judge the competence of government institutions. Ranking high on the Joint External Evaluations, Global Health Security Index, and universal health coverage has not only been insufficient for predicting success in this pandemic—it also appears to be unnecessary. Vietnam is ranked 50th in Global Health Security Index overall scores, 59th in Joint External Evaluations scores, and 104th in universal health coverage but so far has been one of the more successful countries in containing the coronavirus.

Other reports have previously observed the lack of correlation between current measures of pandemic preparedness and success in containing the spread of the coronavirus. This post seeks to better visualize that disconnect with regard to COVID-19 deaths, extends the analysis to universal health coverage, and accounts for national differences in population age-structure and the timing of the epidemic.

The results illustrate that we may not yet understand how to measure and monitor the degree to which countries are prepared to weather severe pandemics. As policymakers scale up investments in programs and funds designed to help countries prepare for future epidemics, we should ensure that we are able to accurately measure countries’ pandemic preparedness and response capabilities in order to establish goals and track nations’ progress over time.

EDITOR’S NOTE: Foreign Affairs magazine, mentioned in the eighth paragraph, is published by the Council on Foreign Relations, which also publishes Think Global Health. Sawyer Cosby and Joseph Dieleman are employed by the University of Washington’s Institute for Health Metrics and Evaluation, (IHME), a partner on Think Global Health. All statements and views expressed in this article are solely those of the individual authors and are not necessarily shared by their institution.