Flu Forecasting Models Consistently More Accurate Than Historical Baseline Models

In a multi-institution collaboration assessing 22 distinct influenza forecasting models across 7 flu seasons, investigators have found that the majority of models consistently showed higher accuracy than historical baseline models.

A team of investigators from the University of Massachusetts, Amherst Carnegie Mellon University, the University of Texas at Austin, the US Centers for Disease Control and Prevention (CDC), Columbia University, Los Alamos National Laboratory, and Mount Holyoke College collaborated on the project, which compared the accuracy of weekly real-time forecasts assembled between 2010 and 2017 to a historical baseline seasonal average.

"The field of infectious disease forecasting is in its infancy and we expect that innovation will spur improvements in forecasting in the coming years," the authors write in their report published in Proceedings of the National Academy of Sciences.

The experts formed a consortium called the FluSight Network and made fine-grained and standardized comparisons of various approaches to forecasting that employed different data sources and modeling frameworks.

"Taken together, our models are used to build an ensemble forecast, a weighted average of all approaches, based on how each of the models have performed in the past," Nicholas G. Reich, PhD, associate professor in the Department of Biostatistics and Epidemiology at the University of Massachusetts, Amherst, and an author on the paper told Contagion®. "This ensemble approach is able to capitalize on the strengths of different models and has shown overall better accuracy than any of the individual models."

In 2013, the CDC launched the “Forecast the Influenza Season Collaborative Challenge,” which has run each year since. The challenge includes prospective, real-time, weekly forecasts from teams across the world, and focuses around forecasts of the weighted percentage of doctor visits for patients exhibiting influenza-like illness symptoms in a particular US Health and Human Services region.

The investigative team observed that the majority of models were more accurate than baseline forecasts in regions with and without predictable seasonal trends.

Across all regions of the United States, over half of the models consistently outperformed the historical baseline ones when forecasting incidence of influenza-like illness 1 week, 2 weeks, and 3 weeks ahead of available data and when forecasting the timing and magnitude of season peaks.

Additionally, a majority of the models produced showed consistent improvement over the historical baseline for 1- and 2-week ahead forecasts, although fewer models consistently outperformed the baseline model for 3- and 4-week ahead forecasts.

The investigators note that they did not identify substantial differences between models relying on “an underlying mechanistic model of disease transmission” and models that were statistical in nature.

In some particular regions of the United States, delays in data reporting were strongly and negatively associated with forecast accuracy. And the authors indicate that a major impediment to predictive ability was the real-time accuracy of available data.

Forecasting for influenza can have a critical impact on public health decisions. Estimates of influenza-like illness symptoms can influence decisions about hospital staffing, allocation of vaccines and therapeutics, the timing of public health communication about the flu, and the implementations of interventions such as vaccination clinics.

The authors indicate that multi-season forecasting comparisons are beneficial because they can improve understanding of how models perform long-term and which models may be reliable in the future.

Through a standardized comparison, it can be determined which forecasting models perform best in certain settings, how results can best be disseminated and used by public health officials, as well as potential issues related to models.

“Overall, this work shows strong evidence that carefully crafted forecasting models consistently out-performed a historical baseline model for targets of a particular public health interest,” the investigators conclude.

In order to improve forecasting models, access to more timely reporting and data sources are needed.

"In my opinion, the single biggest factor that could help us improve infectious disease forecast accuracy is better real-time data of all kinds. I'm not just talking about the timeliness of public surveillance data, but creating systems that aggregate health data from sources in the private sector that could be used in infectious disease forecasting could be a game-changing innovation," Dr. Reich explained. "For example, systematically aggregated data from electronic health records or laboratory test results, delivered in real-time, to forecasters could make a huge impact."

This season, the current flu forecast models predict as of January 15, that flu activity is likely to increase in the next 2 to 4 weeks with highest activity occurring in the next 2 months. Additionally, there is a 65% chance that the highest activity will occur by the conclusion of January and a >95% chance that the highest activity will occur by the end of February.