[Paper Review] Ensemble Machine Learning Methods for Modeling COVID19 Deaths

Rahil Bathwal, Pavan Chitta+2 more|arXiv (Cornell University)|Oct 4, 2020

COVID-19 epidemiological studies12 references4 citations

TL;DR

This paper proposes an ensemble machine learning model that predicts quantile forecasts of daily COVID-19 deaths at the U.S. county level using a hybrid of epidemiological (SEIR-QD) and machine learning models (LSTM, gradient boosted trees, random forests, neural networks). The model minimizes pinball loss across 9 quantiles (10th–90th), achieving state-of-the-art performance with a 14-day forecast pinball loss of 1.4543 and RMSE of 0.0896, outperforming 50+ teams in a Caltech modeling competition.

ABSTRACT

Using a hybrid of machine learning and epidemiological approaches, we propose a novel data-driven approach in predicting US COVID-19 deaths at a county level. The model gives a more complete description of the daily death distribution, outputting quantile-estimates instead of mean deaths, where the model's objective is to minimize the pinball loss on deaths reported by the New York Times coronavirus county dataset. The resulting quantile estimates accurately forecast deaths at an individual-county level for a variable-length forecast period, and the approach generalizes well across different forecast period lengths. We won the Caltech-run modeling competition out of 50+ teams, and our aggregate is competitive with the best COVID-19 modeling systems (on root mean squared error).

Motivation & Objective

• To address the lack of high-resolution, localized pandemic forecasting tools for U.S. counties, where state-level models fail to capture regional disparities in disease spread.
• To improve upon mean-only predictions by providing full quantile forecasts (10th–90th percentiles) that convey uncertainty and severity distribution.
• To integrate epidemiological priors (SEIR-QD) with data-driven machine learning models (LSTM, gradient boosting, random forests) for robust, generalizable county-level forecasting.
• To optimize model performance using the pinball loss metric, which weights under- and over-prediction errors differently across quantiles, enabling better uncertainty quantification.
• To deliver actionable, localized forecasts for policymakers to guide resource allocation and policy decisions during the pandemic.

Proposed method

• The model combines dynamic features (lagged cases, deaths, mobility) and static features (population density, county demographics) from New York Times, Berkeley, and Descartes Labs datasets.
• Lagged features (e.g., 7-day and 14-day lags) are used to capture temporal dependencies, with mobility data imputed via forward-fill to handle missing values.
• A novel time-series embedding method transforms daily death sequences into 4x4 pooled feature maps (16-dimensional vectors), which are clustered using K-means (k=6) to generate county-specific cluster features.
• The final ensemble model combines predictions from multiple architectures—LSTM, gradient boosted trees, random forests, neural networks, and SEIR-QD—using weighted averaging to improve robustness.
• Model evaluation uses the 9-quantile pinball loss, defined as the average of quantile-specific losses that penalize under- and over-prediction asymmetrically based on the target quantile.
• The model is trained and evaluated on a variable-length forecast window, with performance measured on U.S. county-level data from May 2020.

Experimental results

Research questions

• RQ1 Can a hybrid machine learning and epidemiological model improve county-level COVID-19 death forecasting compared to mean-only or single-model approaches?
• RQ2 Does using quantile forecasts (10th–90th percentiles) with pinball loss optimization yield more reliable and informative predictions than mean forecasts?
• RQ3 How do different machine learning architectures (LSTM, gradient boosting, random forests) and epidemiological models (SEIR-QD) compare in performance when combined in an ensemble?
• RQ4 To what extent do lagged dynamic features (cases, deaths, mobility) and temporal features (day of week, date offset) contribute to predictive accuracy?
• RQ5 Can the model generalize across variable forecast lengths and maintain low error across diverse U.S. counties?

Key findings

• The ensemble model achieved a pinball loss of 1.4543 and RMSE of 0.0896 on a 14-day forecast period (May 25–June 4, 2020), outperforming 50+ teams in the Caltech modeling competition.
• LSTM and the ensemble model were the top-performing architectures, with the ensemble model achieving the lowest pinball loss (1.4543) and RMSE (0.0896) across both forecast periods.
• Lagged cases, deaths, and mobility features were the most predictive inputs, with lagged cases being the most important predictor across models.
• Temporal features such as day of the week and date offsets improved model performance, likely by capturing reporting cycles and pandemic phase transitions.
• Static features like population density had limited but non-zero predictive power, while dynamic features dominated model performance.
• The model generalized well across different forecast lengths, maintaining low error and consistent performance on variable-length prediction windows.