[Paper Review] Dual Interpretation of Machine Learning Forecasts

Philippe Goulet Coulombe, Maximilian Goebel+1 more|arXiv (Cornell University)|Dec 17, 2024

Energy Load and Power Forecasting4 citations

TL;DR

This paper introduces a dual interpretation framework for machine learning forecasts in macroeconomics, decomposing out-of-sample predictions into contributions from individual historical training observations using proximity-based weights. By treating these weights as data portfolio holdings, the method enables intuitive visualization of forecast drivers as historical analogies and introduces novel diagnostic metrics like forecast concentration and turnover, demonstrating strong interpretability gains in low-sample, high-dimensional settings such as post-pandemic inflation and recession forecasting.

ABSTRACT

Machine learning predictions are typically interpreted as the sum of contributions of predictors. Yet, each out-of-sample prediction can also be expressed as a linear combination of in-sample values of the predicted variable, with weights corresponding to pairwise proximity scores between current and past economic events. While this dual route leads nowhere in some contexts (e.g., large cross-sectional datasets), it provides sparser interpretations in settings with many regressors and little training data-like macroeconomic forecasting. In this case, the sequence of contributions can be visualized as a time series, allowing analysts to explain predictions as quantifiable combinations of historical analogies. Moreover, the weights can be viewed as those of a data portfolio, inspiring new diagnostic measures such as forecast concentration, short position, and turnover. We show how weights can be retrieved seamlessly for (kernel) ridge regression, random forest, boosted trees, and neural networks. Then, we apply these tools to analyze post-pandemic forecasts of inflation, GDP growth, and recession probabilities. In all cases, the approach opens the black box from a new angle and demonstrates how machine learning models leverage history partly repeating itself.

Motivation & Objective

• To address the interpretability challenge in high-dimensional macroeconomic forecasting where traditional feature-based interpretation breaks down due to collinearity and sparsity.
• To propose a dual route to interpretation—beyond feature contributions—by expressing forecasts as linear combinations of in-sample target values weighted by model-perceived proximity to training observations.
• To enable visualization of forecast drivers as time series of historical contributions, enhancing narrative and policy-relevant interpretation.
• To introduce financial portfolio-inspired metrics (e.g., concentration, turnover, short position) for diagnosing forecast stability and risk.
• To demonstrate the method’s applicability across diverse ML models, including ridge regression, random forests, boosted trees, and neural networks with linear output layers.

Proposed method

• The method leverages the dual solution to regularized least squares problems, where out-of-sample predictions are expressed as weighted sums of in-sample target values, with weights derived from pairwise proximity scores between current and past data points.
• For (kernel) ridge regression, proximity weights are directly recoverable from the dual solution via the matrix inversion lemma, enabling exact decomposition without retraining.
• For neural networks with linear output layers, the final layer optimization is reformulated as a regularized linear regression over hidden layer activations, allowing proximity weights to be computed via the same ridge regression machinery.
• For tree-based models (e.g., random forest, XGBoost), proximity weights are derived through ensemble averaging and accounting operations on individual tree contributions, exploiting their implicit partitioning of the feature space.
• The representer theorem underpins the approach by ensuring that any solution in a reproducing kernel Hilbert space can be expressed as a finite linear combination of kernel functions evaluated at training points, enabling the dual representation.
• The method is implemented using R code available on GitHub, and applied to forecast inflation, GDP growth, and recession probabilities using post-pandemic data from 1961 to 2019.

Experimental results

Research questions

• RQ1 How can machine learning forecasts be interpreted not only through feature contributions but also through historical analogies based on proximity to past data points?
• RQ2 In settings with many predictors and few training observations, does expressing forecasts as weighted combinations of historical outcomes improve interpretability and diagnostic insight?
• RQ3 Can proximity-based weights be systematically retrieved for diverse ML models, including neural networks and tree ensembles, beyond linear models?
• RQ4 How do portfolio-style metrics—such as forecast concentration, turnover, and short position—enhance the interpretation and evaluation of ML forecasts in macroeconomics?
• RQ5 To what extent do different models (e.g., KRR, RF, NN) converge on similar historical analogies when forecasting post-pandemic economic outcomes?

Key findings

• The dual interpretation method successfully decomposes forecasts into contributions from individual historical observations, with proximity weights providing a clear narrative link between current conditions and past economic episodes.
• For kernel ridge regression, proximity weights are directly obtainable from the dual solution, enabling exact and efficient interpretation without additional computation.
• Neural networks with linear output layers allow exact retrieval of proximity weights by reformulating the final layer as a ridge regression over hidden activations, preserving interpretability.
• Tree-based models like random forest and boosted trees enable proximity-weighted interpretation through ensemble-level accounting of individual tree contributions, despite not natively outputting proximity matrices.
• In post-pandemic recession forecasting, KRR, RF, and NN models show qualitative agreement in identifying the early 1990s and 2000s recessions as key analogs, with KRR predicting alarmingly high risks (89% in 2023M8) due to strong proximity to past twin recessions.
• The method reveals that YC’s predictions exhibit upward-sloping contribution paths due to the combined effect of positive recession contributions and negative expansion contributions, while RF shows flatter paths due to the absence of short positions in its 0-1 encoding.