[Paper Review] Adjusting for Autocorrelated Errors in Neural Networks for Time Series

Fan-Keng Sun, Christopher I. Lang+1 more|arXiv (Cornell University)|Jan 28, 2021

Time Series Analysis and ForecastingComputer Science51 references17 citations

TL;DR

This paper proposes a method to improve neural network performance on time series by jointly learning the autocorrelation coefficient of errors alongside model parameters via gradient descent. Experiments across diverse datasets and architectures show consistent performance gains in forecasting, regression, classification, and anomaly detection, with empirical critical values provided to guide when adjustment is necessary.

ABSTRACT

An increasing body of research focuses on using neural networks to model time series. A common assumption in training neural networks via maximum likelihood estimation on time series is that the errors across time steps are uncorrelated. However, errors are actually autocorrelated in many cases due to the temporality of the data, which makes such maximum likelihood estimations inaccurate. In this paper, in order to adjust for autocorrelated errors, we propose to learn the autocorrelation coefficient jointly with the model parameters. In our experiments, we verify the effectiveness of our approach on time series forecasting. Results across a wide range of real-world datasets with various state-of-the-art models show that our method enhances performance in almost all cases. Based on these results, we suggest empirical critical values to determine the severity of autocorrelated errors. We also analyze several aspects of our method to demonstrate its advantages. Finally, other time series tasks are also considered to validate that our method is not restricted to only forecasting.

Motivation & Objective

• To address the common but problematic assumption in neural network training that time series errors are uncorrelated, despite real-world autocorrelation due to omitted variables, measurement noise, and model misspecification.
• To develop a generalizable method that adjusts for autocorrelated errors in neural networks without requiring prior knowledge of the true data-generating process.
• To empirically determine critical thresholds for residual autocorrelation that signal when adjustment is beneficial.
• To validate the method's effectiveness across multiple time series tasks beyond forecasting, including regression, classification, and anomaly detection.

Proposed method

• The method introduces a differentiable loss function that incorporates an estimated autocorrelation coefficient ρ, learned jointly with neural network parameters via gradient descent.
• The loss function modifies the standard maximum likelihood estimation by accounting for first-order autocorrelation in the error terms using a multivariate normal distribution with a Toeplitz covariance matrix.
• The model is trained end-to-end using backpropagation, allowing ρ to be optimized alongside network weights, enabling automatic detection and correction of autocorrelated errors.
• The approach is applied to both input and output sequences in the loss function, with ablation studies confirming that joint adjustment yields the greatest performance gains.
• Empirical critical values for residual autocorrelation are derived from large-scale experiments to guide practitioners on when to apply the adjustment.
• The method is extended to other time series tasks by treating each task's prediction error as a time series to be adjusted for autocorrelation.

Experimental results

Research questions

• RQ1 Can joint learning of the autocorrelation coefficient ρ with neural network parameters improve time series forecasting performance across diverse datasets and architectures?
• RQ2 What empirical thresholds of residual autocorrelation indicate a need for adjustment, and how can they be used as practical guidelines?
• RQ3 How does the performance gain from adjusting for autocorrelated errors vary with model expressiveness, such as in LSTMs with different numbers of hidden units?
• RQ4 Does the proposed method generalize beyond forecasting to other time series tasks like regression, classification, and anomaly detection?
• RQ5 What is the relative contribution of adjusting for autocorrelation in input vs. output sequences, and is joint adjustment optimal?

Key findings

• The proposed method improves performance in almost all time series forecasting experiments across a wide range of real-world datasets and state-of-the-art neural network architectures.
• The relative improvement from adjusting for autocorrelation decreases as model expressiveness increases, such as with larger LSTM architectures, indicating that more complex models reduce the need for correction.
• Empirical critical values for residual autocorrelation are identified, providing a practical threshold to determine whether adjustment is necessary.
• Ablation studies show that adjusting for autocorrelation in both input and output sequences yields the greatest performance gains, with individual adjustments offering smaller but still beneficial improvements.
• The method maintains or improves performance across multiple time series tasks, including regression, classification, and unsupervised anomaly detection, demonstrating broad applicability.
• The method is less effective when the model is already highly expressive and well-specified, as residual autocorrelation diminishes, but remains beneficial in real-world scenarios where the true data-generating process is unknown.