[Paper Review] AdaRNN: Adaptive Learning and Forecasting of Time Series

Yuntao Du, Jindong Wang+5 more|arXiv (Cornell University)|Aug 10, 2021

Time Series Analysis and Forecasting56 references36 citations

TL;DR

AdaRNN introduces Temporal Covariate Shift (TCS) and a two-module framework (Temporal Distribution Characterization and Temporal Distribution Matching) to adapt RNNs for non-stationary time series, improving classification and regression performance.

ABSTRACT

Time series has wide applications in the real world and is known to be difficult to forecast. Since its statistical properties change over time, its distribution also changes temporally, which will cause severe distribution shift problem to existing methods. However, it remains unexplored to model the time series in the distribution perspective. In this paper, we term this as Temporal Covariate Shift (TCS). This paper proposes Adaptive RNNs (AdaRNN) to tackle the TCS problem by building an adaptive model that generalizes well on the unseen test data. AdaRNN is sequentially composed of two novel algorithms. First, we propose Temporal Distribution Characterization to better characterize the distribution information in the TS. Second, we propose Temporal Distribution Matching to reduce the distribution mismatch in TS to learn the adaptive TS model. AdaRNN is a general framework with flexible distribution distances integrated. Experiments on human activity recognition, air quality prediction, and financial analysis show that AdaRNN outperforms the latest methods by a classification accuracy of 2.6% and significantly reduces the RMSE by 9.0%. We also show that the temporal distribution matching algorithm can be extended in Transformer structure to boost its performance.

Motivation & Objective

• Formulate Temporal Covariate Shift (TCS) for non-stationary time series forecasting.
• Split time series into periods that maximize distribution diversity to capture shared knowledge.
• Learn an adaptive RNN-based predictor by matching distributions across discovered periods.

Proposed method

• Define TCS and propose AdaRNN with two modules: Temporal Distribution Characterization (TDC) and Temporal Distribution Matching (TDM).
• TDC splits the series into K periods by maximizing inter-period distribution distance (max-entropy principle).
• TDM learns a prediction model by minimizing prediction loss plus a cross-period distribution-matching regularizer applied to RNN hidden states.
• Introduce a Boosting-based scheme to adapt per-state importance weights (alpha) for distribution matching.
• Allow flexibility in distance metrics (e.g., cosine, MMD, adversarial) and RNN variants (GRU/LSTM).
• Provide an end-to-end training algorithm (AdaRNN) that alternates period discovery and joint optimization of network parameters and state importances.

Experimental results

Research questions

• RQ1 How to model time series under temporally varying distributions (TCS) for robust forecasting?
• RQ2 Can we automatically discover periods with maximal distribution diversity to capture shared knowledge across time?
• RQ3 Does distribution matching across discovered periods improve predictive generalization for RNN-based models?

Key findings

• AdaRNN outperforms state-of-the-art baselines on classification and regression tasks in non-stationary time series settings.
• On UCI activity recognition, AdaRNN achieves 2.56% higher accuracy and 3.07% higher F1 than the best baseline DANN-RNN.
• On air quality and electric power prediction, AdaRNN achieves notable RMSE/MAE reductions across stations and datasets (as reported in the study).
• The temporal distribution matching module can be extended to Transformer architectures for further gains.
• AdaRNN is agnostic to the underlying RNN (vanilla RNN, LSTM, GRU) and distribution distance (cosine, MMD, adversarial).