[Paper Review] The DeepCAR Method: Forecasting Time-Series Data That Have Change Points

Ayla Jungbluth, Johannes Lederer|arXiv (Cornell University)|Feb 22, 2023

Time Series Analysis and Forecasting4 citations

TL;DR

This paper proposes DeepCAR, a novel extension of the DeepAR framework that improves time-series forecasting by detecting and incorporating change points through batch size adaptation. By modifying training batches to exclude or isolate change points, DeepCAR maintains performance on clean data and significantly improves accuracy on data with change points, outperforming standard DeepAR and naïve baselines by up to 9% in test RMSE on real-world and synthetic datasets.

ABSTRACT

Many methods for time-series forecasting are known in classical statistics, such as autoregression, moving averages, and exponential smoothing. The DeepAR framework is a novel, recent approach for time-series forecasting based on deep learning. DeepAR has shown very promising results already. However, time series often have change points, which can degrade the DeepAR's prediction performance substantially. This paper extends the DeepAR framework by detecting and including those change points. We show that our method performs as well as standard DeepAR when there are no change points and considerably better when there are change points. More generally, we show that the batch size provides an effective and surprisingly simple way to deal with change points in DeepAR, Transformers, and other modern forecasting models.

Motivation & Objective

• Address the limitation of DeepAR and other deep learning models in handling time-series with change points, which degrade forecasting performance.
• Develop a method that integrates change-point detection into existing deep learning forecasting frameworks without requiring model retraining or architectural changes.
• Preserve the original forecasting model's characteristics and implementation while improving robustness to structural shifts in time-series data.
• Demonstrate that batch size selection can serve as a general-purpose mechanism to account for change points across various models, including Transformers and TFT.
• Enable seamless integration of any existing change-point detection method into the training pipeline of batch-trained forecasting models.

Proposed method

• Adapt the training batch generation process in DeepAR to exclude or isolate time windows that contain change points, using detected change points as boundaries.
• Apply change-point detection algorithms (e.g., MOSUM) to identify structural shifts in time-series data prior to model training.
• Use the detected change points to define batch boundaries such that no training window spans across a change point, reducing model confusion.
• Maintain the original DeepAR architecture and likelihood-based training process, only modifying the sampling of training instances.
• Generalize the approach to other batch-trained models like Transformers and Temporal Fusion Transformers (TFT) by applying the same batch-level filtering mechanism.
• Implement the method via a modular pipeline (called BatchCP) that can be plugged into existing frameworks like GluonTS without altering the core model.

Experimental results

Research questions

• RQ1 Can change points in time-series data be effectively handled by modifying the training batch sampling process in DeepAR?
• RQ2 Does integrating detected change points into the training batches improve forecasting accuracy compared to standard DeepAR on data with structural shifts?
• RQ3 Can the batch size be used as a general-purpose mechanism to account for change points across diverse deep learning forecasting models?
• RQ4 How does the performance of DeepCAR compare to vanilla DeepAR and naïve baselines on real-world time-series with known change points?
• RQ5 To what extent can the proposed method be generalized to attention-based models like Transformers and TFT?

Key findings

• DeepCAR outperforms standard DeepAR by up to 9% in test RMSE on the Treasury dataset when using MOSUM-based change-point detection, achieving a test RMSE of 5.61 compared to DeepAR’s 5.93.
• On synthetic data with 13 controlled change points, DeepCAR with manual detection reduced test RMSE from 5.77 (DeepAR) to 5.40, demonstrating a 6.4% improvement.
• In the pollution dataset, DeepCAR with MOSUM-detection reduced test RMSE from 96.23 (Transformer without CPD) to 93.12, showing a 3.2% improvement despite higher baseline error.
• DeepCAR maintains performance comparable to DeepAR on data without change points, confirming it does not degrade performance when change points are absent.
• The method generalizes effectively to Transformers and TFT, improving their forecasting accuracy when change points are detected and excluded from training batches.
• The batch size selection strategy provides a surprisingly simple and effective way to handle change points across multiple models, including DeepAR, Transformers, and TFT.