[Paper Review] Forecasting with sktime: Designing sktime's New Forecasting API and Applying It to Replicate and Extend the M4 Study

Markus Löning, Franz J. Király|arXiv (Cornell University)|May 16, 2020

Forecasting Techniques and Applications70 references4 citations

TL;DR

This paper introduces sktime's new forecasting API, a scikit-learn-compatible framework for building, tuning, and evaluating composite machine learning models for univariate time series forecasting. It replicates and extends the M4 forecasting study, demonstrating that simple hybrid and pure machine learning approaches can outperform statistical models and approach the performance of top M4 models, especially on hourly data.

ABSTRACT

We present a new open-source framework for forecasting in Python. Our framework forms part of sktime, a more general machine learning toolbox for time series with scikit-learn compatible interfaces for different learning tasks. Our new framework provides dedicated forecasting algorithms and tools to build, tune and evaluate composite models. We use sktime to both replicate and extend key results from the M4 forecasting study. In particular, we further investigate the potential of simple off-the-shelf machine learning approaches for univariate forecasting. Our main results are that simple hybrid approaches can boost the performance of statistical models, and that simple pure approaches can achieve competitive performance on the hourly data set, outperforming the statistical algorithms and coming close to the M4 winner.

Motivation & Objective

• To address the lack of open-source, scikit-learn-compatible toolkits for building, tuning, and evaluating composite machine learning models in time series forecasting.
• To provide a composable and modular forecasting API that supports pipelining, tuning, and meta-forecasting techniques like reduction and detrending.
• To replicate and validate the results of the M4 forecasting competition using a standardized, open-source framework.
• To extend the M4 study by evaluating off-the-shelf machine learning approaches—such as reduction, boosting, and tuning—on univariate forecasting tasks.
• To demonstrate that simple, off-the-shelf machine learning models can achieve competitive performance, rivaling or surpassing statistical models and approaching the M4 winner’s accuracy.

Proposed method

• Design a forecasting API in sktime that mirrors scikit-learn’s estimator interface, enabling consistent use of estimators, transformers, and pipelines for time series.
• Implement meta-forecasters such as reduction (for transforming forecasting into regression), detrending, and residual boosting to enable hybrid modeling.
• Use nested cross-validation with time series splits to tune hyperparameters of machine learning models (e.g., random forests, XGBoost, KNN, linear regression) across M4 data sets.
• Apply pipelining to combine feature engineering, model fitting, and forecasting steps into reusable, composable workflows.
• Replicate M4 results using the same data splits and evaluation metrics (MASE, sMAPE) to validate implementation correctness.
• Extend the M4 study by evaluating new model configurations, including tuned and hybrid models, across all M4 frequency groups (yearly to hourly).

Experimental results

Research questions

• RQ1 Can sktime’s new forecasting API accurately replicate the M4 forecasting competition results using off-the-shelf machine learning models?
• RQ2 How do simple, off-the-shelf machine learning models (e.g., random forests, XGBoost) perform on univariate time series forecasting compared to statistical models in the M4 benchmark?
• RQ3 To what extent can hybrid approaches—such as residual boosting of statistical models—improve forecasting accuracy?
• RQ4 Can simple model tuning and composition techniques (e.g., pipelining, reduction) significantly enhance performance on challenging M4 data sets, especially hourly data?
• RQ5 How does the performance of pure machine learning models compare to the M4 competition winner, particularly on the hourly forecasting task?

Key findings

• The replication of the M4 study shows no significant differences for naive models and small differences for statistical models, but large performance improvements for machine learning models when using sktime’s implementation.
• Simple hybrid models that boost statistical forecasts using machine learning on residuals significantly improve accuracy, particularly on the daily and weekly data sets.
• Pure machine learning models—especially tuned random forests and XGBoost—achieved competitive performance on the hourly data set, outperforming all statistical models and coming close to the M4 winner’s performance.
• The tuned XGBoost-s model achieved an OWA of 1.131 across all M4 data sets, while the tuned random forest (RF-t-s) achieved 1.047, indicating strong generalization across frequencies.
• On the hourly data set, the best-performing model (XGB-t-s) achieved an OWA of 0.746, significantly outperforming statistical models and approaching the M4 winner’s performance.
• Linear regression models were unstable and excluded from results due to exploding forecasts, particularly on low-data-frequency series, highlighting limitations of linear extrapolation in univariate settings.