[Paper Review] Machine Learning vs Statistical Methods for Time Series Forecasting: Size Matters
The paper shows that sample size influences the relative performance of machine learning versus statistical methods in univariate time series forecasting, with ML gaining advantage as data size grows.
Time series forecasting is one of the most active research topics. Machine learning methods have been increasingly adopted to solve these predictive tasks. However, in a recent work, these were shown to systematically present a lower predictive performance relative to simple statistical methods. In this work, we counter these results. We show that these are only valid under an extremely low sample size. Using a learning curve method, our results suggest that machine learning methods improve their relative predictive performance as the sample size grows. The code to reproduce the experiments is available at https://github.com/vcerqueira/MLforForecasting.
Motivation & Objective
- Motivate a re-examination of the ML vs statistical methods debate in time series forecasting beyond small sample sizes.
- Control for sample size to compare forecasts from machine learning and statistical approaches.
- Assess one-step-ahead and multi-step-ahead forecasting under growing training data.
- Provide an empirical, reproducible framework with prequential learning curves to evaluate forecasting methods.
Proposed method
- Use 90 univariate time series with at least 1000 observations, truncated to 1000 for uniformity.
- Compare five statistical methods (ARIMA, Naive2, Theta, ETS, Tbats) with five ML-based AR(p) approaches (RBR, RF, GP, MARS, GLM).
- Apply Box-Cox transformations, seasonality handling, and trend removal prior to forecasting as in Makridakis et al. (2018).
- Employ a prequential learning-curve procedure where training size grows from 18 observations upward, forecasting the next 1 or 18 steps.
- Evaluate using mean absolute scaled error (MASE) and model ranks across time series; report average ranks per model.
Experimental results
Research questions
- RQ1Does the relative predictive performance of statistical versus machine learning methods depend on sample size in univariate time series forecasting?
- RQ2How do one-step-ahead and multi-step-ahead forecasting scenarios differ in terms of which method performs best as data size increases?
- RQ3What is the computational cost profile of the competing methods under automated parameter tuning?
- RQ4Do results vary when excluding clearly weak baselines (e.g., Naive2) from the comparison?
Key findings
- Statistical methods outperform ML methods at very small training sizes, but ML methods surpass statistical methods as the sample size grows.
- When excluding Naive2, ML methods catch up and often outperform statistical methods earlier in the learning curve for one-step-ahead forecasting.
- For multi-step-ahead forecasting, ARIMA becomes the best by average rank, but overall ML vs statistics show convergence as training size increases (except Naive2).
- ML approaches benefit from larger data but may require more data to handle increased uncertainty in multi-step forecasts.
- Computation time is higher for some automated statistical methods (e.g., ARIMA, Tbats) due to parameter optimization, while ML methods show lower relative complexity under the tested setup.
Read this paper
Open the full text in the Nubint reader and ask about any passage as you read.
Read this paper on NubintThis review was generated by AI from the original paper. Please check the paper before citing it.