[Paper Review] Comparing statistical and machine learning methods for time series forecasting in data-driven logistics -- A simulation study

Lena Schmid, Moritz Roidl+1 more|arXiv (Cornell University)|Mar 13, 2023

Big Data and Business IntelligenceBusiness, Management and Accounting3 citations

TL;DR

This simulation study compares statistical (ARIMA, TBATS) and machine learning (Random Forest, XGBoost) methods for one-step time series forecasting in logistics. Using synthetic data with varying complexities, it finds that ML methods, especially with differenced data, outperform traditional models in non-linear and jump-affected settings, while both approaches perform similarly in linear or random walk scenarios.

ABSTRACT

Many planning and decision activities in logistics and supply chain management are based on forecasts of multiple time dependent factors. Therefore, the quality of planning depends on the quality of the forecasts. We compare various forecasting methods in terms of out of the box forecasting performance on a broad set of simulated time series. We simulate various linear and non-linear time series and look at the one step forecast performance of statistical learning methods.

Motivation & Objective

• To evaluate and compare the out-of-the-box forecasting performance of statistical and machine learning methods in logistics-relevant time series.
• To investigate how different data-generating processes—linear, non-linear, and those with jumps or random walks—affect forecasting accuracy.
• To assess the impact of data preprocessing, particularly differencing, on machine learning model performance.
• To provide empirical guidance for practitioners on selecting forecasting methods based on data characteristics and complexity.
• To identify settings where ML methods significantly outperform traditional time series models in logistics applications.

Proposed method

• Simulated 15 distinct time series data-generating processes, including AR, BL, SAR, NAR, NMA, STAR, and TAR models, with and without additional complexities.
• Applied four forecasting methods: (seasonal) ARIMA, TBATS, Random Forest, and XGBoost, all using one-step-ahead predictions.
• Used a sliding window approach with a window size of 8 for training and forecasting, ensuring consistent evaluation across methods.
• Evaluated performance using Mean Squared Error (MSE) across multiple sample sizes (100 to 500), with 100 replications per setting.
• Applied data differencing to ML models to assess its impact on predictive accuracy, particularly in non-stationary or complex settings.
• Conducted a comprehensive simulation framework with controlled data structures to isolate the effects of non-linearity, jumps, and random noise.

Experimental results

Research questions

• RQ1 In what data-generating processes do machine learning methods (Random Forest, XGBoost) outperform traditional time series models (ARIMA, TBATS) in one-step forecasting?
• RQ2 How does data differencing affect the predictive performance of tree-based machine learning models in complex time series settings?
• RQ3 How do additional complexities—such as compound Poisson jumps or random walks—affect the forecasting accuracy of statistical and machine learning methods?
• RQ4 Under which conditions do time series models outperform or match the performance of machine learning models in synthetic logistics-relevant time series?
• RQ5 Does the performance gap between ML and statistical models vary systematically with time series length and data complexity?

Key findings

• Random Forest with differenced data outperformed all other methods in settings with a compound Poisson jump process, especially as time series length increased to 500.
• For data with a random walk component, MSE values remained relatively stable or slightly decreased with increasing time series length, and ML methods with differenced data showed improved performance over non-differenced versions.
• In linear and stable settings (e.g., AR, BL2), time series models performed slightly better or comparably to ML methods, with minimal differences in MSE.
• In non-linear settings (e.g., NAR1, NAR2, STAR1, TAR1), ML methods showed mixed results: Random Forests outperformed XGBoost, but both were outperformed by time series models in some cases.
• The inclusion of a jump process led to a monotonic increase in MSE across all methods, with values exceeding 2000 for time series of length 500, indicating high sensitivity to structural breaks.
• Data differencing significantly improved the performance of ML models in non-linear and complex settings, suggesting that preprocessing is crucial for ML success in time series forecasting.