[Paper Review] Demand Prediction Using Machine Learning Methods and Stacked Generalization

Resul Tugay, Şule Gündüz Öğüdücü|arXiv (Cornell University)|Sep 21, 2020

Customer churn and segmentation4 citations

TL;DR

This paper proposes a stacked generalization (stacking) ensemble model for demand prediction on a Turkish e-commerce platform with competing sellers offering the same product at different prices. By combining multiple machine learning regressors—gradient boosting, random forest, linear regression, and decision trees—the model achieves superior performance, with the best configuration (GBT + RF + LR) reducing RMSE to 1.864, outperforming individual models and demonstrating statistical significance with less training data.

ABSTRACT

Supply and demand are two fundamental concepts of sellers and customers. Predicting demand accurately is critical for organizations in order to be able to make plans. In this paper, we propose a new approach for demand prediction on an e-commerce web site. The proposed model differs from earlier models in several ways. The business model used in the e-commerce web site, for which the model is implemented, includes many sellers that sell the same product at the same time at different prices where the company operates a market place model. The demand prediction for such a model should consider the price of the same product sold by competing sellers along the features of these sellers. In this study we first applied different regression algorithms for specific set of products of one department of a company that is one of the most popular online e-commerce companies in Turkey. Then we used stacked generalization or also known as stacking ensemble learning to predict demand. Finally, all the approaches are evaluated on a real world data set obtained from the e-commerce company. The experimental results show that some of the machine learning methods do produce almost as good results as the stacked generalization method.

Motivation & Objective

• To address the challenge of demand prediction in a marketplace e-commerce model with competing sellers offering identical products at varying prices.
• To evaluate the effectiveness of stacked generalization in improving demand forecasting accuracy compared to individual machine learning models.
• To determine whether ensemble learning with diverse regressors can reduce prediction error while using less training data.
• To identify the optimal combination of base learners and meta-learner for demand forecasting in a real-world e-commerce setting.

Proposed method

• Employed stacked generalization (stacking) with two-level learning: first-level regressors (linear regression, random forest, gradient boosting, decision trees) trained on training data, and a meta-learner combining their predictions.
• Used 10-fold cross-validation on the training set to optimize first-level models, with validation set used to train the meta-learner.
• Split data into 50% training, 20% validation, and 30% test sets, with 20 random splits to ensure robust evaluation.
• Evaluated models using RMSE and applied ANOVA and t-tests to assess statistical significance of performance differences.
• Tested various combinations of first-level models (binary and triple combinations) and different meta-learners (e.g., linear regression, decision trees) to identify optimal configurations.
• Used real-world e-commerce data from n11.com, focusing on a single department with multiple sellers offering the same product at different prices.

Experimental results

Research questions

• RQ1 Can stacked generalization improve demand prediction accuracy compared to individual machine learning models in an e-commerce marketplace with price-competitive products?
• RQ2 Which combination of base learners (linear regression, random forest, gradient boosting, decision trees) yields the lowest prediction error in demand forecasting?
• RQ3 Does the stacking model achieve better performance with less training data than single models?
• RQ4 Is the performance improvement of the stacking model statistically significant compared to individual models?

Key findings

• The stacked generalization model with gradient boosting, random forest, and linear regression as first-level learners achieved the lowest RMSE of 1.864 on the test set.
• The binary combination of gradient boosting and random forest yielded the best single binary pair with an RMSE of 1.870.
• The meta-learner using linear regression in the second level outperformed other meta-learners, including decision trees and random forest.
• ANOVA results rejected the null hypothesis at the 5% significance level, indicating statistically significant differences in model performance.
• The t-test showed no statistically significant difference between the performance of random forest (first-level) and linear regression (second-level) models, suggesting robustness of the ensemble approach.
• The proposed stacking model achieved performance comparable to individual models but with only 20% of the training data, indicating data efficiency.