[Paper Review] Deep Learning in Business Analytics: A Clash of Expectations and Reality

Marc Andreas Schmitt|arXiv (Cornell University)|May 19, 2022

Big Data and Business Intelligence17 citations

TL;DR

This paper investigates the mismatch between high expectations and real-world performance of deep learning (DL) in business analytics, demonstrating that DL underperforms traditional machine learning models—particularly gradient boosting—on structured, fixed-length tabular data. The study concludes that DL should be viewed as a complementary tool, not a universal solution, with gradient boosting emerging as the superior choice for most business analytics tasks involving structured data.

ABSTRACT

Our fast-paced digital economy shaped by global competition requires increased data-driven decision-making based on artificial intelligence (AI) and machine learning (ML). The benefits of deep learning (DL) are manifold, but it comes with limitations that have, so far, interfered with widespread industry adoption. This paper explains why DL, despite its popularity, has difficulties speeding up its adoption within business analytics. It is shown that the adoption of deep learning is not only affected by computational complexity, lacking big data architecture, lack of transparency (black-box), skill shortage, and leadership commitment, but also by the fact that DL does not outperform traditional ML models in the case of structured datasets with fixed-length feature vectors. Deep learning should be regarded as a powerful addition to the existing body of ML models instead of a one size fits all solution. The results strongly suggest that gradient boosting can be seen as the go-to model for predictions on structured datasets within business analytics. In addition to the empirical study based on three industry use cases, the paper offers a comprehensive discussion of those results, practical implications, and a roadmap for future research.

Motivation & Objective

• To investigate why deep learning adoption remains limited in business analytics despite its popularity.
• To compare the performance of deep learning models against traditional machine learning models on structured business datasets.
• To identify key barriers—such as computational complexity, lack of transparency, and skill shortages—that hinder DL adoption in industry.
• To provide evidence-based guidance on when and how to apply deep learning versus other ML models in business analytics contexts.
• To propose a roadmap for future research on DL in structured data environments.

Proposed method

• Empirical evaluation using three real-world industry use cases involving structured tabular datasets.
• Implementation and comparison of deep learning models (e.g., feedforward neural networks) with traditional machine learning models, particularly gradient boosting (XGBoost, LightGBM).
• Standardized data preprocessing and feature engineering applied across all models to ensure fair comparison.
• Model evaluation using standard regression and classification metrics (e.g., RMSE, AUC-ROC) on held-out test sets.
• Analysis of model interpretability, training time, and resource requirements to assess practical feasibility.
• Qualitative discussion of organizational and technical barriers to DL adoption in business settings.

Experimental results

Research questions

• RQ1 Does deep learning outperform traditional machine learning models on structured business datasets with fixed-length feature vectors?
• RQ2 What are the primary technical and organizational barriers limiting deep learning adoption in business analytics?
• RQ3 How do factors like model interpretability, computational cost, and data infrastructure affect the practical deployment of deep learning in industry?
• RQ4 In what scenarios does gradient boosting surpass deep learning in predictive performance for business analytics tasks?
• RQ5 What are the implications of these findings for model selection and organizational strategy in data-driven decision-making?

Key findings

• Deep learning models did not outperform gradient boosting models on any of the three industry use cases involving structured tabular data.
• Gradient boosting consistently achieved superior predictive performance across all evaluated metrics, including RMSE and AUC-ROC.
• Deep learning models required significantly more computational resources and training time compared to gradient boosting.
• The lack of interpretability and 'black-box' nature of deep learning models were major concerns for business stakeholders, even when performance was comparable.
• Organizational barriers—such as skill shortages, leadership resistance, and insufficient data infrastructure—were identified as key inhibitors to DL adoption.
• The study concludes that deep learning should not be considered a one-size-fits-all solution but rather a specialized tool best used in conjunction with established models like gradient boosting.