[Paper Review] Deep Neural Networks and Tabular Data: A Survey

Vadim Borisov, Tobias Leemann+4 more|arXiv (Cornell University)|Oct 5, 2021

Explainable Artificial Intelligence (XAI)29 citations

TL;DR

A comprehensive survey of deep learning for heterogeneous tabular data, proposing a taxonomy (data transformation, specialized architectures, regularization), surveying generation and explainability, and an empirical benchmark showing gradient-boosted trees often outperform deep models on supervised tasks.

ABSTRACT

Heterogeneous tabular data are the most commonly used form of data and are essential for numerous critical and computationally demanding applications. On homogeneous data sets, deep neural networks have repeatedly shown excellent performance and have therefore been widely adopted. However, their adaptation to tabular data for inference or data generation tasks remains challenging. To facilitate further progress in the field, this work provides an overview of state-of-the-art deep learning methods for tabular data. We categorize these methods into three groups: data transformations, specialized architectures, and regularization models. For each of these groups, our work offers a comprehensive overview of the main approaches. Moreover, we discuss deep learning approaches for generating tabular data, and we also provide an overview over strategies for explaining deep models on tabular data. Thus, our first contribution is to address the main research streams and existing methodologies in the mentioned areas, while highlighting relevant challenges and open research questions. Our second contribution is to provide an empirical comparison of traditional machine learning methods with eleven deep learning approaches across five popular real-world tabular data sets of different sizes and with different learning objectives. Our results, which we have made publicly available as competitive benchmarks, indicate that algorithms based on gradient-boosted tree ensembles still mostly outperform deep learning models on supervised learning tasks, suggesting that the research progress on competitive deep learning models for tabular data is stagnating. To the best of our knowledge, this is the first in-depth overview of deep learning approaches for tabular data; as such, this work can serve as a valuable starting point to guide researchers and practitioners interested in deep learning with tabular data.

Motivation & Objective

• Provide a thorough review of existing literature on deep learning for tabular data across supervised, unsupervised, data generation, and interpretability tasks.
• Propose a unified taxonomy (data transformation, specialized architectures, regularization) to organize methods for heterogeneous tabular data.
• Summarize tabular data generation techniques and explainability approaches for deep models on tabular data.
• Deliver an extensive empirical comparison between traditional ML methods and deep learning approaches on real-world datasets, and offer open benchmarks to enable replication.

Proposed method

• Introduce a unified taxonomy of deep learning methods for tabular data (data transformation, specialized architectures, regularization).
• Survey data transformation techniques for categorical and numerical features, including single- and multi-dimensional encodings.
• Describe specialized architectures, including hybrid models and transformer-based models, for tabular data.
• Review regularization strategies to improve deep models on tabular data.
• Discuss tabular data generation approaches and evaluation of generative quality.
• Present explanations mechanisms for deep models on tabular data and an empirical benchmarking framework.

Experimental results

Research questions

• RQ1 What are the main research streams and methodologies for applying deep learning to heterogeneous tabular data?
• RQ2 How do deep learning approaches for tabular data compare with traditional methods on real-world datasets?
• RQ3 What are the open challenges and future directions for inference, generation, and interpretability in tabular data with deep learning?
• RQ4 Can a unified taxonomy help practitioners select appropriate methods for tabular data tasks?

Key findings

• Gradient-boosted tree ensembles still largely outperform deep learning models on supervised tabular tasks across tested datasets.
• Deep learning progress for tabular data appears stagnating relative to strong tree-based baselines, indicating room for methodological advances.
• The paper provides an open benchmark and code to reproduce and extend comparative results.
• A broad survey identifies core challenges in tabular data: data quality, irregular feature dependencies, preprocessing reliance, and feature-wise sensitivity.