[Paper Review] From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models

Xumeng Wen, Han Zhang+3 more|arXiv (Cornell University)|Oct 11, 2023

Topic Modeling4 citations

TL;DR

This paper introduces Tabular Foundation Models (TabFMs), a novel paradigm that fine-tunes large language models (LLMs) using generative tabular learning (GTL) on 115 diverse tabular datasets to enable both instruction following and few-shot fine-tuning. The approach achieves performance comparable to or exceeding GPT-4 in zero-shot and in-context learning, while demonstrating strong data efficiency in low-shot scenarios.

ABSTRACT

Tabular data is foundational to predictive modeling in various crucial industries, including healthcare, finance, retail, sustainability, etc. Despite the progress made in specialized models, there is an increasing demand for universal models that can transfer knowledge, generalize from limited data, and follow human instructions. These are challenges that current tabular deep learning approaches have not fully tackled. Here we introduce Generative Tabular Learning (GTL), a novel framework that integrates the advanced functionalities of large language models (LLMs)-such as prompt-based zero-shot generalization and in-context learning-into tabular deep learning. GTL capitalizes on the pre-training of LLMs on diverse tabular data, enhancing their understanding of domain-specific knowledge, numerical sequences, and statistical dependencies critical for accurate predictions. Our empirical study spans 384 public datasets, rigorously analyzing GTL's convergence and scaling behaviors and assessing the impact of varied data templates. The GTL-enhanced LLaMA-2 model demonstrates superior zero-shot and in-context learning capabilities across numerous classification and regression tasks. Notably, it achieves this without fine-tuning, outperforming traditional methods and rivaling state-of-the-art models like GPT-4 in certain cases. Through GTL, we not only foster a deeper integration of LLMs' sophisticated abilities into tabular data comprehension and application but also offer a new training resource and a test bed for LLMs to enhance their ability to comprehend tabular data. To facilitate reproducible research, we release our code, data, and model checkpoints at https://github.com/microsoft/Industrial-Foundation-Models.

Motivation & Objective

• To address the limitations of existing transferable tabular models that lack robust instruction-following and foundational knowledge acquisition.
• To enable large language models to learn data-driven knowledge from diverse tabular datasets, beyond general text knowledge.
• To develop a unified framework that supports both zero-shot instruction following and few-shot fine-tuning for new tabular tasks.
• To improve the generalization and data efficiency of LLMs in tabular learning through domain-agnostic pre-training on heterogeneous tabular data.
• To explore the potential of tabular data as a new corpus for enhancing LLM capabilities in structured data understanding.

Proposed method

• Fine-tune a pre-trained LLM (LLaMA-2) using a purpose-designed generative tabular learning (GTL) objective on 115 public tabular datasets.
• Design a sequence-to-sequence training objective that enables the LLM to generate complete, coherent tabular data samples from natural language descriptions.
• Use a diverse collection of tabular datasets spanning healthcare, finance, retail, and climate domains to ensure broad foundational knowledge acquisition.
• Support both instruction-following (zero-shot) and few-shot fine-tuning by leveraging the LLM’s in-context learning and prompt engineering capabilities.
• Employ structured prompt templates to guide the LLM in understanding tabular schema, numerical patterns, and statistical relationships.
• Evaluate performance across nine held-out datasets using standard tabular benchmarks, including AUROC and accuracy metrics.

Figure 1: An overview of generative tabuar learning for LLMs.

Experimental results

Research questions

• RQ1 Can a large language model be effectively fine-tuned to learn foundational knowledge from diverse tabular datasets to support both instruction following and few-shot adaptation?
• RQ2 How does generative pre-training on tabular data improve the performance of LLMs in zero-shot and in-context learning compared to standard fine-tuning?
• RQ3 To what extent can a foundation model trained on tabular data match or surpass closed-source models like GPT-4 in tabular reasoning tasks?
• RQ4 How does the model perform under data-scarce conditions, and does it demonstrate data efficiency in few-shot scenarios?
• RQ5 What are the key limitations of current LLMs in understanding tabular data, and how can they be mitigated through domain-specific pre-training?

Key findings

• The proposed TabFMs significantly outperform the base LLaMA-2 model and GPT-3.5 in instruction-following tasks, especially in zero-shot and in-context learning settings.
• On nine held-out tabular datasets, TabFMs achieve performance that approaches or occasionally surpasses GPT-4, particularly in few-shot and zero-shot scenarios.
• In few-shot fine-tuning, the model achieves competitive performance with state-of-the-art tabular models like TabPFN and XGBoost, demonstrating high data efficiency.
• The model shows substantial performance gains on datasets like Bank, Blood, and C. Hous. in few-shot settings, where the base LLaMA model fails to generalize.
• With abundant training data, the performance gain from GTL diminishes, indicating that statistical patterns dominate generalization when data is plentiful.
• Failure cases, such as on the Jungle dataset (AUROC 0.69 vs. 0.80 for SOTA), suggest limitations in LLMs’ knowledge of niche domains like gaming, possibly due to pre-training corpus bias or suboptimal tokenization.

Figure 2: Ablation study of instruction following task. Both the numerical loss and in-context training improves the prediction performance.