[Paper Review] Embeddings for Tabular Data: A Survey

Rajat Singh, Srikanta Bedathur|arXiv (Cornell University)|Feb 23, 2023

Data Quality and Management4 citations

TL;DR

This survey provides a comprehensive overview of representation learning techniques for tabular data, categorizing methods by how tables are visualized—image, graph, text, or native table format. It identifies key challenges in handling heterogeneous features and dependencies, and evaluates state-of-the-art models across tasks like Table-to-SQL, metadata prediction, and formula generation, highlighting deep learning's superiority on large datasets.

ABSTRACT

Tabular data comprising rows (samples) with the same set of columns (attributes, is one of the most widely used data-type among various industries, including financial services, health care, research, retail, and logistics, to name a few. Tables are becoming the natural way of storing data among various industries and academia. The data stored in these tables serve as an essential source of information for making various decisions. As computational power and internet connectivity increase, the data stored by these companies grow exponentially, and not only do the databases become vast and challenging to maintain and operate, but the quantity of database tasks also increases. Thus a new line of research work has been started, which applies various learning techniques to support various database tasks for such large and complex tables. In this work, we split the quest of learning on tabular data into two phases: The Classical Learning Phase and The Modern Machine Learning Phase. The classical learning phase consists of the models such as SVMs, linear and logistic regression, and tree-based methods. These models are best suited for small-size tables. However, the number of tasks these models can address is limited to classification and regression. In contrast, the Modern Machine Learning Phase contains models that use deep learning for learning latent space representation of table entities. The objective of this survey is to scrutinize the varied approaches used by practitioners to learn representation for the structured data, and to compare their efficacy.

Motivation & Objective

• To systematize and categorize existing approaches for learning distributed representations (embeddings) of tabular data.
• To identify and analyze the unique challenges posed by tabular data, including heterogeneous features, inter-column dependencies, and data sparsity.
• To map current machine learning techniques to specific tabular data tasks such as Table-to-SQL, metadata prediction, and formula generation.
• To provide a structured overview of benchmark datasets and their roles in training and evaluating tabular representation models.
• To compare classical machine learning models with modern deep learning approaches in terms of scalability, interpretability, and performance on structured data.

Proposed method

• Categorizing tabular representation methods based on how tables are visualized: as images, graphs, natural language sequences, or native tabular structures.
• Classifying models into four main categories: image-based (e.g., treating tables as pixel grids), graph-based (modeling table structure as a graph), text-based (encoding tables as token sequences), and native table models (processing tabular format directly).
• Using a two-phase framework: Classical Learning Phase (SVMs, logistic regression, tree-based models) and Modern Machine Learning Phase (deep neural networks for end-to-end representation learning).
• Applying attention mechanisms and pretraining strategies (e.g., in TaBERT and TURL) to capture column-level semantics and entity relations in tables.
• Employing visibility matrices and token concatenation (e.g., cell value + column type) to model column types and table structure in sequence-based models.
• Evaluating models on standard benchmarks such as SPIDER (for Table-to-SQL), Wikipedia Tables (for TQA and SP), and WikiSQL (for semantic parsing and retrieval).

Experimental results

Research questions

• RQ1 How do different visualization paradigms (image, graph, text, native table) affect the performance and generalization of tabular representation models?
• RQ2 What are the key limitations of classical machine learning models (e.g., SVMs, tree-based models) when applied to large-scale or complex tabular data?
• RQ3 To what extent do deep learning models improve performance on downstream tasks such as Table-to-SQL parsing, formula prediction, and metadata detection?
• RQ4 How do model architectures handle heterogeneous feature types (numerical, categorical, text) in tabular data, and what techniques are used to unify them in a shared embedding space?
• RQ5 Which benchmark datasets are most representative of real-world tabular data challenges, and how do they support the evaluation of representation learning methods?

Key findings

• Deep learning models outperform classical models on large-scale tabular datasets, particularly when end-to-end training and complex feature interactions are required.
• Image-based and graph-based models show strong performance on tasks requiring structural understanding, such as Table-to-SQL and relation detection, but may struggle with long-range dependencies.
• Text-based models like TaBERT and TURL achieve state-of-the-art results on semantic parsing tasks by jointly encoding table content and column metadata using attention mechanisms.
• The SPIDER dataset, with 10,181 natural language questions and 5,693 corresponding SQL queries, is a key benchmark for evaluating Table-to-SQL models, highlighting the importance of natural language grounding in tabular data.
• Wikipedia Tables, containing 1.6 million tables with surrounding text, is one of the largest publicly available datasets for training and evaluating table representation models.
• Heterogeneous tabular data (e.g., Arrhythmia, MNIST with image modality) poses greater challenges than homogeneous data, requiring multimodal embedding strategies for effective representation learning.