[Paper Review] TAPEX: Table Pre-training via Learning a Neural SQL Executor

Qian Liu, Bei Chen+5 more|arXiv (Cornell University)|Jul 16, 2021

Topic ModelingComputer Science41 references90 citations

TL;DR

TaPEx pre-trains language models by mimicking a neural SQL executor over a synthetic SQL-table corpus, achieving state-of-the-art results on four table-related benchmarks.

ABSTRACT

Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.

Motivation & Objective

• Address data scarcity in table pre-training by creating a large, high-quality synthetic corpus.
• Develop an execution-centric pre-training task that teaches a model to mimic a SQL executor over tables.
• Integrate table pre-training with encoder-decoder architectures to improve downstream table reasoning tasks.
• Empirically evaluate TaPEx on diverse table QA and fact verification benchmarks.
• Demonstrate efficiency and effectiveness in low-data regimes compared to prior table pre-training methods.

Proposed method

• Use an encoder-decoder LM (BART) as the backbone to model table-aware inputs.
• Flatten tables into a sequence with special tokens to indicate headers and rows, enabling direct input to the model.
• Pre-train the model to output SQL execution results given a SQL query and a table, effectively making the model a neural SQL executor.
• Synthesise a large pre-training corpus by instantiating SQL templates over publicly available tables (e.g., WikiTableQuestions) and filtering out empty results.
• Fine-tune on downstream tasks (TableQA and TableFV) by formulating outputs as sequences, enabling flexible generation for answers and binary verifications.
• Optionally perform multi-task fine-tuning, and explore execution-guided decoding during inference.

Experimental results

Research questions

• RQ1 Can pre-training a language model to execute SQL-like queries over tables improve table reasoning tasks?
• RQ2 How does TaPEx perform across diverse table QA and fact verification benchmarks compared to prior table pre-training approaches?
• RQ3 Is synthetic, execution-based pre-training more data-efficient than traditional table pre-training methods?
• RQ4 Does the approach generalize well to low-resource downstream settings?

Key findings

• TaPEx achieves new state-of-the-art results on four benchmarks: WikiSQL-Weak, WikiTableQuestions, SQA, and TabFact.
• On WikiSQL-Weak, TaPEx attains 89.5% denotation accuracy on the test set, outperforming the previous best by 2.3 percentage points.
• On WikiTableQuestions, TaPEx reaches 57.5% denotation accuracy (test), a 4.8-point improvement over prior best.
• On SQA, TaPEx achieves 74.5% ALL denotation accuracy (test set), with notable gains across conversation and sentence-level metrics.
• On TabFact, TaPEx attains 74.5% ALL accuracy along with strong sub-criteria across Q1–Q3, surpassing prior models.
• TaPEx demonstrates strong performance in low-data regimes and can achieve substantial gains with smaller pre-training corpora.