[Paper Review] GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

Tao Yu, Chien-Sheng Wu+7 more|arXiv (Cornell University)|Sep 29, 2020

Topic Modeling55 references59 citations

TL;DR

GraPPa pre-trains a language model with grammar-augmented synthetic data for text-to-SQL, achieving state-of-the-art results on four table semantic parsing benchmarks in both fully and weakly supervised settings.

ABSTRACT

We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG) induced from existing text-to-SQL datasets. We pre-train our model on the synthetic data using a novel text-schema linking objective that predicts the syntactic role of a table field in the SQL for each question-SQL pair. To maintain the model's ability to represent real-world data, we also include masked language modeling (MLM) over several existing table-and-language datasets to regularize the pre-training process. On four popular fully supervised and weakly supervised table semantic parsing benchmarks, GraPPa significantly outperforms RoBERTa-large as the feature representation layers and establishes new state-of-the-art results on all of them.

Motivation & Objective

• Motivate improved generalization for table semantic parsing by grounding NL queries to tabular schemas.
• Inject a compositional inductive bias into language models via grammar-guided data synthesis.
• Maintain model capacity on real-world data by balancing synthetic data with table-related MLM regularization.
• Demonstrate data-efficient pre-training that yields strong downstream performance across multiple benchmarks.

Proposed method

• Induce a synchronous context-free grammar (SCFG) from text-to-SQL datasets to generate synthetic question-SQL pairs grounded on tables.
• Sample from the SCFG to create large synthetic datasets (475k examples) using WikiTables and Spider/WikiSQL tables.
• Pre-train a RoBERTa-based model (GraPPa) with two objectives: MLM on table-language data and SQL semantic prediction (SSP) for column-wise SQL grounding.
• Fine-tune Grappa on downstream parsers (e.g., RAT-SQL) to assess cross-domain table semantic parsing performance.
• Balance synthetic data with real table-language data by applying MLM on table-related utterances to regularize pre-training.

Experimental results

Research questions

• RQ1 Does Grappa provide better representations for cross-domain table semantic parsing compared to RoBERTa-based baselines?
• RQ2 What is the impact of the two pre-training objectives (MLM and SSP) and their combination on downstream performance?
• RQ3 How well does grammar-augmented pre-training generalize to unseen tables across fully supervised and weakly supervised settings?

Key findings

• Grappa consistently outperforms RoBERTa baselines across four semantic parsing tasks.
• Spider: Grappa with MLM+SSP achieves new state-of-the-art, surpassing prior best by about 4%.
• WikiSQL fully supervised: Grappa improves SQLova by 3.0% under low-resource (10k) settings.
• WikiTableQuestions: Grappa with MLM+SSP yields new state-of-the-art and >6% gains over RoBERTa baselines.
• Weakly supervised WikiSQL: Grappa with MLM+SSP achieves 84.7% execution accuracy, a new state-of-the-art.
• Across tasks, the combination MLM+SSP typically outperforms MLM or SSP alone, highlighting the importance of balanced pre-training.