[Paper Review] Content Enhanced BERT-based Text-to-SQL Generation

Tong Guo, Huilin Gao|arXiv (Cornell University)|Oct 16, 2019

Topic Modeling5 references46 citations

TL;DR

This paper adds two external feature vectors derived from table content and headers to a BERT-based NL2SQL model, improving WikiSQL performance and achieving state-of-the-art results.

ABSTRACT

We present a simple methods to leverage the table content for the BERT-based model to solve the text-to-SQL problem. Based on the observation that some of the table content match some words in question string and some of the table header also match some words in question string, we encode two addition feature vector for the deep model. Our methods also benefit the model inference in testing time as the tables are almost the same in training and testing time. We test our model on the WikiSQL dataset and outperform the BERT-based baseline by 3.7% in logic form and 3.7% in execution accuracy and achieve state-of-the-art.

Motivation & Objective

• Motivate leveraging table content as external knowledge for NL2SQL models.
• Propose two feature vectors aligning question with table cells and headers.
• Integrate these features into a BERT-based architecture for SELECT, AGG, and WHERE components.

Proposed method

• Construct question-content and header-content match vectors (QV and HV) via word matching between question and table elements.
• Concatenate question and table header inputs for BERT to obtain contextual representations Q and H.
• Predict SELECT column, SELECT agg, WHERE number, WHERE column, WHERE op, and WHERE value using Q, H, QV, HV with respective probabilities.
• Train end-to-end on WikiSQL with ablation to assess vector contributions.
• Optionally apply execution-guided decoding to further boost results.

Experimental results

Research questions

• RQ1 Can external features derived from table contents improve NL2SQL performance over standard BERT baselines?
• RQ2 How do table cell/header matches influence NU2SQL component predictions (SELECT, WHERE, AGG)?
• RQ3 Does incorporating QV and HV at inference time provide benefits when training and testing tables are similar?

Key findings

• Outperforms BERT-based baseline by 3.7 percentage points in both logic form and execution accuracy on WikiSQL.
• Achieves state-of-the-art results on the WikiSQL task with the proposed external features.
• Ablation shows header vector mainly improves WHERE OP while question vector mainly improves WHERE VALUE.
• Adding external feature vectors yields notable gains across SELECT column, WHERE number, WHERE column, WHERE value tasks.
• Execution-guided decoding provides incremental gains when used with the external features.