[Paper Review] SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language

Ning Li, Bethany Keller+2 more|arXiv (Cornell University)|Nov 7, 2020

Natural Language Processing Techniques12 references4 citations

TL;DR

This paper proposes SeqGenSQL, a robust sequence-to-sequence model based on T5 fine-tuned for direct natural language to SQL generation, incorporating pointer-generator decoding and table schema-augmented question data. It achieves 90.5% execution accuracy on WikiSQL, setting a new state-of-the-art in weakly supervised text-to-SQL with a 6.6% absolute improvement over prior work.

ABSTRACT

We explore using T5 (Raffel et al. (2019)) to directly translate natural language questions into SQL statements. General purpose natural language that interfaces to information stored within databases requires flexibly translating natural language questions into database queries. The best performing text-to-SQL systems approach this task by first converting questions into an intermediate logical form (LF) (Lyu et al. (2020)). While LFs provide a convenient intermediate representation and simplify query generation, they introduce an additional layer of complexity and annotation requirements. However, weakly supervised modeling that directly converts questions to SQL statements has proven more difficult without the scaffolding provided by LFs (Min et al. (2019)). We approach direct conversion of questions to SQL statements using T5 (Raffel et al. (2019)), a pre-trained textto-text generation model, modified to support pointer-generator style decoding (See et al. (2017)). We explore using question augmentation with table schema information and the use of automatically generated silver training data. The resulting model achieves 90.5% execution accuracy on the WikiSQL (Zhong et al. (2017)) test data set, a new state-of-the-art on weakly supervised SQL generation. The performance improvement is 6.6% absolute over the prior state-of-the-art (Min et al. (2019)) and approaches the performance of state-ofthe-art systems making use of LFs.

Motivation & Objective

• To develop a direct, weakly supervised approach to text-to-SQL generation without relying on intermediate logical forms.
• To improve execution accuracy in text-to-SQL translation using pre-trained sequence-to-sequence models.
• To reduce dependency on expensive logical form annotations by leveraging schema-augmented question data and silver training data.
• To achieve state-of-the-art performance in weakly supervised text-to-SQL generation.
• To demonstrate that direct generation can approach the performance of systems using intermediate logical forms.

Proposed method

• Fine-tuning the T5 model for end-to-end text-to-SQL generation using natural language questions and corresponding SQL queries.
• Modifying T5 with pointer-generator style decoding to better handle OOV (out-of-vocabulary) tokens in SQL.
• Augmenting training questions with relevant table schema information to improve context awareness.
• Generating silver training data via model distillation to increase training data scale and diversity.
• Using a sequence-to-sequence architecture with shared attention mechanisms for robust generation.
• Applying data augmentation and schema injection to improve generalization and reduce annotation dependency.

Experimental results

Research questions

• RQ1 Can a weakly supervised, end-to-end sequence-to-sequence model achieve high execution accuracy in text-to-SQL generation without intermediate logical forms?
• RQ2 How does incorporating table schema information into question encoding affect model performance?
• RQ3 To what extent can silver training data improve model generalization and execution accuracy?
• RQ4 Can pointer-generator decoding mitigate OOV issues in SQL generation compared to standard autoregressive decoding?
• RQ5 How does direct generation compare to state-of-the-art systems that use intermediate logical forms in terms of execution accuracy?

Key findings

• SeqGenSQL achieves 90.5% execution accuracy on the WikiSQL test set, setting a new state-of-the-art for weakly supervised text-to-SQL models.
• The model improves execution accuracy by 6.6 percentage points over the prior state-of-the-art (Min et al., 2019), which used a similar weakly supervised setup.
• The use of schema-augmented questions and silver training data significantly boosts model generalization and robustness.
• Pointer-generator decoding effectively reduces OOV token errors in SQL generation, improving output fidelity.
• The performance of SeqGenSQL is within reach of state-of-the-art systems that use intermediate logical forms, demonstrating the viability of direct generation.
• The model shows strong generalization on unseen database schemas, indicating robustness to distributional shifts.