[Paper Review] Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Victor W. Zhong, Caiming Xiong+1 more|arXiv (Cornell University)|Aug 31, 2017

Data Mining Algorithms and Applications40 references785 citations

TL;DR

Seq2SQL translates natural language questions into SQL queries by exploiting SQL structure and training a policy with in-the-loop execution rewards, achieving state-of-the-art results on WikiSQL.

ABSTRACT

A significant amount of the world's knowledge is stored in relational databases. However, the ability for users to retrieve facts from a database is limited due to a lack of understanding of query languages such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model leverages the structure of SQL queries to significantly reduce the output space of generated queries. Moreover, we use rewards from in-the-loop query execution over the database to learn a policy to generate unordered parts of the query, which we show are less suitable for optimization via cross entropy loss. In addition, we will publish WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia. This dataset is required to train our model and is an order of magnitude larger than comparable datasets. By applying policy-based reinforcement learning with a query execution environment to WikiSQL, our model Seq2SQL outperforms attentional sequence to sequence models, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

Motivation & Objective

• Motivate natural language interfaces to databases to lower the barrier to querying relational data.
• Propose a neural model that leverages SQL structure to prune the output space.
• Address unordered WHERE clause generation with reinforcement learning using live query execution rewards.
• Release WikiSQL, a large, real-world dataset of NL questions with SQL queries and table schemas.

Proposed method

• Use an augmented pointer network that generates SQL by selecting tokens from the input (columns, questions, and SQL vocabulary).
• Decompose SQL into three components: aggregation operator, SELECT column, and WHERE clause; supervise aggregation and SELECT with cross-entropy losses.
• Train the WHERE clause with policy gradient using rewards from in-the-loop query execution to handle unordered conditions.
• Leverage SQL structure to prune output space and improve generation of rare words/columns.
• Train with a mixed objective combining L_agg, L_sel (cross-entropy) and L_whe (reinforcement learning).
• Provide WikiSQL dataset (80,654 examples over 24,241 tables) and an execution engine for training and evaluation.

Experimental results

Research questions

• RQ1 Can a neural model generate correct SQL queries from natural language questions for unseen table schemas?
• RQ2 Does incorporating SQL structure and reinforcement learning improve execution accuracy over existing semantic parsers?
• RQ3 How does a large, real-world NLQ-to-SQL dataset (WikiSQL) compare to prior semantic parsing datasets in enabling cross-table generalization?

Key findings

• Seq2SQL outperforms a state-of-the-art semantic parser on WikiSQL in execution accuracy (from 35.9% to 59.4%).
• Seq2SQL also improves logical form accuracy (from 23.4% to 48.3%).
• Augmented pointer network (without RL) already surpasses baseline by 17.4% in execution accuracy.
• Incorporating SQL structure (three-part decomposition) adds a further 3.8% gain.
• Policy-gradient RL on the WHERE clause yields an additional 2.3% gain.
• On the COUNT operator, Seq2SQL achieves higher precision, recall, and F1 than the augmented pointer baseline (72.6, 66.2, 69.2 respectively).