[Paper Review] SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning

Xiaojun Xu, Chang Liu+1 more|arXiv (Cornell University)|Nov 13, 2017

Topic Modeling35 references303 citations

TL;DR

SQLNet avoids the order-matters problem of seq2seq NL-to-SQL by using a sketch-based, sequence-to-set approach with column attention, achieving state-of-the-art results on WikiSQL without reinforcement learning.

ABSTRACT

Synthesizing SQL queries from natural language is a long-standing open problem and has been attracting considerable interest recently. Toward solving the problem, the de facto approach is to employ a sequence-to-sequence-style model. Such an approach will necessarily require the SQL queries to be serialized. Since the same SQL query may have multiple equivalent serializations, training a sequence-to-sequence-style model is sensitive to the choice from one of them. This phenomenon is documented as the "order-matters" problem. Existing state-of-the-art approaches rely on reinforcement learning to reward the decoder when it generates any of the equivalent serializations. However, we observe that the improvement from reinforcement learning is limited. In this paper, we propose a novel approach, i.e., SQLNet, to fundamentally solve this problem by avoiding the sequence-to-sequence structure when the order does not matter. In particular, we employ a sketch-based approach where the sketch contains a dependency graph so that one prediction can be done by taking into consideration only the previous predictions that it depends on. In addition, we propose a sequence-to-set model as well as the column attention mechanism to synthesize the query based on the sketch. By combining all these novel techniques, we show that SQLNet can outperform the prior art by 9% to 13% on the WikiSQL task.

Motivation & Objective

• Address the NL2SQL challenge where query order can be non-deterministic and hinder seq2seq models.
• Propose a sketch-based generation framework to avoid the order-matters issue.
• Introduce a sequence-to-set prediction and a column attention mechanism to improve SQL synthesis.
• Demonstrate improved performance over Seq2SQL on the WikiSQL dataset without RL.
• Provide insights into how sketch-based generation can generalize to unseen schemas.

Proposed method

• Define a SQL sketch that mirrors SQL grammar and dependencies among slots.
• Predict the WHERE clause as a set of column-value constraints (sequence-to-set) rather than an ordered sequence.
• Introduce a column attention mechanism to condition question embeddings on specific columns.
• Use a pointer-based decoder with column-aware attention to generate VALUE substrings.
• Predict SELECT clause components (column and aggregator) similarly to WHERE but for a single column.
• Train with a modified loss for set-based predictions and standard cross-entropy for other components.

Experimental results

Research questions

• RQ1 Can a sketch-based, sequence-to-set approach remove the need for reinforcement learning in NL2SQL tasks where clause ordering is arbitrary?
• RQ2 Does column attention improve the accuracy of predicting which columns appear in the WHERE and SELECT clauses?
• RQ3 How does SQLNet compare to Seq2SQL on WikiSQL in terms of logical-form, query-match, and execution accuracy?
• RQ4 What is the impact of training word embeddings during optimization on NL2SQL performance?
• RQ5 How does SQLNet perform when the test schema comes from unseen tables or is drawn from the training distribution?

Key findings

• SQLNet achieves state-of-the-art WikiSQL results with exact query-match and execution accuracy of 61.5% and 68.3% on the test set, respectively.
• Replacing the sequence-to-sequence generator with a sequence-to-set approach removes the order-matters limitation and eliminates the need for reinforcement learning.
• Column attention yields about a 3-point improvement over a plain sequence-to-set model, and combined with training-time embedding updates, adds roughly 2 more points.
• The WHERE clause prediction benefits the most from the proposed architecture, with substantial gains over Seq2SQL in that sub-task.
• Overall, SQLNet improves over Seq2SQL by approximately 9–13 points across multiple metrics on WikiSQL, establishing new baselines for NL2SQL without reinforcement learning.