[Paper Review] IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles

Tianze Shi, Kedar Tatwawadi+4 more|arXiv (Cornell University)|Sep 13, 2018

Natural Language Processing TechniquesComputer Science34 references59 citations

TL;DR

IncSQL trains an incremental text-to-SQL parser using non-deterministic oracles to handle multiple correct SQL forms; with execution-guided decoding it achieves state-of-the-art execution accuracy on WikiSQL.

ABSTRACT

We present a sequence-to-action parsing approach for the natural language to SQL task that incrementally fills the slots of a SQL query with feasible actions from a pre-defined inventory. To account for the fact that typically there are multiple correct SQL queries with the same or very similar semantics, we draw inspiration from syntactic parsing techniques and propose to train our sequence-to-action models with non-deterministic oracles. We evaluate our models on the WikiSQL dataset and achieve an execution accuracy of 83.7% on the test set, a 2.1% absolute improvement over the models trained with traditional static oracles assuming a single correct target SQL query. When further combined with the execution-guided decoding strategy, our model sets a new state-of-the-art performance at an execution accuracy of 87.1%.

Motivation & Objective

• Motivate NL2SQL as a practical database question interface and address semantic variation in correct SQL forms.
• Propose an incremental sequence-to-action parser that fills SQL slots via an action inventory.
• Introduce non-deterministic (dynamic) training oracles to handle multiple valid action sequences.
• Show that non-deterministic training improves execution accuracy on WikiSQL.
• Demonstrate the benefit of execution-guided decoding to reach state-of-the-art performance.

Proposed method

• Define a predefined inventory of actions that incrementally constructs SQL queries from an initial state to a terminal state.
• Use an LSTM-based decoder with bilinear scoring to select the next action from the feasible set.
• Employ context-sensitive representations of words and column headers via a bi-LSTM encoder with intra- and cross-sequence attention.
• Train with non-deterministic oracles that allow multiple correct action continuations, improving robustness to different valid SQL forms.
• Optionally apply execution-guided decoding to prune and select high-likelihood partial SQLs during beam search.
• Evaluate on WikiSQL with and without execution-guided decoding to measure execution accuracy and logical form accuracy.

Experimental results

Research questions

• RQ1 Can non-deterministic (dynamic) training oracles improve learning for incremental NL2SQL parsers compared to static oracles?
• RQ2 How does execution-guided decoding interact with non-deterministic training to boost execution accuracy?
• RQ3 To what extent can an incremental, action-slot filling parser handle ordering variability in WHERE clauses and implicit column mentions?
• RQ4 Is the approach generalizable to other NL2SQL datasets beyond WikiSQL (e.g., ATIS)?

Key findings

• Non-deterministic oracles improve execution accuracy by 2.1 percentage points over static oracles on WikiSQL.
• Execution-guided decoding with beam size 5 yields a new state-of-the-art test execution accuracy of 87.1%.
• Static oracle performance is competitive with prior SOTA models; non-deterministic training mainly boosts execution metrics.
• Allowing ANYCOL in the oracle helps capture implicit column references and boosts execution accuracy from 81.8% to 83.7%.
• On ATIS, non-deterministic training provides gains in execution accuracy, indicating generalization to other NL2SQL datasets.