[Paper Review] Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions

Rui Zhang, Yu Tao+8 more|arXiv (Cornell University)|Sep 2, 2019

Topic Modeling53 references49 citations

TL;DR

The paper introduces an editing-based encoder-decoder model that generates SQL by editing the previously predicted query, augmented with an utterance-table encoder and a table-aware decoder to handle cross-domain context-dependent text-to-SQL. It achieves improvements over state-of-the-art baselines on SParC and Spider datasets, especially with query editing and BERT-based utterance-table representations.

ABSTRACT

We focus on the cross-domain context-dependent text-to-SQL generation task. Based on the observation that adjacent natural language questions are often linguistically dependent and their corresponding SQL queries tend to overlap, we utilize the interaction history by editing the previous predicted query to improve the generation quality. Our editing mechanism views SQL as sequences and reuses generation results at the token level in a simple manner. It is flexible to change individual tokens and robust to error propagation. Furthermore, to deal with complex table structures in different domains, we employ an utterance-table encoder and a table-aware decoder to incorporate the context of the user utterance and the table schema. We evaluate our approach on the SParC dataset and demonstrate the benefit of editing compared with the state-of-the-art baselines which generate SQL from scratch. Our code is available at https://github.com/ryanzhumich/sparc_atis_pytorch.

Motivation & Objective

• Motivate cross-domain, context-dependent text-to-SQL generation where history matters across turns.
• Leverage editing of the previously generated SQL query to reuse generation results and reduce error propagation.
• Incorporate complex table schemas with an utterance-table encoder and a table-aware decoder.
• Evaluate on SParC (cross-domain, context-dependent) and Spider (context-independent cross-domain) datasets.
• Demonstrate robustness of editing over segment copying and quantify gains from BERT-based encodings.

Proposed method

• Encode the user utterance and table schema with a turn-aware utterance-table encoder that uses bi-LSTMs and co-attention between utterances and column headers; optionally use BERT-based embeddings for Utterance-Table representations.
• Maintain an interaction-level history encoder with turn attention to capture dependencies across turns.
• Decode SQL with a table-aware decoder that attends to column headers and utterance tokens, and outputs a distribution over SQL keywords or column headers; adapt outputs via a two-way scoring mechanism for SQL keywords vs. column headers.
• Extend the decoder with a query editing mechanism that conditions on the previous query and learns a switch p_copy to copy from the previous query or insert new tokens (P(y_k)) using a combination of previous-query and current-output distributions.
• Incorporate a query-attention component that attends to tokens of the previous query and previous turns to further inform editing decisions.

Experimental results

Research questions

• RQ1 Can editing the previously generated SQL query improve cross-domain, context-dependent text-to-SQL generation compared to generating from scratch?
• RQ2 Does an utterance-table encoder with co-attention and a table-aware decoder better handle diverse schemas across domains?
• RQ3 How does the query editing mechanism affect robustness to error propagation in multi-turn interactions?
• RQ4 What is the impact of using BERT-based utterance-table embeddings on cross-domain text-to-SQL performance?
• RQ5 How do the proposed components perform on SParC and Spider datasets, in terms of question match and interaction match metrics?

Key findings

• Editing-based generation yields improvements over state-of-the-art baselines that generate SQL from scratch on SParC, with notable gains in interaction-related metrics.
• Using utterance-table embeddings (especially with BERT) substantially improves cross-domain performance, achieving strong results on Spider and substantial gains on SParC.
• The query editing mechanism is more robust to error propagation than full segment copying, particularly when combined with BERT-based encodings.
• Turn attention and table-utterance co-attention help the model handle cross-domain schemas and context dependencies across turns.
• On SParC, the model outperforms baselines in question match accuracy and interaction match accuracy when employing editing + query attention with predicted history; when using gold history, editing yields even larger gains.
• In Spider (context-independent), the utterance-table BERT embedding boosts dev and test scores to levels comparable with state-of-the-art cross-domain models.