[Paper Review] Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation

Jiaqi Guo, Zecheng Zhan+5 more|arXiv (Cornell University)|May 20, 2019

Topic Modeling54 references20 citations

TL;DR

This paper proposes IRNet, a neural Text-to-SQL framework that decomposes complex, cross-domain query generation into three stages: schema linking, SemQL intermediate representation (IR) synthesis via a grammar-based neural model, and deterministic SQL generation using domain knowledge. On the Spider benchmark, IRNet achieves 46.7% exact match accuracy, a 19.5% absolute improvement over prior state-of-the-art methods, and ranks first on the Spider leaderboard.

ABSTRACT

We present a neural approach called IRNet for complex and cross-domain Text-to-SQL. IRNet aims to address two challenges: 1) the mismatch between intents expressed in natural language (NL) and the implementation details in SQL; 2) the challenge in predicting columns caused by the large number of out-of-domain words. Instead of end-to-end synthesizing a SQL query, IRNet decomposes the synthesis process into three phases. In the first phase, IRNet performs a schema linking over a question and a database schema. Then, IRNet adopts a grammar-based neural model to synthesize a SemQL query which is an intermediate representation that we design to bridge NL and SQL. Finally, IRNet deterministically infers a SQL query from the synthesized SemQL query with domain knowledge. On the challenging Text-to-SQL benchmark Spider, IRNet achieves 46.7% accuracy, obtaining 19.5% absolute improvement over previous state-of-the-art approaches. At the time of writing, IRNet achieves the first position on the Spider leaderboard.

Motivation & Objective

• To address the mismatch between natural language intent and SQL implementation details in complex queries.
• To mitigate the lexical challenge posed by out-of-domain (OOD) words in cross-domain Text-to-SQL.
• To improve generalization and accuracy in complex, cross-domain settings by introducing an intermediate representation (SemQL) between natural language and SQL.
• To enable robust column prediction despite limited or no exposure to OOD schema terms during training.
• To demonstrate that learning SemQL queries improves downstream performance across multiple neural Text-to-SQL models.

Proposed method

• IRNet performs schema linking to identify and type columns and tables mentioned in the question, enhancing representation learning in the presence of OOD terms.
• It employs a grammar-based neural model to generate a SemQL query, an intermediate representation designed to bridge natural language semantics and SQL syntax.
• SemQL is a simplified, structured representation that supports common SQL constructs like selection, aggregation, and selection conditions, but excludes complex features like self-joins.
• The SemQL query is deterministically transformed into a final SQL query using a rule-based system enriched with domain-specific knowledge.
• The framework is trained end-to-end on the Spider benchmark, with schema linking and SemQL generation jointly optimized.
• When augmented with BERT, IRNet achieves 54.7% exact match accuracy, demonstrating the effectiveness of contextual embeddings.

Experimental results

Research questions

• RQ1 Can an intermediate representation like SemQL effectively bridge the semantic gap between natural language questions and complex SQL queries in cross-domain settings?
• RQ2 How does schema linking improve generalization when encountering out-of-domain words in database schemas?
• RQ3 Does decomposing Text-to-SQL into multiple stages—schema linking, IR generation, and SQL inference—lead to better performance than end-to-end approaches?
• RQ4 To what extent does learning SemQL queries improve the performance of other neural Text-to-SQL models such as SQLNet, TypeSQL, and SyntaxSQLNet?
• RQ5 Can a rule-based SQL generator produce accurate and valid queries from a SemQL representation without requiring differentiable training?

Key findings

• IRNet achieves 46.7% exact match accuracy on the Spider benchmark, representing a 19.5% absolute improvement over the previous state-of-the-art method.
• The model ranks first on the official Spider leaderboard at the time of submission, demonstrating strong generalization on complex, cross-domain queries.
• When enhanced with BERT, IRNet's accuracy increases to 54.7%, highlighting the benefit of contextualized embeddings in the framework.
• The SemQL intermediate representation significantly improves performance across multiple baseline models, including SQLNet, TypeSQL, and SyntaxSQLNet, when used as a shared intermediate output.
• The framework effectively mitigates the impact of out-of-domain words through schema linking, which enhances representation learning even when OOD terms lack strong embeddings.
• The ablation study confirms that schema linking and the use of SemQL as an intermediate step are both critical for the model’s success, especially on hard and extra-hard query types.