[Paper Review] RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Bailin Wang, Richard Shin+3 more|arXiv (Cornell University)|Nov 10, 2019

Natural Language Processing Techniques72 citations

TL;DR

RAT-SQL introduces relation-aware self-attention to jointly encode and link database schemas with questions for Text-to-SQL parsing, achieving state-of-the-art results on Spider (57.2% exact match) and 65.6% with BERT.

ABSTRACT

When translating natural language questions into SQL queries to answer questions from a database, contemporary semantic parsing models struggle to generalize to unseen database schemas. The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query. We present a unified framework, based on the relation-aware self-attention mechanism, to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder. On the challenging Spider dataset this framework boosts the exact match accuracy to 57.2%, surpassing its best counterparts by 8.7% absolute improvement. Further augmented with BERT, it achieves the new state-of-the-art performance of 65.6% on the Spider leaderboard. In addition, we observe qualitative improvements in the model's understanding of schema linking and alignment. Our implementation will be open-sourced at https://github.com/Microsoft/rat-sql.

Motivation & Objective

• Motivate robust generalization to unseen database schemas in text-to-SQL parsing.
• Develop a unified encoding framework that integrates schema relations and question context.
• Enable effective schema linking by modeling both predefined schema relations and question–schema interactions.
• Demonstrate improvements on the Spider dataset and analyze ablations to identify key factors.

Proposed method

• Propose relation-aware self-attention to jointly encode schema, tables, and question words within a single input graph.
• Represent the database schema as a directed graph with labeled edges capturing foreign-key and table/column relations.
• Augment Transformer attention with predefined relation features r_{ij}^K and r_{ij}^V to bias attention toward schema relations.
• Introduce name-based and value-based schema linking to align question tokens with schema columns/tables.
• Compute memory-alignment matrices L_col and L_tab to explicitly capture question-to-schema alignments for decoder guidance.
• Use a tree-structured decoder to generate SQL by expanding grammar rules and selecting columns/tables based on context.],
• research_questions:[

Experimental results

Research questions

• RQ1 How can relation-aware attention improve encoding of relational schema information for unseen databases?
• RQ2 Can explicit schema linking (name-based and value-based) improve alignment between questions and schema elements?
• RQ3 What is the impact of combining predefined schema relations with soft, learned relationships on parsing accuracy?
• RQ4 How does RAT-SQL perform on Spider and WikiSQL, and how does BERT augmentation influence results?

Key findings

• RAT-SQL achieves 57.2% exact-match on Spider test set, outperforming non-BERT baselines by 8.7% absolute.
• RAT-SQL with BERT reaches 65.6% exact-match on Spider test, setting a new state-of-the-art at the time for BERT-augmented models.
• On development data, RAT-SQL achieves 62.7% (dev) and 57.2% (test); with BERT, 69.7% (dev) and 65.6% (test).
• Ablations show that removing schema linking or graph relations significantly reduces accuracy (e.g., w/o schema linking 40.37% and w/o graph relations 35.59% on dev).
• Oracle experiments indicate that if the correct columns/tables or AST sketch are provided, accuracy can reach near-perfect levels (99.4% with both oracle sketch and oracle columns).
• Value-based linking substantially improves performance (60.54% dev with value-based linking vs 55.13% without).