[Paper Review] A Transfer-Learnable Natural Language Interface for Databases

Wenlu Wang, Yingtao Tian+3 more|arXiv (Cornell University)|Sep 7, 2018

Natural Language Processing Techniques23 references18 citations

TL;DR

This paper proposes a transfer-learnable natural language interface for databases that decouples schema and data from natural language semantics through an automatic annotation mechanism, enabling a single model to generalize across diverse databases. The approach achieves 82% execution accuracy on WikiSQL and zero-shot transfer performance on OVERNIGHT without retraining, demonstrating strong cross-domain generalization via a customized sequence-to-sequence model with column stacks and copy mechanisms.

ABSTRACT

Relational database management systems (RDBMSs) are powerful because they are able to optimize and answer queries against any relational database. A natural language interface (NLI) for a database, on the other hand, is tailored to support that specific database. In this work, we introduce a general purpose transfer-learnable NLI with the goal of learning one model that can be used as NLI for any relational database. We adopt the data management principle of separating data and its schema, but with the additional support for the idiosyncrasy and complexity of natural languages. Specifically, we introduce an automatic annotation mechanism that separates the schema and the data, where the schema also covers knowledge about natural language. Furthermore, we propose a customized sequence model that translates annotated natural language queries to SQL statements. We show in experiments that our approach outperforms previous NLI methods on the WikiSQL dataset and the model we learned can be applied to another benchmark dataset OVERNIGHT without retraining.

Motivation & Objective

• To address the lack of generalization in existing natural language interfaces for databases (NLIDB), which are typically trained and fine-tuned per database.
• To decouple the semantic structure of relational queries from the idiosyncrasies of natural language and database-specific schemas.
• To develop a unified, transfer-learnable model that can translate natural language questions into executable SQL queries across diverse databases without retraining.
• To enable zero-shot transfer learning from one database domain (e.g., WikiSQL) to another (e.g., OVERNIGHT) by isolating latent semantic structures in queries.
• To improve robustness and generalization by encoding column semantics and value mentions independently through a novel annotation scheme.

Proposed method

• Annotate natural language questions by replacing column mentions with placeholders $c_i$ and values with $v_i$, preserving semantic roles while abstracting from lexical variation.
• Use a customized sequence-to-sequence model with a GRU encoder, copy mechanism, and column stack to map annotated questions to annotated SQL.
• Introduce a column stack mechanism that encodes column names and their semantic roles to improve alignment between natural language and SQL structure.
• Separate the annotation process from sequence modeling, enabling the model to be trained on one database and applied to others without retraining.
• Apply a deterministic mapping to convert annotated SQL ($s_a$) back to executable SQL ($s$), ensuring correctness and reproducibility.
• Use pre-trained GloVe embeddings and evaluate the model with both GRU-based and transformer-based architectures, finding GRU-based models outperform transformers in this task.

Experimental results

Research questions

• RQ1 Can a single NLI model be trained to generalize across multiple, diverse relational databases without retraining?
• RQ2 How does separating schema and natural language idiosyncrasies affect model performance and transferability?
• RQ3 What components of the model architecture—such as the copy mechanism, column stack, or multi-layer encoding—most significantly contribute to zero-shot transfer performance?
• RQ4 Does encoding table headers improve or hinder transferability in cross-domain settings?
• RQ5 How does the model’s performance on a source domain (WikiSQL) correlate with its ability to generalize to unseen domains (OVERNIGHT)?

Key findings

• The model achieves 82% query execution accuracy on the WikiSQL benchmark, outperforming previous state-of-the-art methods.
• The model demonstrates strong zero-shot transfer performance on the OVERNIGHT dataset, with a transfer accuracy of 78.5% across five sub-domains after training only on WikiSQL.
• Ablation studies show that removing the column stack reduces overall transfer accuracy by 22%, highlighting its critical role in cross-domain generalization.
• The copy mechanism improves performance on WikiSQL but slightly reduces transfer accuracy on OVERNIGHT, suggesting a trade-off between in-domain accuracy and generalization.
• Using a single-layer GRU instead of a deeper architecture reduces transfer accuracy by 1%, indicating that deeper modeling is essential for capturing complex dependencies across domains.
• Encoding table headers hurts transfer performance on OVERNIGHT, likely due to redundant or mismatched information in the target domain, despite improving performance on the source domain.