[Paper Review] Natural Language Data Interfaces: A Data Access Odyssey (Invited Talk)

Koutrika, Georgia|arXiv (Cornell University)|Jul 9, 2018

Topic Modeling36 references68 citations

TL;DR

This paper introduces execution-guided decoding, a method that improves neural text-to-SQL generation by filtering out semantically invalid SQL queries during decoding using partial program execution. By conditioning autoregressive models on runtime execution results, it boosts execution accuracy by up to 5.4% on WikiSQL, achieving a new state-of-the-art result of 83.8%.

ABSTRACT

Back in 1970’s, E. F. Codd worked on a prototype of a natural language question and answer application that would sit on top of a relational database system. Soon, natural language interfaces for databases (NLIDBs) became the holy grail for the database community. Different approaches have been proposed from the database, machine learning and NLP communities. Interest in the topic has had its peaks and valleys. After a long and adventurous journey of almost 50 years, there is a rekindled interest in NLIDBs in recent years, fueled by the need for democratizing data access and by the recent advances in deep learning and natural language processing in particular. There is a surge of works on natural language interfaces for databases using neural translation, and suddenly it becomes hard to keep up with advancements in the field. Are we close to finding the holy grail of data access? What are the lurking challenges that we need to surpass and what research opportunities arise? Finally, what is the role of the database community?

Motivation & Objective

• To address the challenge of generating syntactically and semantically valid SQL queries from natural language questions.
• To reduce runtime errors and empty result sets in neural text-to-SQL generation by leveraging partial program execution during decoding.
• To develop a general-purpose inference-time technique that enhances diverse state-of-the-art models without requiring retraining.
• To improve execution accuracy on benchmark datasets like WikiSQL, ATIS, and GeoQuery by filtering incorrect partial programs early in decoding.

Proposed method

• Integrates a SQL execution engine into the beam search decoding process to evaluate partially generated queries at intermediate timesteps.
• Filters out beam candidates that produce parsing errors, runtime type mismatches, or empty result sets during decoding.
• Applies execution guidance as a non-differentiable conditioning signal on the decoder, enabling filtering without modifying the model's training objective.
• Extends four state-of-the-art models—Pointer-SQL, Seq2Seq with attention, a template-based model, and Coarse2Fine—by integrating execution-guided decoding.
• Uses a beam search algorithm where only executable partial programs are retained at each decoding step, based on execution outcomes.
• Employs a fixed beam size (k=5) and applies execution checks at specific, strategically chosen decoding steps to balance efficiency and effectiveness.

Experimental results

Research questions

• RQ1 Can partial program execution during decoding effectively eliminate semantically invalid SQL queries in neural text-to-SQL generation?
• RQ2 Does execution-guided decoding universally improve performance across diverse text-to-SQL models and datasets with varying complexity?
• RQ3 How does execution guidance impact the generation of specific SQL components, such as conditions and aggregation functions?
• RQ4 Can execution guidance be applied as a plug-in inference-time module to existing models without retraining or architectural changes?

Key findings

• Execution-guided decoding improves execution accuracy by 1% to 6% across all evaluated models and datasets, demonstrating consistent gains.
• The Coarse2Fine model with execution guidance achieves a new state-of-the-art execution accuracy of 83.8% on the WikiSQL dataset, up from 78.4%.
• Execution guidance has a particularly strong impact on condition generation, significantly reducing incorrect or overly restrictive WHERE clauses.
• Filtering based on execution results—especially for runtime errors and empty outputs—proves more effective than relying solely on model likelihood during decoding.
• The method is universally applicable and effective across both sequence-to-sequence and template-based models, showing broad compatibility.
• Ablation studies confirm that fine-grained execution checks at intermediate decoding steps are crucial for eliminating incorrect candidates early.