[Paper Review] On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries

Tianze Shi, Chen Zhao+3 more|arXiv (Cornell University)|Oct 21, 2020

Natural Language Processing Techniques5 references4 citations

TL;DR

This paper introduces Squall, a large-scale semantic parsing dataset with manual lexical-to-SQL alignments for 11,276 WikiTableQuestions. By incorporating supervised attention and column reference prediction into encoder-decoder models, the authors achieve a 4.4% absolute improvement in execution accuracy over strong baselines, demonstrating that fine-grained lexical supervision significantly enhances semantic parsing performance.

ABSTRACT

Large-scale semantic parsing datasets annotated with logical forms have enabled major advances in supervised approaches. But can richer supervision help even more? To explore the utility of fine-grained, lexical-level supervision, we introduce Squall, a dataset that enriches 11,276 WikiTableQuestions English-language questions with manually created SQL equivalents plus alignments between SQL and question fragments. Our annotation enables new training possibilities for encoder-decoder models, including approaches from machine translation previously precluded by the absence of alignments. We propose and test two methods: (1) supervised attention; (2) adopting an auxiliary objective of disambiguating references in the input queries to table columns. In 5-fold cross validation, these strategies improve over strong baselines by 4.4% execution accuracy. Oracle experiments suggest that annotated alignments can support further accuracy gains of up to 23.9%.

Motivation & Objective

• To investigate whether fine-grained lexical supervision, in the form of question-to-SQL fragment alignments, can improve semantic parsing performance.
• To create a large-scale, high-quality dataset—Squall—that provides both SQL logical forms and manual lexical-level alignments between natural language questions and their corresponding SQL components.
• To evaluate the effectiveness of two novel training strategies—supervised attention and column prediction—using the annotated alignments.
• To assess the potential of lexical alignments to enable further accuracy gains beyond current state-of-the-art models.
• To analyze the annotation cost and learning curve of lexical alignments to determine their cost-effectiveness in semantic parsing.

Proposed method

• Annotate 11,276 WikiTableQuestions with SQL equivalents and manual alignments between question tokens and SQL fragments, creating the Squall dataset.
• Train sequence-to-sequence models with supervised attention, where attention weights are guided by human-annotated alignments during both encoding and decoding.
• Introduce a column prediction head that predicts which table column a question fragment refers to, using BERT-based features.
• Compare models trained with and without lexical supervision using 5-fold cross-validation on the Squall dev set.
• Evaluate model performance using both logical form accuracy (exact match) and execution accuracy (correct answer regardless of form).
• Conduct oracle experiments where models are given access to full alignments at test time to estimate the upper bound of potential gains.

Experimental results

Research questions

• RQ1 Can manual lexical-to-logical form alignments significantly improve semantic parsing accuracy beyond standard supervision with logical forms alone?
• RQ2 How effective are supervised attention and column reference prediction as training strategies when fine-grained alignment supervision is available?
• RQ3 What is the upper bound of performance improvement achievable with full access to lexical alignments during inference?
• RQ4 How cost-effective is annotating lexical alignments compared to full logical form annotation?
• RQ5 Does the addition of alignment supervision lead to more robust generalization, especially in cases of lexical variation or paraphrasing?

Key findings

• In 5-fold cross-validation, the proposed method with both supervised attention and column prediction improved execution accuracy by 4.4% over a strong baseline that only uses logical forms.
• Supervised encoder attention was the most effective individual component, contributing a 3.8% absolute improvement in execution accuracy over the baseline.
• Column prediction contributed a 1.3% absolute improvement in execution accuracy, indicating that reference disambiguation is a valuable signal.
• Oracle models that use full alignment information at test time achieved up to 23.9% absolute improvement in logical form accuracy, revealing substantial unrealized potential.
• The annotation of lexical alignments took less than half the time per example compared to full logical form annotation, indicating strong cost-effectiveness.
• Training with just 5% of aligned examples yielded a 2.5 percentage point improvement in execution accuracy, suggesting that partial alignment annotation is highly effective.