[Paper Review] AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data

Silei Xu, Sina J. Semnani+2 more|arXiv (Cornell University)|Oct 9, 2020

Natural Language Processing Techniques42 references4 citations

TL;DR

AutoQA is a fully automated toolkit that generates semantic parsers for question-answering over databases using only synthetic training data, eliminating the need for manual annotation or human paraphrasing. It leverages automatic paraphrasing and a filtered auto-paraphraser to generate diverse, high-quality training examples, achieving 62.9% logical form accuracy on Schema2QA and 69.8% answer accuracy on Overnight—within 6.4% and 5.2% of models trained with expert-annotated data, respectively.

ABSTRACT

We propose AutoQA, a methodology and toolkit to generate semantic parsers that answer questions on databases, with no manual effort. Given a database schema and its data, AutoQA automatically generates a large set of high-quality questions for training that covers different database operations. It uses automatic paraphrasing combined with template-based parsing to find alternative expressions of an attribute in different parts of speech. It also uses a novel filtered auto-paraphraser to generate correct paraphrases of entire sentences. We apply AutoQA to the Schema2QA dataset and obtain an average logical form accuracy of 62.9% when tested on natural questions, which is only 6.4% lower than a model trained with expert natural language annotations and paraphrase data collected from crowdworkers. To demonstrate the generality of AutoQA, we also apply it to the Overnight dataset. AutoQA achieves 69.8% answer accuracy, 16.4% higher than the state-of-the-art zero-shot models and only 5.2% lower than the same model trained with human data.

Motivation & Objective

• To eliminate the high cost of manual data annotation in training semantic parsers for question-answering over databases.
• To develop a fully automated pipeline that generates high-quality training data without requiring expert-annotated logical forms or crowdworker-generated paraphrases.
• To evaluate whether synthetic data augmented with automatic paraphrasing can match the performance of models trained with human-annotated data.
• To demonstrate generalization across diverse databases and logical forms, including knowledge base and web schema datasets.

Proposed method

• AutoQA uses pre-trained paraphrasing models to automatically annotate database attributes with alternative phrasings across different parts of speech.
• It applies template-based data synthesis to generate a large, diverse set of complex questions from the annotated attributes.
• A novel filtered auto-paraphraser based on BART generates natural sentence paraphrases, with a filter ensuring semantic fidelity by validating that paraphrased sentences map to the same logical form as the original.
• The filtered paraphrases are iteratively used to retrain the semantic parser via self-training, improving robustness to linguistic variation.
• The final parser is a BERT-LSTM model trained exclusively on the synthetic, auto-annotated, and auto-paraphrased data.
• The pipeline is evaluated on Schema2QA and the Overnight benchmark, using logical form and answer accuracy as metrics.

Experimental results

Research questions

• RQ1 Can a semantic parser be trained to high accuracy using only synthetic training data, without any human-annotated examples?
• RQ2 How effective is automatic paraphrasing—especially when filtered for semantic correctness—in improving generalization over unseen natural language questions?
• RQ3 To what extent can auto-annotation of database attributes using paraphrasing models replace manual curation of attribute synonyms?
• RQ4 How does the performance of a fully synthetic training pipeline compare to state-of-the-art models trained with human-annotated paraphrases?

Key findings

• On the Schema2QA dataset, AutoQA achieves 62.9% logical form accuracy on natural questions, only 6.4% lower than a model trained with expert-annotated data and human paraphrases.
• On the Overnight benchmark, AutoQA achieves 69.8% answer accuracy and 55.6% logical form accuracy, outperforming state-of-the-art zero-shot models by 16.4 and 18.6 percentage points, respectively.
• The full AutoQA pipeline, combining auto-annotation and filtered auto-paraphrasing, outperforms each component alone, with a 6.4% improvement over auto-paraphrasing-only and 8.3% over auto-annotation-only.
• Using an unfiltered auto-paraphraser leads to a 18% drop in accuracy, demonstrating the necessity of semantic filtering in synthetic data generation.
• The performance of AutoQA is within 5.2% of a model trained with in-domain human paraphrases on the Overnight dataset, indicating strong zero-shot generalization.
• The method generalizes across different database schemas and logical forms, showing effectiveness on both web-scale schema data and knowledge base datasets.