[Paper Review] A Survey on Semantic Parsing

Aishwarya Kamath, Rajarshi Das|arXiv (Cornell University)|Dec 3, 2018

Natural Language Processing TechniquesComputer Science83 references22 citations

TL;DR

This survey provides a comprehensive overview of semantic parsing, tracing its evolution from rule-based systems to modern neural approaches, with emphasis on learning paradigms, supervision levels, and structural constraints. It highlights key advancements in neural sequence-to-sequence models, weak supervision, and reinforcement learning-based training, achieving state-of-the-art results on benchmarks like WikiTableQuestions through methods such as memory-augmented exploration and margin-based optimization.

ABSTRACT

A significant amount of information in today's world is stored in structured and semi-structured knowledge bases. Efficient and simple methods to query them are essential and must not be restricted to only those who have expertise in formal query languages. The field of semantic parsing deals with converting natural language utterances to logical forms that can be easily executed on a knowledge base. In this survey, we examine the various components of a semantic parsing system and discuss prominent work ranging from the initial rule based methods to the current neural approaches to program synthesis. We also discuss methods that operate using varying levels of supervision and highlight the key challenges involved in the learning of such systems.

Motivation & Objective

• To provide a structured understanding of semantic parsing components, including meaning representation formalisms, grammars, and execution environments.
• To trace the historical development of semantic parsing from handcrafted rules to end-to-end neural models, emphasizing methodological shifts.
• To compare learning paradigms across supervision levels—fully supervised, weakly supervised, and reinforcement learning—highlighting trade-offs in data needs and performance.
• To identify open challenges such as confidence estimation, cross-domain generalization, and efficient integration of structural constraints.
• To propose future research directions, including multi-task learning, human-in-the-loop uncertainty handling, and leveraging logical forms for evaluation in NLU tasks.

Proposed method

• The survey employs a systematic categorization of semantic parsing systems based on supervision type: fully supervised, weakly supervised (denotation-level), and weakly supervised with reinforcement learning.
• It evaluates encoder-decoder neural architectures for sequence-to-sequence mapping of natural language to logical forms, particularly using attention mechanisms and beam search.
• The paper details exploration strategies in reinforcement learning, such as ε-greedy beam search and memory-augmented trajectory sampling, to improve sample efficiency and reduce variance.
• It introduces maximum margin reward (MMR) training, where models are optimized by maximizing the margin between correct and violating logical forms using a structured loss function.
• The framework incorporates a deterministic executor (e.g., SQL interpreter) to validate logical forms and compute rewards during training.
• It compares three learning paradigms: supervised learning, reinforcement learning, and maximum margin learning, with a generalized update equation to unify their design.

Experimental results

Research questions

• RQ1 How have semantic parsing systems evolved from rule-based to neural end-to-end models, and what are the key drivers of this shift?
• RQ2 What are the trade-offs between different supervision levels—fully annotated, denotation-only, and reinforcement learning—across data efficiency and performance?
• RQ3 How can structural constraints and formal grammars be effectively integrated into neural models to ensure syntactic and semantic correctness?
• RQ4 What role does exploration play in reinforcement learning-based semantic parsing, and how can it be improved using memory buffers and rejection sampling?
• RQ5 How can semantic parsers be made robust to uncertainty and generalize across low-resource domains?

Key findings

• Neural sequence-to-sequence models with attention mechanisms significantly outperform earlier statistical and rule-based systems in semantic parsing tasks.
• Reinforcement learning with memory-augmented exploration reduced variance in policy gradients and improved performance on WikiTableQuestions by effectively preserving high-reward trajectories.
• The ε-greedy beam search strategy achieved a balance between exploitation and exploration, improving sample efficiency in neural semantic parsing.
• Maximum margin reward (MMR) training, which optimizes for the most violating program, led to improved generalization and robustness in model updates.
• The generalized update equation for learning paradigms enables flexible selection of training strategies based on data availability and task requirements.
• The integration of deterministic executors allows for reliable reward shaping and validation of generated logical forms during training.