[Paper Review] Weakly-supervised Neural Semantic Parsing with a Generative Ranker

Jianpeng Cheng, Mirella Lapata|arXiv (Cornell University)|Aug 23, 2018

Natural Language Processing Techniques48 references3 citations

TL;DR

This paper proposes a weakly-supervised neural semantic parser that generates candidate logical forms from utterances and ranks them using a log-linear model for execution correctness and a generative inverse parser for semantic faithfulness. The system uses scheduled training and neural lexicon encoding to reduce spurious forms, achieving state-of-the-art performance on three Freebase datasets.

ABSTRACT

Weakly-supervised semantic parsers are trained on utterance-denotation pairs, treating logical forms as latent. The task is challenging due to the large search space and spuriousness of logical forms. In this paper we introduce a neural parser-ranker system for weakly-supervised semantic parsing. The parser generates candidate tree-structured logical forms from utterances using clues of denotations. These candidates are then ranked based on two criterion: their likelihood of executing to the correct denotation, and their agreement with the utterance semantics. We present a scheduled training procedure to balance the contribution of the two objectives. Furthermore, we propose to use a neurally encoded lexicon to inject prior domain knowledge to the model. Experiments on three Freebase datasets demonstrate the effectiveness of our semantic parser, achieving results within the state-of-the-art range.

Motivation & Objective

• Address the challenge of training neural semantic parsers without gold-standard logical forms by leveraging only utterance-denotation pairs.
• Reduce the impact of spurious logical forms that accidentally produce correct denotations but do not reflect utterance meaning.
• Improve generalization and model reliability by integrating prior domain knowledge through neurally encoded lexicons.
• Balance two training objectives—execution correctness and semantic faithfulness—via a scheduled training procedure to stabilize learning.
• Develop a parser-ranker framework that combines neural sequence-to-tree generation with discriminative ranking for weakly-supervised semantic parsing.

Proposed method

• Use a sequence-to-tree neural parser with a bidirectional LSTM encoder and stack-LSTM decoder to generate well-formed, executable logical forms from utterances.
• Apply a grammar-constrained decoding process to ensure all generated logical forms are syntactically valid and executable against a knowledge base.
• Rank candidate logical forms using a log-linear model that scores their likelihood of producing the correct denotation through execution.
• Incorporate a generative inverse parser that estimates how well a logical form preserves the semantics of the input utterance, acting as a semantic faithfulness signal.
• Implement a scheduled training procedure that dynamically balances the contributions of the execution correctness objective and the semantic faithfulness objective during optimization.
• Neurally encode a lexicon as prior knowledge to regularize the latent space and improve generalization, especially in low-resource settings.

Experimental results

Research questions

• RQ1 Can a neural parser-ranker system effectively learn from utterance-denotation pairs without access to gold logical forms?
• RQ2 How can spurious logical forms—those that produce correct denotations but are semantically incorrect—be mitigated during weakly-supervised training?
• RQ3 To what extent does combining execution correctness and semantic faithfulness in the ranking objective improve parsing accuracy?
• RQ4 Does scheduled training that balances multiple objectives lead to more stable and effective model convergence?
• RQ5 Can neurally encoded lexicons improve model performance by injecting domain-specific prior knowledge into the latent space?

Key findings

• The proposed parser-ranker system achieves state-of-the-art performance on three Freebase-based semantic parsing benchmarks: WebQuestions, GraphQuestions, and Spades.
• The inclusion of the generative inverse parser significantly reduces the model's reliance on spurious logical forms, as evidenced by lower scores assigned to semantically incorrect but denotationally correct forms.
• Scheduled training improves model stability and performance by dynamically balancing the execution correctness and semantic faithfulness objectives during optimization.
• Neural lexicon encoding enhances model generalization and helps regularize the latent space, particularly in low-resource or out-of-vocabulary scenarios.
• The system demonstrates robustness across diverse domains and outperforms previous weakly-supervised approaches that rely solely on execution-based supervision.
• The ablation study confirms that both the inverse parser and scheduled training are critical for performance gains, with each component contributing significantly to the final accuracy.