[Paper Review] Jointly Learning Semantic Parser and Natural Language Generator via Dual Information Maximization

Hai Ye, Wenjie Li+1 more|arXiv (Cornell University)|Jun 3, 2019

Topic Modeling48 references4 citations

TL;DR

This paper proposes Dual Information Maximization (DIM) to jointly train semantic parsers and natural language generators by exploiting their duality, using variational lower bounds to maximize mutual information between natural language and meaning representations. Experiments on dialogue management and code generation datasets show consistent performance gains in both tasks under both supervised and semi-supervised settings.

ABSTRACT

Semantic parsing aims to transform natural language (NL) utterances into formal meaning representations (MRs), whereas an NL generator achieves the reverse: producing a NL description for some given MRs. Despite this intrinsic connection, the two tasks are often studied separately in prior work. In this paper, we model the duality of these two tasks via a joint learning framework, and demonstrate its effectiveness of boosting the performance on both tasks. Concretely, we propose a novel method of dual information maximization (DIM) to regularize the learning process, where DIM empirically maximizes the variational lower bounds of expected joint distributions of NL and MRs. We further extend DIM to a semi-supervision setup (SemiDIM), which leverages unlabeled data of both tasks. Experiments on three datasets of dialogue management and code generation (and summarization) show that performance on both semantic parsing and NL generation can be consistently improved by DIM, in both supervised and semi-supervised setups.

Motivation & Objective

• Address the lack of joint modeling between semantic parsing and natural language generation despite their inherent duality.
• Improve performance on both tasks by leveraging their complementary nature through a unified learning framework.
• Develop a regularization method that captures dual information between natural language and meaning representations.
• Extend the supervised framework to a semi-supervised setup using unlabeled data from both tasks.
• Demonstrate the effectiveness of joint learning on real-world datasets in dialogue management and code generation.

Proposed method

• Model the duality of semantic parsing and NL generation using joint distributions of natural language (x) and meaning representations (y), approximating the unknown true distribution P(x,y).
• Propose Dual Information Maximization (DIM) to optimize variational lower bounds of the expected joint distributions from both the parser pθ(y|x) and generator qϕ(x|y).
• Use variational approximation inspired by Barber and Agakov (2003) and Zhang et al. (2018) to maximize dual information between the two models.
• Integrate DIM as a regularization term alongside supervised objectives for both semantic parsing and NL generation.
• Extend DIM to SemiDIM by incorporating unsupervised learning objectives on unlabeled data from both tasks.
• Optimize the joint objective using stochastic gradient descent with REINFORCE-based policy gradients for the unsupervised component.

Experimental results

Research questions

• RQ1 Can joint learning of semantic parsing and natural language generation improve performance on both tasks compared to training them independently?
• RQ2 How effective is dual information maximization (DIM) in capturing the structural duality between parsing and generation?
• RQ3 Does the inclusion of unlabeled data through SemiDIM further improve performance in low-resource settings?
• RQ4 What is the optimal trade-off between supervised and unsupervised learning signals in the joint training framework?
• RQ5 To what extent are the performance gains of the parser and generator correlated across different hyperparameter settings?

Key findings

• DIM consistently improves performance on both semantic parsing and natural language generation across three datasets: ATIS, Django, and CoNaLa.
• Semi-supervised learning via SemiDIM achieves significant performance gains, especially in low-resource settings, by leveraging unlabeled data from both tasks.
• Optimal performance is achieved when the hyperparameter λ controlling the trade-off between dual information and unsupervised loss is in the range 0.1–1.0.
• A drop in performance is observed when λ = 0, indicating that unsupervised signals are essential for robust joint learning.
• A high Pearson correlation (r > 0.8) is observed between parser and generator performance, suggesting strong co-adaptation in the joint framework.
• The joint model outperforms competitive baselines trained independently on both parsing and generation tasks, demonstrating the benefit of dual learning.