[Paper Review] Zero-shot Transfer Learning for Semantic Parsing

Javid Dadashkarimi, Alexander R. Fabbri+2 more|arXiv (Cornell University)|Aug 27, 2018

Topic Modeling23 references3 citations

TL;DR

This paper proposes a zero-shot transfer learning framework for semantic parsing using a shared encoder-decoder with a learnable domain-mapping matrix to align cross-domain examples in a shared latent space. By leveraging influence functions for cross-domain adversarial attacks, it identifies and augments training data with influential examples from dissimilar domains, significantly improving zero-shot performance on both token and sequence levels without increasing model parameters beyond baseline.

ABSTRACT

While neural networks have shown impressive performance on large datasets, applying these models to tasks where little data is available remains a challenging problem. In this paper we propose to use feature transfer in a zero-shot experimental setting on the task of semantic parsing. We first introduce a new method for learning the shared space between multiple domains based on the prediction of the domain label for each example. Our experiments support the superiority of this method in a zero-shot experimental setting in terms of accuracy metrics compared to state-of-the-art techniques. In the second part of this paper we study the impact of individual domains and examples on semantic parsing performance. We use influence functions to this aim and investigate the sensitivity of domain-label classification loss on each example. Our findings reveal that cross-domain adversarial attacks identify useful examples for training even from the domains the least similar to the target domain. Augmenting our training data with these influential examples further boosts our accuracy at both the token and the sequence level.

Motivation & Objective

• To address the challenge of low-resource semantic parsing in zero-shot settings where labeled data is scarce.
• To improve cross-domain generalization by learning a shared representation across multiple semantic parsing domains.
• To identify and leverage influential training examples from dissimilar domains to enhance model performance.
• To minimize parameter growth while maintaining or improving accuracy in low-data regimes.

Proposed method

• A single shared encoder and decoder are used with a learnable weight matrix W_T to map each example to its domain, enabling parameter-efficient domain adaptation.
• The model minimizes a loss function that encourages predictions of the correct domain label, effectively learning a shared representation space.
• Influence functions are applied to quantify the impact of individual examples on domain classification loss, identifying the most influential training instances.
• Cross-domain adversarial attacks are performed by perturbing examples from one domain and measuring their effect on another, revealing asymmetric influence between domains.
• Training data is augmented by selecting top-k influential examples from less relevant domains (e.g., atis for calendar) and retraining to improve zero-shot performance.
• The method uses a regularization constraint via row-slicing of W_T to limit parameter growth, maintaining model efficiency.

Experimental results

Research questions

• RQ1 Can a shared encoder-decoder with a domain-mapping matrix achieve superior zero-shot performance in semantic parsing compared to existing transfer learning baselines?
• RQ2 How do individual examples and domains contribute to the model's performance, and which are most influential?
• RQ3 Can cross-domain adversarial attacks identify useful training examples even from domains least similar to the target domain?
• RQ4 Does data augmentation using influential examples from distant domains improve zero-shot semantic parsing accuracy?
• RQ5 Is the influence of examples on model performance symmetric across domains, or are some directions of transfer more effective?

Key findings

• The proposed zero-shot k*d method achieves higher accuracy than state-of-the-art shared-encoder and shared-decoder models in zero-shot settings, despite using fewer parameters.
• Adding more parameters for training in low-data regimes degrades performance, confirming the benefit of parameter efficiency.
• Even the least similar domain (e.g., atis for calendar) contains influential examples that, when added to training, improve performance at both token and sequence levels.
• Cross-domain adversarial attacks are not symmetric: influence is stronger in one direction (e.g., atis → calendar) than the reverse, as shown by influence function analysis.
• Data augmentation using influence-based sampling from distributions p (frequent examples) and 1−p (rare examples) improves performance, especially when n ≤ 40 training examples are available.
• The model’s performance remains robust when removing the closest domain (largest accuracy drop), but is more sensitive to removing the farthest domain, indicating non-uniform domain utility.