[Paper Review] Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder

Caio Corro, Ivan Titov|arXiv (Cornell University)|Jul 25, 2018

Natural Language Processing Techniques44 citations

TL;DR

This paper presents a semi-supervised dependency parser based on a structured variational autoencoder and a differentiable perturb-and-parse mechanism to sample and learn from latent trees, with experiments on English, French, and Swedish.

ABSTRACT

Human annotation for syntactic parsing is expensive, and large resources are available only for a fraction of languages. A question we ask is whether one can leverage abundant unlabeled texts to improve syntactic parsers, beyond just using the texts to obtain more generalisable lexical features (i.e. beyond word embeddings). To this end, we propose a novel latent-variable generative model for semi-supervised syntactic dependency parsing. As exact inference is intractable, we introduce a differentiable relaxation to obtain approximate samples and compute gradients with respect to the parser parameters. Our method (Differentiable Perturb-and-Parse) relies on differentiable dynamic programming over stochastically perturbed edge scores. We demonstrate effectiveness of our approach with experiments on English, French and Swedish.

Motivation & Objective

• Address the high cost of syntactic annotations by leveraging unlabeled text beyond generic word embeddings.
• Propose a probabilistic generative model where a sentence is generated from a latent dependency tree and a sentence embedding.
• Train the model with a variational autoencoder objective combined with a discriminative objective on labeled data.
• Introduce a differentiable perturb-and-parse method to sample and differentiate through latent dependency trees.

Proposed method

• Define a generative model where the latent variables are a projective dependency tree T and a sentence embedding z, with p(s|T,z) parameterized by a neural network.
• Use a variational distribution qφ(T,z|s) to approximate the posterior and maximize an ELBO that combines unsupervised and supervised terms.
• Factorize the encoder as qφ(T|s) qφ(z|s), with T parameterized by a log-linear arc-weight model and z via a diagonal Gaussian from an LSTM.
• Decode autoregressively with an LSTM conditioned on z, using a Graph Convolutional Network to incorporate syntactic structure from T.
• Employ Differentiable Perturb-and-Parse: perturb arc weights with Gumbel-like noise and use a differentiable parsing surrogate to allow gradient-based learning over T.

Experimental results

Research questions

• RQ1 Can unlabeled data improve neural dependency parsing beyond learned word representations?
• RQ2 Can differentiable perturb-and-parse provide a workable gradient path for latent dependency trees in a VAE framework?
• RQ3 How does the semi-supervised VAE perform across English, French, and Swedish compared to a supervised baseline?
• RQ4 How should the generative and discriminative objectives be combined to train the parser effectively?

Key findings

• The approach introduces a variational autoencoder framework for semi-supervised dependency parsing.
• A differentiable Perturb-and-Parse operator enables gradient-based learning through latent trees.
• The method demonstrates effectiveness on English, French, and Swedish compared to a comparable supervised baseline.