[Paper Review] Unsupervised Learning of Syntactic Structure with Invertible Neural Projections

Junxian He, Graham Neubig+1 more|arXiv (Cornell University)|Aug 28, 2018

Topic Modeling36 references4 citations

TL;DR

This paper proposes a novel unsupervised generative model that jointly learns discrete syntactic structures (e.g., POS tags, dependency trees) and continuous word embeddings using invertible neural projections. By leveraging invertibility for exact inference and marginal likelihood computation, the model achieves state-of-the-art results on POS induction and unsupervised dependency parsing without gold POS or punctuation constraints on the Penn Treebank.

ABSTRACT

Unsupervised learning of syntactic structure is typically performed using generative models with discrete latent variables and multinomial parameters. In most cases, these models have not leveraged continuous word representations. In this work, we propose a novel generative model that jointly learns discrete syntactic structure and continuous word representations in an unsupervised fashion by cascading an invertible neural network with a structured generative prior. We show that the invertibility condition allows for efficient exact inference and marginal likelihood computation in our model so long as the prior is well-behaved. In experiments we instantiate our approach with both Markov and tree-structured priors, evaluating on two tasks: part-of-speech (POS) induction, and unsupervised dependency parsing without gold POS annotation. On the Penn Treebank, our Markov-structured model surpasses state-of-the-art results on POS induction. Similarly, we find that our tree-structured model achieves state-of-the-art performance on unsupervised dependency parsing for the difficult training condition where neither gold POS annotation nor punctuation-based constraints are available.

Motivation & Objective

• To bridge the gap between unsupervised syntactic modeling and continuous word representations by jointly learning discrete syntactic structures and continuous embeddings.
• To enable exact inference and marginal likelihood computation in a generative model with discrete latent variables by enforcing invertibility in the neural projection.
• To improve unsupervised POS induction and dependency parsing by learning a more structured, syntax-aware embedding space from raw text.
• To demonstrate that invertible projections can guide learned embeddings to focus on syntactic properties without supervision or morphological annotations.

Proposed method

• The model uses a structured prior (Markov or tree-structured) to generate discrete syntactic labels for each word.
• Latent continuous embeddings are sampled from a Gaussian distribution conditioned on the discrete syntactic labels.
• Observed pre-trained word embeddings are generated via an invertible neural network that maps the latent embeddings to the observed space.
• Invertibility of the neural projector enables exact posterior inference and marginal likelihood computation, provided the prior is tractable.
• The model is trained end-to-end by maximizing the marginal likelihood, with the inverse of the projector used to parameterize the likelihood.
• A Jacobian regularization term is included to prevent information loss during the invertible transformation.

Experimental results

Research questions

• RQ1 Can invertible neural networks enable exact inference and marginal likelihood computation in a generative model with discrete latent syntactic variables?
• RQ2 Does jointly learning continuous word embeddings and syntactic structure improve performance on unsupervised syntactic analysis tasks?
• RQ3 Can the learned embedding space better capture syntactic similarity than pre-trained embeddings like skip-gram?
• RQ4 How does the model perform in low-resource settings, such as without gold POS tags or punctuation constraints?

Key findings

• The Markov-structured model achieves state-of-the-art performance on POS induction on the Penn Treebank, surpassing previous unsupervised models.
• The tree-structured model (using DMV) achieves state-of-the-art results on unsupervised dependency parsing under the most challenging training condition—without gold POS tags or punctuation constraints.
• t-SNE visualizations show that the learned latent embeddings form well-separated clusters corresponding to gold POS tags, especially under the Markov structure.
• The DMV-structured model learns embeddings that distinguish between subject and object nouns based on syntactic role, even without dependency supervision.
• The model improves over a Gaussian baseline and maintains strong performance even when using fastText embeddings, indicating robustness to embedding source.
• Qualitative analysis confirms that the learned embeddings focus on syntactic similarity, with nearest neighbors reflecting syntactic rather than semantic similarity.