[Paper Review] CRF Autoencoder for Unsupervised Dependency Parsing

Jiong Cai, Yong Jiang+1 more|arXiv (Cornell University)|Aug 3, 2017

Natural Language Processing Techniques14 references4 citations

TL;DR

This paper proposes a discriminative unsupervised dependency parsing model using a CRF autoencoder framework, where the encoder is a globally normalized, feature-rich CRF model for parsing and the decoder reconstructs input tokens from predicted parse structures. The method achieves state-of-the-art performance on eight multilingual treebanks, outperforming prior unsupervised models, especially when using universal linguistic priors.

ABSTRACT

Unsupervised dependency parsing, which tries to discover linguistic dependency structures from unannotated data, is a very challenging task. Almost all previous work on this task focuses on learning generative models. In this paper, we develop an unsupervised dependency parsing model based on the CRF autoencoder. The encoder part of our model is discriminative and globally normalized which allows us to use rich features as well as universal linguistic priors. We propose an exact algorithm for parsing as well as a tractable learning algorithm. We evaluated the performance of our model on eight multilingual treebanks and found that our model achieved comparable performance with state-of-the-art approaches.

Motivation & Objective

• To develop a tractable, discriminative approach to unsupervised dependency parsing that avoids the limitations of generative models.
• To leverage rich features and global normalization in a CRF-based encoder for improved parsing accuracy.
• To integrate a reconstruction-based training objective via a CRF autoencoder to guide latent structure learning.
• To evaluate the model on multilingual data and compare with state-of-the-art unsupervised parsing systems.
• To investigate the impact of universal linguistic priors on parsing performance.

Proposed method

• The model uses a CRF autoencoder with a discriminative encoder that models the conditional probability of parse trees given input sentences using a log-linear model with rich features.
• Parse tree scores are computed as the sum of feature-weighted dependency arc scores, normalized by a partition function computed via the Matrix-Tree Theorem or inside-outside algorithm.
• The decoder uses a categorical model to reconstruct input tokens from predicted heads, with parameters updated via Viterbi EM.
• Training alternates between optimizing the CRF weights using stochastic gradient descent and updating the decoder parameters via maximum likelihood estimation on the Viterbi parses.
• The objective function combines log-likelihood of the joint distribution of parse and reconstruction, with a Viterbi approximation to enable end-to-end training.
• Exact parsing is performed using a dynamic programming algorithm that finds the highest-scoring valid parse tree for each sentence.

Experimental results

Research questions

• RQ1 Can a discriminative, globally normalized CRF model outperform generative models in unsupervised dependency parsing?
• RQ2 Does incorporating rich features and universal linguistic priors improve parsing accuracy in an unsupervised setting?
• RQ3 Can the CRF autoencoder framework support tractable learning and exact inference for complex tree-structured outputs?
• RQ4 How does the Viterbi approximation of the objective function compare to standard negative log-likelihood in terms of performance?
• RQ5 Does the model generalize across multiple languages and achieve competitive results without gold annotations?

Key findings

• The proposed CRF autoencoder model achieves a directed dependency accuracy of 55.7% on the Wall Street Journal test set, outperforming the baseline negative log-likelihood objective (41.8%).
• On average across seven languages from the PASCAL Grammar Induction Challenge, the model achieves the highest performance among all compared systems, including Convex-MST and Neural DMV.
• The use of universal linguistic priors significantly improves performance, especially for models like Convex-MST and the proposed CRFAE.
• The Viterbi version of the objective function leads to substantially better parsing accuracy than the standard negative log-likelihood objective, confirming its effectiveness.
• The model demonstrates strong multilingual generalization, achieving competitive results on diverse languages without any labeled data.
• The method enables exact parsing and tractable learning, with efficient computation of partition functions using the Matrix-Tree Theorem for non-projective trees.