[Paper Review] Exploring phrase-compositionality in skip-gram models

Xiaochang Peng, Daniel Gildea|arXiv (Cornell University)|Jul 21, 2016

Natural Language Processing Techniques17 references3 citations

TL;DR

This paper proposes a joint learning framework that extends the skip-gram model to jointly optimize word and phrase embeddings while explicitly modeling phrase compositionality through a learnable composition function. By incorporating phrase-level context and enforcing compositional constraints on both input and output embeddings, the model improves performance on word/phrase similarity, syntactic analogy, and dependency parsing tasks, with gains up to 0.87% in UAS on the PTB dependency parsing benchmark.

ABSTRACT

In this paper, we introduce a variation of the skip-gram model which jointly learns distributed word vector representations and their way of composing to form phrase embeddings. In particular, we propose a learning procedure that incorporates a phrase-compositionality function which can capture how we want to compose phrases vectors from their component word vectors. Our experiments show improvement in word and phrase similarity tasks as well as syntactic tasks like dependency parsing using the proposed joint models.

Motivation & Objective

• To address data sparsity in phrase-level distributed representations by jointly learning word and phrase embeddings.
• To model how word vectors compose into phrase vectors using a learnable composition function.
• To improve performance on word similarity, phrase similarity, and syntactic tasks like dependency parsing.
• To incorporate positional and syntactic information from phrases into the embedding learning process.
• To evaluate the impact of joint modeling of word-level and phrase-level skip-gram on downstream NLP tasks.

Proposed method

• Extends the standard skip-gram model to include phrase-level prediction, where the current phrase vector predicts context phrases.
• Uses negative sampling to optimize the phrase-level objective function, with phrase embeddings learned via stochastic gradient descent.
• Introduces a composition function Φ(⊕(σ(v_w1), ..., σ(v_wn))) that combines word vectors into phrase vectors using a differentiable operation ⊕ (e.g., sum, concatenation) and a non-linear transform Φ.
• Jointly optimizes word-level and phrase-level skip-gram objectives with a hyperparameter β balancing the two losses.
• Applies the senna toolkit to extract syntactic chunks (e.g., NP, VP) from text to define phrases for the phrase-level model.
• Employs a shared composition function for both input and output phrase embeddings, enforcing consistency in the embedding space.

Experimental results

Research questions

• RQ1 Can joint learning of word and phrase embeddings with a learnable composition function improve performance on word and phrase similarity tasks?
• RQ2 Does modeling phrase-level co-occurrence improve syntactic downstream tasks such as dependency parsing?
• RQ3 How does the inclusion of positional and syntactic information in phrase embeddings affect performance on similarity and parsing tasks?
• RQ4 What is the relative contribution of compositional modeling versus positional modeling in improving embedding quality?
• RQ5 Can a general composition function effectively model phrase embeddings across diverse syntactic structures?

Key findings

• The joint model with compositional modeling achieves the highest Spearman’s correlation of 0.704 on the word similarity task (WordSim-353), outperforming both word2vec and positional-only baselines.
• On the syntactic analogy (SYN) and mixed analogy (MIXED) tasks, combining compositional and positional modeling yields the best performance, with 80.5% accuracy on MIXED.
• The dependency parsing model using the compositional+positional joint embeddings achieves a UAS of 92.19% and LAS of 90.82% on the PTB test set, a 0.28% and 0.28% improvement over the baseline word2vec model.
• Modeling phrase-level skip-gram over syntactic phrases leads to a 0.18% absolute improvement in UAS over the baseline, demonstrating the value of syntactic structure in embedding learning.
• The inclusion of positional information slightly degrades performance on word similarity tasks, suggesting that compositional modeling is more effective for semantic similarity.
• The best performance on the intransitive verb disambiguation task is achieved with the joint compositional model, confirming its ability to learn meaningful phrase representations.