[Paper Review] Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction

Yichun Yin, Furu Wei+4 more|arXiv (Cornell University)|May 25, 2016

Advanced Text Analysis TechniquesComputer Science82 citations

TL;DR

This paper proposes an unsupervised method for aspect term extraction using distributed word and dependency path embeddings, modeling syntactic relationships via the objective w₁ + r ≈ w₂ with RNN-learned path representations. It achieves state-of-the-art performance using only embedding-based features in CRF models, demonstrating that syntactic structure significantly improves aspect term detection beyond word-level representations.

ABSTRACT

In this paper, we develop a novel approach to aspect term extraction based on unsupervised learning of distributed representations of words and dependency paths. The basic idea is to connect two words (w1 and w2) with the dependency path (r) between them in the embedding space. Specifically, our method optimizes the objective w1 + r = w2 in the low-dimensional space, where the multi-hop dependency paths are treated as a sequence of grammatical relations and modeled by a recurrent neural network. Then, we design the embedding features that consider linear context and dependency context information, for the conditional random field (CRF) based aspect term extraction. Experimental results on the SemEval datasets show that, (1) with only embedding features, we can achieve state-of-the-art results; (2) our embedding method which incorporates the syntactic information among words yields better performance than other representative ones in aspect term extraction.

Motivation & Objective

• To improve aspect term extraction by incorporating syntactic structure through unsupervised representation learning.
• To model multi-hop dependency paths as sequences of grammatical relations using RNNs for improved word representation.
• To design embedding-based features—linear context, dependency context, and word embeddings—for CRF-based sequence labeling.
• To achieve competitive performance with top systems using only unsupervised embeddings, avoiding reliance on hand-crafted features.
• To validate that syntactic-aware embeddings outperform standard word embeddings in aspect term extraction.

Proposed method

• The method optimizes the objective w₁ + r ≈ w₂ in a low-dimensional space, where w₁ and w₂ are word embeddings and r is a dependency path.
• Dependency paths are modeled as sequences of grammatical relations and embedded using a recurrent neural network (RNN).
• Word embeddings are enhanced via multi-task learning with linear context information.
• Continuous embeddings are discretized to make them suitable for CRF models.
• Feature engineering combines target word embeddings, linear context embeddings, and dependency context embeddings for CRF input.
• The final model uses only these learned embeddings as features, without hand-crafted rules or external knowledge.

Experimental results

Research questions

• RQ1 Can unsupervised word and dependency path embeddings improve aspect term extraction performance without labeled data?
• RQ2 How effective is the w₁ + r ≈ w₂ objective in capturing syntactic relationships for aspect term detection?
• RQ3 Does incorporating multi-hop dependency path representations lead to better performance than word-only embeddings?
• RQ4 Can a CRF model achieve state-of-the-art results using only learned embedding features?
• RQ5 How do syntactic structure and context information jointly contribute to distinguishing aspect terms from non-aspect terms?

Key findings

• The proposed method achieves state-of-the-art F1 scores of 79.23 on the SemEval 2014 dataset using only embedding-based features.
• The WDEmb model (with word and dependency path embeddings) outperforms the baseline DepEmb by 1.46 F1 points (p < 0.05).
• Performance peaks at l=15 for interval number and d=100 for embedding dimension, balancing accuracy and training efficiency.
• Qualitative analysis confirms that learned embeddings capture syntactic and topical similarity, with red and blue highlights indicating functionally and topically mismatched words.
• The method significantly improves performance on aspect terms with similar linear context but different syntactic roles, demonstrating the value of syntactic structure.
• Discretized embeddings are effective for CRF models, and the approach achieves competitive results without hand-crafted features.