[Paper Review] Distributional Term Set Expansion

Amaru Cuba Gyllensten, Magnus Sahlgren|arXiv (Cornell University)|Feb 14, 2018

Topic Modeling6 references7 citations

TL;DR

This paper evaluates iterative term set expansion methods using distributional semantic models, comparing centrality-based approaches (e.g., eigencentrality, signal-to-noise) with active learning-based methods (Simple Margin with linear and RBF kernels). The key finding is that Simple Margin with an RBF kernel consistently outperforms all other methods across five term sets and four distributional models, establishing it as the best practice for distributional term set expansion.

ABSTRACT

This paper is a short empirical study of the performance of centrality and classification based iterative term set expansion methods for distributional semantic models. Iterative term set expansion is an interactive process using distributional semantics models where a user labels terms as belonging to some sought after term set, and a system uses this labeling to supply the user with new, candidate, terms to label, trying to maximize the number of positive examples found. While centrality based methods have a long history in term set expansion, we compare them to classification methods based on the the Simple Margin method, an Active Learning approach to classification using Support Vector Machines. Examining the performance of various centrality and classification based methods for a variety of distributional models over five different term sets, we can show that active learning based methods consistently outperform centrality based methods.

Motivation & Objective

• To evaluate and compare the performance of centrality-based and classification-based iterative term set expansion methods using distributional semantic models.
• To identify the most effective method for expanding term sets in a semi-automated, interactive process guided by user labeling.
• To determine whether certain expansion methods are more suitable for specific distributional models or term set types.
• To establish a best practice for distributional term set expansion based on empirical evaluation.

Proposed method

• The study employs iterative term set expansion, where a user labels initial seed terms, and the system proposes new candidates based on distributional similarity.
• Centrality-based methods include centroid expansion, eigencentrality, and signal-to-noise ratio, which rank terms by their proximity to seed terms in vector space.
• Classification-based methods use the Simple Margin active learning approach with Support Vector Machines, selecting the most informative terms for labeling.
• Two variants of Simple Margin are evaluated: linear and RBF kernel, both using labeled examples to train a binary classifier that predicts new positive terms.
• Four distributional models are used: Factorized PPMI, Factorized Smoothed PPMI, CBOW, and SGNS, all trained on the British National Corpus with window size 2 and 200 dimensions.
• Performance is measured by the average number of positive examples found per iteration over ten random initializations with five positive and five negative seed terms.

Experimental results

Research questions

• RQ1 Which methods are most commonly used for iterative term set expansion using distributional models?
• RQ2 What are the performance differences between centrality-based and classification-based expansion methods across various distributional models?
• RQ3 Are certain expansion methods more effective for specific distributional models or term set types?
• RQ4 Is there a single method that consistently outperforms others across different models and term sets, suggesting a best practice?

Key findings

• Simple Margin with an RBF kernel achieved the highest average number of positive examples found per iteration across all five term sets and all distributional models.
• For the AFINN POS term set, Simple Margin with RBF kernel found 3.25 positive examples per iteration on average with CBOW, and 4.27 with PPMI, outperforming all other methods.
• Simple Margin with a linear kernel also performed strongly, especially with PPMI and SPMI, but was outperformed by the RBF variant in all cases.
• Centrality-based methods like centroid expansion and signal-to-noise ratio were consistently outperformed by all classification-based methods, with eigencentrality showing the weakest performance.
• SGNS showed poor performance with centrality-based methods but performed well with Simple Margin using RBF kernel, suggesting local noise but global coherence in its representations.
• CBOW exhibited a notable drop in performance with Simple Margin using a linear kernel, indicating potential global noise in its vector space, while RBF kernel mitigated this issue effectively.