[Paper Review] Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representation on Sequence Labelling Tasks

Lizhen Qu, Gabriela Ferraro+4 more|arXiv (Cornell University)|Apr 21, 2015

Topic Modeling37 references23 citations

TL;DR

This paper evaluates five word representation methods—Brown clustering and four neural word embeddings—on four sequence labelling tasks (POS-tagging, chunking, NER, MWE). It finds that word embeddings and Brown clusters significantly improve performance on OOV and out-of-domain words, and that updating embeddings during training offers minimal gains and risks overfitting, with no single embedding method consistently outperforming others across tasks.

ABSTRACT

Word embeddings -- distributed word representations that can be learned from unlabelled data -- have been shown to have high utility in many natural language processing applications. In this paper, we perform an extrinsic evaluation of five popular word embedding methods in the context of four sequence labelling tasks: POS-tagging, syntactic chunking, NER and MWE identification. A particular focus of the paper is analysing the effects of task-based updating of word representations. We show that when using word embeddings as features, as few as several hundred training instances are sufficient to achieve competitive results, and that word embeddings lead to improvements over OOV words and out of domain. Perhaps more surprisingly, our results indicate there is little difference between the different word embedding methods, and that simple Brown clusters are often competitive with word embeddings across all tasks we consider.

Motivation & Objective

• To evaluate the impact of different word representation methods on sequence labelling tasks under controlled conditions.
• To investigate whether word embeddings generalize better than one-hot or Brown clusters, especially with limited training data.
• To assess the effect of task-specific updating of word embeddings on performance and vector geometry.
• To analyze performance on out-of-vocabulary (OOV) and out-of-domain words across different representation methods.
• To determine whether any word embedding method consistently outperforms others across multiple sequence labelling tasks.

Proposed method

• Five word representation methods are evaluated: Brown clustering, Collobert & Weston (CW), CBOW, Skip-gram, and GloVe.
• All word representations are used as input features in CRF-based sequence labelling models for POS-tagging, chunking, NER, and MWE identification.
• Models are trained with varying amounts of labelled data, from as few as 100 to full training sets, to assess data efficiency.
• For updating experiments, word embeddings are fine-tuned during sequence labelling training using backpropagation, while others are kept fixed.
• Performance is measured using standard metrics (F1, accuracy) on in-domain, out-of-domain, and OOV word subsets.
• Geometric analysis of vector changes during updating is performed to assess impact on word representation space.

Experimental results

Research questions

• RQ1 RQ1: Do word embeddings and Brown clusters outperform one-hot unigram features in sequence labelling tasks?
• RQ2 RQ2: Can word embeddings reduce the need for large amounts of labelled data, especially for low-resource settings?
• RQ3 RQ3: What is the empirical and geometric impact of updating pre-trained word embeddings during task-specific training?
• RQ4 RQ4: How do word representations perform on OOV words and out-of-domain data?
• RQ5 RQ5: Is there a consistently superior word embedding method across different sequence labelling tasks?

Key findings

• Word embeddings and Brown clusters significantly outperform one-hot unigram features, especially with limited training data, with as few as 100–200 instances yielding competitive results.
• Updating word embeddings during training provides only marginal performance gains and increases the risk of overfitting, particularly on low-frequency and OOV words.
• Brown clusters are often competitive with neural word embeddings across all four tasks, suggesting their strong inductive bias and robustness.
• Both word embeddings and Brown clusters improve performance on OOV and out-of-domain words, with the best results achieved when embeddings are not updated.
• No single word embedding method consistently outperforms others across all tasks; Skip-gram shows a slight edge in POS-tagging, but this does not generalize.
• The performance gap between the authors' best models and state-of-the-art systems is attributed to model complexity (e.g., first-order CRF vs. second-order) and hyperparameter tuning, not word representation choice.