[Paper Review] Joint Embedding of Words and Labels for Text Classification

Guoyin Wang, Chunyuan Li+6 more|arXiv (Cornell University)|May 10, 2018

Topic Modeling34 references45 citations

TL;DR

LEAM embeds words and labels in a shared space, using label-word compatibility to attend and weight word embeddings for text classification, achieving strong accuracy with low complexity.

ABSTRACT

Word embeddings are effective intermediate representations for capturing semantic regularities between words, when learning the representations of text sequences. We propose to view text classification as a label-word joint embedding problem: each label is embedded in the same space with the word vectors. We introduce an attention framework that measures the compatibility of embeddings between text sequences and labels. The attention is learned on a training set of labeled samples to ensure that, given a text sequence, the relevant words are weighted higher than the irrelevant ones. Our method maintains the interpretability of word embeddings, and enjoys a built-in ability to leverage alternative sources of information, in addition to input text sequences. Extensive results on the several large text datasets show that the proposed framework outperforms the state-of-the-art methods by a large margin, in terms of both accuracy and speed.

Motivation & Objective

• Motivate why label information should influence early text representations, not just the final classifier.
• Propose a joint word-label embedding framework to produce label-aware, interpretable text representations.
• Develop an attention mechanism based on word-label compatibility to weight words for classification.
• Demonstrate that label embeddings provide computationally efficient and accurate text classification across multiple datasets.
• Show potential clinical applicability through multi-label medical code prediction and interpretability of attention words.

Proposed method

• Embed both words and class labels into a shared latent space.
• Compute word-label compatibility G via cosine similarities and a nonlinear, localized attention using phrase windows.
• Construct text representation z as a weighted average of word embeddings using the attention scores (SoftMax over phrases).
• Train end-to-end with standard classification losses (cross-entropy for single-label; sigmoid-based for multi-label) and a regularization term placing label embeddings on their class manifolds.
• Regularize label embeddings to align with actual class descriptions, enabling interpretability and meaningful anchors.
• Optionally initialize word embeddings with pre-trained vectors (e.g., GloVe) and learn label embeddings alongside.

Experimental results

Research questions

• RQ1 Can joint word-label embeddings improve text classification by guiding word representations with label information?
• RQ2 Does the proposed label-embedding attentive model LEAM achieve competitive accuracy with lower computational cost compared to deep attention architectures?
• RQ3 Do label embeddings provide interpretable attention that highlights informative words for prediction?
• RQ4 How does LEAM perform on standard benchmark datasets and a real-world medical code prediction task?

Key findings

• LEAM outperforms several state-of-the-art baselines on benchmark document classification datasets, achieving top results on Yahoo and DBPedia.
• LEAM offers favorable model size and speed, with faster convergence than some baselines and lower parameter count than CNN/LSTM models.
• The nonlinear, spatially-aware attention (phrase-based) is necessary for high performance, outperforming a linear variant.
• Label embeddings are meaningful, correlating well with class centers and enabling interpretable attention that highlights task-relevant keywords.
• In multi-label clinical text, LEAM achieves best AUC and competitive F1/P@5 metrics, with attention visualization highlighting health-related terms.
• LEAM’s attention can reduce a clinician’s reading burden by making informative words more salient.