[Paper Review] Class Vectors: Embedding representation of Document Classes

Devendra Singh Sachan, Shailesh Kumar|arXiv (Cornell University)|Aug 2, 2015

Sentiment Analysis and Opinion Mining13 references3 citations

TL;DR

This paper introduces Class Vectors, a method that learns dense vector representations for document classes within the same embedding space as words and paragraphs. By jointly training class vectors with word embeddings using a modified skip-gram model, the approach uses cosine similarity between class vectors and word vectors as features for text classification, achieving state-of-the-art performance on sentiment analysis tasks like Yelp and Amazon reviews.

ABSTRACT

Distributed representations of words and paragraphs as semantic embeddings in high dimensional data are used across a number of Natural Language Understanding tasks such as retrieval, translation, and classification. In this work, we propose "Class Vectors" - a framework for learning a vector per class in the same embedding space as the word and paragraph embeddings. Similarity between these class vectors and word vectors are used as features to classify a document to a class. In experiment on several sentiment analysis tasks such as Yelp reviews and Amazon electronic product reviews, class vectors have shown better or comparable results in classification while learning very meaningful class embeddings.

Motivation & Objective

• To address the limitation of traditional text classification methods that rely on sparse, bag-of-words features by introducing dense, semantic class representations.
• To explore whether class-level embeddings can capture discriminative semantic patterns similar to word and paragraph embeddings.
• To improve document classification performance by leveraging the semantic similarity between class vectors and words in the vocabulary.
• To evaluate the effectiveness of class vectors in low-resource and high-resource settings using sentiment analysis benchmarks.
• To investigate the interpretability and quality of learned class vectors through semantic similarity analysis with discriminative words.

Proposed method

• The model extends the skip-gram framework to jointly train word vectors and class vectors in a shared embedding space.
• Each class is represented by a unique class ID that co-occurs with every word in its corresponding documents within a sliding window.
• The objective function combines standard skip-gram prediction loss with a new term that maximizes the likelihood of predicting words given their class vector.
• Class vector similarity to words is computed using the dot product and softmax normalization to produce a probability score for each class.
• The model uses negative sampling and hierarchical softmax to efficiently optimize the large-scale objective function.
• For classification, word-class similarity scores are used as features with a Logistic Regression classifier.

Experimental results

Research questions

• RQ1 Can class-level embeddings be effectively learned in the same vector space as word embeddings to improve text classification?
• RQ2 How do class vectors compare to traditional bag-of-words and deep learning baselines (e.g., PV-DBOW, CNN) in sentiment classification tasks?
• RQ3 Do class vectors capture meaningful semantic relationships with discriminative words that define each class?
• RQ4 What impact does data shuffling and joint training over all classes have on the quality and discriminative power of class vectors?
• RQ5 Can class vector-based feature selection outperform frequency-based methods in identifying informative, low-frequency words?

Key findings

• On the Yelp reviews dataset, the proposed class vector method with Logistic Regression (CV-LR) achieved 94.83% accuracy, outperforming all baseline methods except the CNN-based approach.
• On the Amazon electronic product reviews dataset, the CV-LR model achieved 91.70% accuracy, surpassing bag-of-words, Naive Bayes, and PV-DBOW baselines.
• Class vectors learned meaningful semantic associations: the positive class vector was highly similar to words like 'fantastic', 'awesome', and 'very_pleased', while the negative class vector was similar to 'awful', 'terrible', and 'piece_of_crap'.
• Normalization of class vectors (norm CV-LR) further improved performance, achieving 94.91% accuracy on Yelp, indicating that scaling enhances generalization.
• The method demonstrated that low-frequency words can have high mutual information with classes, showing advantages over traditional document frequency-based feature selection.
• Shuffling the corpus during training was critical—training class vectors on isolated class data degraded their discriminative power, confirming the need for joint learning across all classes.