[Paper Review] Efficient Vector Representation for Documents through Corruption

Minmin Chen|arXiv (Cornell University)|Jul 8, 2017

Topic Modeling29 references78 citations

TL;DR

Doc2VecC represents a document as the average of its word embeddings learned with a corruption-based regularization, enabling fast and scalable document representations that perform well on sentiment analysis, classification, and semantic relatedness.

ABSTRACT

We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC). Doc2VecC represents each document as a simple average of word embeddings. It ensures a representation generated as such captures the semantic meanings of the document during learning. A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero. Doc2VecC produces significantly better word embeddings than Word2Vec. We compare Doc2VecC with several state-of-the-art document representation learning algorithms. The simple model architecture introduced by Doc2VecC matches or out-performs the state-of-the-art in generating high-quality document representations for sentiment analysis, document classification as well as semantic relatedness tasks. The simplicity of the model enables training on billions of words per hour on a single machine. At the same time, the model is very efficient in generating representations of unseen documents at test time.

Motivation & Objective

• Motivate efficient document representations beyond BoW and prior neural methods.
• Propose a simple averaging-based document vector with a corruption mechanism.
• Show that corruption acts as data-dependent regularization favoring informative words.
• Demonstrate competitive or superior performance across sentiment analysis, classification, and semantic relatedness tasks.

Proposed method

• Represent each document as the average of its word embeddings learned jointly with local context.
• Introduce a corruption (dropout) mechanism that randomly removes words during learning and scales remaining components to be unbiased.
• Model the target word probability with local context and a global document context via P(w|c, x̃) and optimize with negative sampling.
• Provide a Taylor expansion around the corruption mean to derive a data-dependent regularization term that suppresses common uninformative words.
• Train projection matrices U and V similarly to Word2Vec, enabling efficient training and test-time inference.
• Represent unseen documents by simply averaging their learned word embeddings.

Experimental results

Research questions

• RQ1 Can a simple average of word embeddings, learned with a corruption-based objective, produce high-quality document representations?
• RQ2 Does the corruption mechanism act as a data-dependent regularizer that improves performance and training speed?
• RQ3 How does Doc2VecC compare to state-of-the-art document representations across sentiment analysis, classification, and semantic relatedness?
• RQ4 Is test-time representation generation efficient when using averaged word embeddings?

Key findings

• Doc2VecC achieves competitive or superior performance to Paragraph Vectors and other baselines on sentiment analysis, classification, and semantic relatedness.
• Training time is fast and scales to large corpora, with test-time representation requiring only a simple average of word embeddings.
• The corruption mechanism acts as a data-dependent regularizer, penalizing embeddings of common but non-discriminative words and reducing test-time computation.
• Empirically, Doc2VecC yields word embeddings that are less dominated by stop words and more informative for downstream tasks.
• Word analogy and semantic relatedness tasks show Doc2VecC embeddings outperform Word2Vec in many settings, especially with larger corpora.