[Paper Review] A Mutual Information Maximization Perspective of Language Representation Learning

Lingpeng Kong, Cyprien de Masson d’Autume+4 more|arXiv (Cornell University)|Oct 18, 2019

Topic Modeling31 references81 citations

TL;DR

The paper reframes word representation learning as mutual information maximization via InfoNCE, unifying Skip-gram, BERT, and XLNet, and introduces InfoWord, a self-supervised objective combining DIM and MLM to improve downstream tasks like GLUE and SQuAD.

ABSTRACT

We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).

Motivation & Objective

• Motivate a unified, information-theoretic view of word representation learning.
• Show that Skip-gram, BERT, and XLNet approximate mutual information maximization objectives.
• Provide a general, extensible framework to create new self-supervised tasks.
• Demonstrate a new objective that combines global-sentence and local-n-gram mutual information.

Proposed method

• Frame Skip-gram, BERT, and XLNet as instances of the InfoNCE lower bound on mutual information (I(A,B)).
• Use f_theta(a,b)=g_psi(b)ᵀg_omega(a) to score cross-view representations.
• Derive how MLM and permutation-based objectives fit the InfoNCE perspective.
• Propose a new self-supervised objective (DIM-based) that maximizes mutual information between a global sentence representation and local n-grams.
• Introduce InfoWord as a weighted combination of a DIM term and a masked language modeling term: I_InfoWord = lambda_MLM * I_MLM + lambda_DIM * I_DIM.
• Demonstrate how negative sampling serves as an efficient approximation to large-vocabulary softmax in InfoNCE.

Experimental results

Research questions

• RQ1 Can a mutual information maximization perspective unify traditional and modern language representation learning methods?
• RQ2 What new self-supervised tasks can be constructed within this framework to improve language representations?
• RQ3 Does combining a global-sentence DIM objective with MLM improve downstream NLP tasks compared to standard BERT-style pretraining?
• RQ4 How does the proposed InfoWord method perform on GLUE and SQuAD relative to BERT variants?

Key findings

• InfoNCE-based framing unifies Skip-gram, BERT, and XLNet as instances of mutual information maximization.
• A simple new objective (DIM) enables learning a global sentence representation that aligns with its local n-gram representations.
• InfoWord, which combines I_MLM and I_DIM, yields better results than BERT-NCE on GLUE and SQuAD, especially for tasks needing longer-phrase understanding.
• Reimplementation variants (BERT-NCE) are competitive with original BERT in some settings and underperform in others due to masking and data presentation differences.
• Experiments indicate InfoWord’s advantage is most pronounced with smaller training sets, highlighting pretraining quality’s role when labeled data is scarce.