[Paper Review] CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

Pengyu Cheng, Weituo Hao+4 more|arXiv (Cornell University)|Jun 22, 2020

Face and Expression Recognition59 citations

TL;DR

CLUB introduces a contrastive log-ratio upper bound for mutual information and its variational form for unknown conditionals, enabling MI minimization and improvements in information bottleneck and domain adaptation tasks.

ABSTRACT

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.

Motivation & Objective

• Motivate MI minimization in high-dimensional settings where distributions are unknown or intractable.
• Provide a new MI upper-bound estimator that is differentiable and trainable with gradient-based methods.
• Develop a variational extension (vCLUB) for cases where p(y|x) is unknown and show its theoretical properties.
• Enable MI minimization algorithms based on CLUB/vCLUB, with acceleration via negative sampling.
• Demonstrate CLUB's effectiveness through simulations and real-world tasks like Information Bottleneck and UD A.

Proposed method

• Define CLUB as an MI upper bound using the contrastive log-ratio between conditional probabilities with positive and negative pairs.
• Show CLUB is an upper bound to I(X;Y) and derive an unbiased estimator hat{I}_CLUB based on log p(y|x).
• Introduce vCLUB by approximating p(y|x) with a variational network q_{θ}(y|x) and derive its properties.
• Propose an MI minimization algorithm that alternates updating the variational network and the joint distribution p_{σ}(x,y).
• Accelerate computation with a negative-sampling strategy resulting in sampled vCLUB (vCLUB-S) with linear time complexity in N.
• Provide optional sampling to CLUB when p(y|x) is known or unknown.

Experimental results

Research questions

• RQ1 Can CLUB provide a reliable and trainable upper bound for mutual information that supports MI minimization?
• RQ2 How does the variational extension vCLUB behave when p(y|x) is unknown and q_{θ}(y|x) approximates it well?
• RQ3 Does CLUB/vCLUB offer favorable bias-variance trade-offs compared to existing MI estimators in estimation and minimization tasks?
• RQ4 Can CLUB-based MI minimization improve representations in Information Bottleneck and related domain adaptation tasks?
• RQ5 Does negative sampling (vCLUB-S) maintain unbiasedness and improve efficiency and generalization?

Key findings

• CLUB consistently yields lower bias and competitive MSE in MI estimation compared to baseline bounds and lower bounds on toy data.
• The vCLUB bound remains an upper bound or reliable estimator when the variational approximation q_{θ}(y|x) closely matches p(y|x).
• BI and UDA experiments show CLUB-based MI minimization improves downstream performance and representation quality versus previous bounds.
• Negative sampling (vCLUB-S) reduces computational cost to O(N) per iteration while maintaining unbiasedness of the estimator.
• In information bottleneck settings, CLUB and its sampled variants achieve competitive misclassification rates, often outperforming lower-bound estimators and L1-out in training stability.
• Sampling-based CLUB variants can enhance generalization and training efficiency without sacrificing estimation accuracy.