[Paper Review] Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

Yaodong Yu, Kwan Ho Ryan Chan+3 more|arXiv (Cornell University)|Jun 15, 2020

Domain Adaptation and Few-Shot Learning39 references58 citations

TL;DR

The paper introduces Maximal Coding Rate Reduction (MCR2), an information-theoretic objective that maximizes the coding rate difference between the whole data and per-class subsets to learn diverse and discriminative representations across supervised, self-supervised, and unsupervised settings, with robustness to label noise and strong clustering performance.

ABSTRACT

To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction ($ ext{MCR}^2$), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class. We clarify its relationships with most existing frameworks such as cross-entropy, information bottleneck, information gain, contractive and contrastive learning, and provide theoretical guarantees for learning diverse and discriminative features. The coding rate can be accurately computed from finite samples of degenerate subspace-like distributions and can learn intrinsic representations in supervised, self-supervised, and unsupervised settings in a unified manner. Empirically, the representations learned using this principle alone are significantly more robust to label corruptions in classification than those using cross-entropy, and can lead to state-of-the-art results in clustering mixed data from self-learned invariant features.

Motivation & Objective

• Motivate learning intrinsic low-dimensional structures that discriminate between classes in high-dimensional data.
• Define and optimize a principled measure (MCR2) based on coding rate reduction to encourage inter-class diversity and intra-class compactness.
• Provide theoretical guarantees for learning diverse and discriminative features.
• Show that MCR2 works across supervised, self-supervised, and unsupervised settings within a unified framework.
• Demonstrate robustness to label corruption and competitive clustering performance on real data.

Proposed method

• Define the coding rate R(Z, ε) and the class-wise coding rate R^c(Z, ε|Π) for a partition Π of samples.
• Formulate ΔR(Z, Π, ε) = R(Z, ε) – R^c(Z, ε|Π) and maximize ΔR with normalization constraints on Z_j (Σ) to ensure comparable scales.
• Theorize that optimal MCR2 representations produce between-class orthogonality and maximal within-class subspace dimensions under suitable conditions.
• Relate MCR2 to information gain and contrastive learning, and contrast it with nuclear-norm based losses (OLE).
• Optimize the loss via stochastic gradient descent on mini-batches, using supervised (fixed Π from labels) and self-supervised (augmentations labeled by sample) settings.
• Evaluate representations with a nearest subspace classifier in supervised learning and with clustering metrics in self-supervised/unsupervised settings.

Experimental results

Research questions

• RQ1 Can a label-agnostic, rate-based measure quantify a representation’s compactness suitable for supervised, self-supervised, and unsupervised learning?
• RQ2 Does maximizing coding rate reduction yield discriminative, diverse, and robust representations across different learning paradigms?
• RQ3 How does MCR2 compare with cross-entropy, information bottleneck, and contrastive learning in terms of robustness to label noise and clustering performance?
• RQ4 What are the theoretical properties of MCR2 representations, especially in the subspace (low-dimensional) regime?
• RQ5 Can MCR2 automatically promote multi-class subspace structures that are orthogonal and of maximal dimension under appropriate conditions?

Key findings

• MCR2 learned representations exhibit increased whole-data coding rate R while per-class rates R^c decrease during supervised training, indicating expansion of global structure and compression of class-specific structure.
• Representations learned with MCR2 show higher diversity (larger subspace dimensions within each class) compared to cross-entropy, suggesting more robust and multifaceted features.
• MCR2 training demonstrates robustness to corrupted labels, maintaining higher accuracy as label corruption increases compared with cross-entropy.
• In self-supervised/augmentation-based settings, MCR2 (with controlled dynamics) achieves competitive clustering performance, including strong results on CIFAR10 with invariant features.
• Theoretical analysis indicates that, under suitable conditions, optimal Z* has orthogonal class subspaces and maximal dimensions, promoting diverse and discriminative representations.