[Paper Review] Self-supervised Representation Learning with Relative Predictive Coding

Yao-Hung Hubert Tsai, Quintín Martín Martín+4 more|arXiv (Cornell University)|Mar 21, 2021

Speech and Audio Processing58 references4 citations

TL;DR

This paper proposes Relative Predictive Coding (RPC), a novel contrastive representation learning objective that improves training stability, reduces minibatch size sensitivity, and enhances downstream performance by introducing relative parameters to regularize the loss and eliminating logarithmic and exponential terms that cause instability. RPC achieves state-of-the-art performance on vision and speech benchmarks with lower variance and consistent gains across datasets.

ABSTRACT

This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.

Motivation & Objective

• To address training instability in contrastive self-supervised learning, particularly due to high variance in loss functions.
• To reduce sensitivity to minibatch size, which limits practical deployment of contrastive methods requiring large batches.
• To improve downstream task performance while maintaining training stability and robustness.
• To develop a contrastive objective that avoids logarithmic and exponential operations, known sources of numerical instability.
• To establish a connection between RPC and mutual information estimation with low variance.

Proposed method

• RPC introduces relative parameters (α, β, γ) to regularize the contrastive loss, ensuring boundedness and reducing variance.
• The objective function replaces logarithmic and exponential terms with quadratic and linear terms, improving numerical stability.
• The core loss is defined as: J_RPC = -[f_k(h_{t+k}, c_t) - (α/|N|)Σf_k(h_i, c_t) - (β/2)f_k²(h_{t+k}, c_t) - (γ/(2|N|))Σf_k²(h_i, c_t)], where h are representations and c is context.
• Relative parameters act as ℓ₂ regularization, stabilizing the optimization landscape and preventing extreme values.
• The method uses a temperature τ=16 for vision and τ=16 for speech tasks, with no hidden normalization to preserve raw loss dynamics.
• RPC is empirically validated on CIFAR-10/100, STL-10, ImageNet, and LibriSpeech, using ResNet-18 and models from prior work.

Experimental results

Research questions

• RQ1 Can a contrastive learning objective achieve high downstream performance while maintaining training stability?
• RQ2 Does removing logarithmic and exponential terms in the loss function reduce training variance and improve stability?
• RQ3 How sensitive is the proposed method to minibatch size compared to prior contrastive objectives?
• RQ4 Can RPC be used to estimate mutual information with low variance?
• RQ5 What is the optimal configuration of relative parameters (α, β, γ) for stable and accurate MI estimation?

Key findings

• RPC achieves lower training variance than DV, NWJ, and CPC, especially without hidden normalization, indicating improved stability.
• RPC shows reduced sensitivity to minibatch size, maintaining strong performance even at smaller batch sizes, unlike CPC which requires large batches.
• On CIFAR-10, RPC achieves 93.8% top-1 accuracy with a ResNet-18, outperforming CPC and approaching fully supervised performance.
• On ImageNet, RPC achieves 76.5% top-1 accuracy in linear evaluation, demonstrating strong transferability.
• RPC estimates mutual information with low bias and variance when β ≈ 10⁻³ and γ ≈ 1.0, outperforming DoE in low-MI regimes.
• In MI estimation on 20D correlated Gaussians, RPC with β=10⁻³ and γ=1.0 yields stable and accurate estimates, while DoE becomes unstable when MI is small.