[Paper Review] An Empirical Study on Disentanglement of Negative-free Contrastive Learning

Jinkun Cao, Ruiqian Nai+3 more|arXiv (Cornell University)|Jun 9, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a new disentanglement metric, MED (Mutual Information based Entropy Disentanglement), to evaluate negative-free contrastive learning in high-dimensional representation spaces—where existing metrics fail. It empirically demonstrates that negative-free contrastive methods can learn a well-disentangled subspace of representations, extending disentanglement research to real-world, complex data like CelebA and high-dimensional models beyond low-dimensional synthetic benchmarks.

ABSTRACT

Negative-free contrastive learning methods have attracted a lot of attention with simplicity and impressive performances for large-scale pretraining. However, its disentanglement property remains unexplored. In this paper, we examine negative-free contrastive learning methods to study the disentanglement property empirically. We find that existing disentanglement metrics fail to make meaningful measurements for high-dimensional representation models, so we propose a new disentanglement metric based on Mutual Information between latent representations and data factors. With this proposed metric, we benchmark the disentanglement property of negative-free contrastive learning on both popular synthetic datasets and a real-world dataset CelebA. Our study shows that the investigated methods can learn a well-disentangled subset of representation. As far as we know, we are the first to extend the study of disentangled representation learning to high-dimensional representation space and introduce negative-free contrastive learning methods into this area. The source code of this paper is available at \url{https://github.com/noahcao/disentanglement_lib_med}.

Motivation & Objective

• To investigate whether negative-free contrastive learning can produce disentangled representations in high-dimensional space, where existing disentanglement metrics are ineffective.
• To extend disentangled representation learning beyond synthetic, low-dimensional datasets to real-world, complex data such as CelebA.
• To develop a robust disentanglement metric suitable for high-dimensional representation models, overcoming limitations of prior metrics.
• To empirically benchmark the disentanglement performance of state-of-the-art negative-free contrastive learning methods on both synthetic and real-world datasets.

Proposed method

• Propose a new disentanglement metric, MED, based on mutual information between latent representations and underlying data factors, to enable meaningful evaluation in high-dimensional spaces.
• Use a top-k selection strategy to identify the most disentangled dimensions from the full representation space, based on MED scores.
• Apply the MED metric to evaluate disentanglement on both synthetic datasets (dSprites, Shapes3D, SmallNORB, Cars3D) and the real-world CelebA dataset.
• Train and evaluate multiple negative-free contrastive learning models (SimSiam, BYOL, Barlow Twins, etc.) using the proposed MED metric.
• Compare downstream performance of the top-k disentangled subspace against full representations and low-dimensional VAE baselines.
• Conduct ablation studies on the hyperparameter k to validate the robustness of the top-k MED selection across different subspace sizes.

Experimental results

Research questions

• RQ1 Can existing disentanglement metrics meaningfully evaluate high-dimensional representation models, particularly those from contrastive learning?
• RQ2 Do negative-free contrastive learning methods learn disentangled representations in high-dimensional latent spaces?
• RQ3 Can a well-disentangled subspace be identified from high-dimensional contrastive representations, and how does its performance compare to full representations?
• RQ4 How does the disentanglement quality of contrastive learning compare to traditional VAE-based generative models on real-world data like CelebA?
• RQ5 Does the top-k selected subspace from contrastive models achieve competitive or superior downstream performance compared to low-dimensional VAEs?

Key findings

• The proposed MED metric successfully evaluates disentanglement in high-dimensional representation spaces, where prior metrics fail due to design limitations.
• Negative-free contrastive learning methods such as SimSiam and BYOL achieve strong disentanglement in a subset of latent dimensions, as measured by MED.
• On the CelebA dataset, traditional VAE-based methods fail to learn good disentangled representations, highlighting a gap between current disentanglement research and real-world data complexity.
• The top-k disentangled subspace from contrastive models achieves comparable or better downstream performance than full representations from low-dimensional VAEs on dSprites, Shapes3D, and CelebA.
• The MED scores of the top-k subspace remain stable across different values of k, indicating robustness and reliability of the selection process.
• The study reveals that while the full representation contains factor-related information across many dimensions, the top-k selected dimensions are highly disentangled and interpretable.