[Paper Review] Information-Preserving Contrastive Learning for Self-Supervised Representations.

Tianhong Li, Lijie Fan+4 more|arXiv (Cornell University)|Dec 17, 2020

Domain Adaptation and Few-Shot Learning52 references4 citations

TL;DR

This paper proposes Information-Preserving Contrastive Learning (IPCL), a self-supervised representation learning framework that preserves task-relevant information even when shortcuts—spurious correlations in data—exist. Unlike prior methods that eliminate shortcuts via data augmentation, IPCL explicitly retains both shortcut and meaningful features, leading to improved performance across diverse downstream tasks and modalities, including non-interpretable ones like radio signals.

ABSTRACT

Contrastive learning is very effective at learning useful representations without supervision. Yet contrastive learning has its limitations. It may learn a shortcut that is irrelevant to the downstream task, and discard relevant information. Past work has addressed this limitation via custom data augmentations that eliminate the shortcut. This solution however does not work for data modalities that are not interpretable by humans, e.g., radio signals. For such modalities, it is hard for a human to guess which shortcuts may exist in the signal, or how they can be eliminated. Even for interpretable data, sometimes eliminating the shortcut may be undesirable. The shortcut may be irrelevant to one downstream task but important to another. In this case, it is desirable to learn a representation that captures both the shortcut information and the information relevant to the other downstream task. This paper presents information-preserving contrastive learning (IPCL), a new framework for unsupervised representation learning that preserves relevant information even in the presence of shortcuts. We empirically show that the representations learned by IPCL outperforms contrastive learning in supporting different modalities and multiple diverse downstream tasks.

Motivation & Objective

• To address the limitation of contrastive learning in discarding relevant information when shortcuts exist in data.
• To develop a method that preserves both shortcut and task-relevant features, especially in non-interpretable data modalities like radio signals.
• To enable representation learning that supports multiple downstream tasks, even when shortcuts are beneficial for some but not others.
• To overcome the constraints of human-designed data augmentations, which are impractical for complex or non-interpretive data.

Proposed method

• IPCL introduces a contrastive learning framework that explicitly preserves information from both the shortcut and the underlying semantic content.
• It uses a dual-branch network to learn representations from augmented views while maintaining a trade-off between invariance and information preservation.
• The method incorporates a regularization term that encourages the model to retain information relevant to downstream tasks, even if it is correlated with spurious shortcuts.
• It leverages a contrastive objective that balances positive pairs (augmented views) while minimizing the risk of discarding useful signal.
• The framework is designed to be modular and applicable across different data modalities, including those not interpretable by humans.
• It avoids relying on human intuition for data augmentation by learning to preserve information intrinsically through the architecture and loss design.

Experimental results

Research questions

• RQ1 Can contrastive learning be improved to preserve relevant information when spurious shortcuts exist in the data?
• RQ2 How can representations be learned that maintain utility across multiple downstream tasks, even when shortcuts benefit only some?
• RQ3 Is it possible to preserve information in non-interpretable data modalities like radio signals without relying on human-designed augmentations?
• RQ4 Does preserving shortcut information degrade performance on downstream tasks that require semantic understanding?
• RQ5 Can a self-supervised framework outperform standard contrastive learning in diverse modalities and tasks by retaining both shortcut and meaningful features?

Key findings

• IPCL outperforms standard contrastive learning across multiple downstream tasks and data modalities, including non-interpretable signals like radio waves.
• The framework successfully preserves information from both shortcuts and semantically meaningful content, enabling better generalization.
• IPCL maintains performance on tasks where shortcuts are beneficial, while still capturing task-relevant features for others.
• The method generalizes well to data types where human-designed augmentations are impractical or infeasible.
• Empirical results show that IPCL achieves superior representation quality compared to baseline contrastive learning, especially in complex or ambiguous data environments.