[Paper Review] Unsupervised Feature Learning by Cross-Level Discrimination between Instances and Groups

Xudong Wang, Ziwei Liu+1 more|arXiv (Cornell University)|Aug 9, 2020

Machine Learning and Data Classification43 references14 citations

TL;DR

This paper proposes cross-level discrimination between instances and groups to stabilize unsupervised feature learning in real-world data with high instance correlation and long-tail class distributions. By leveraging both local attraction and long-range repulsion across groups, and decoupling grouping and discrimination on separate feature branches, it achieves state-of-the-art performance on self-supervised and semi-supervised benchmarks, overcoming instability and degeneracy issues.

ABSTRACT

Unsupervised feature learning has made great strides with invariant mapping and instance-level discrimination, as benchmarked by classification on common datasets. However, these datasets are curated to be distinctive and class-balanced, whereas naturally collected data could be highly correlated within the class (with repeats at the extreme) and long-tail distributed across classes. The natural grouping of instances conflicts with the fundamental assumption of instance-level discrimination. Contrastive feature learning is thus unstable without grouping, whereas grouping without contrastive feature learning is easily trapped into degeneracy. We propose to integrate grouping into instance-level discrimination, not by imposing group-level discrimination, but by imposing cross-level discrimination between instances and groups. Our key insight is that grouping results from not just attraction, but also repulsion. While invariant mapping is achieved by local attraction between augmented instances, instance similarity emerges from long-range repulsion against common instance groups. To further avoid the clash between grouping and discrimination objectives, we also impose them on separate features derived from the common feature. Our extensive experimentation demonstrates not only significant gain on datasets with high correlation and long-tail distributions, but also leading performance on multiple self-supervision and semi-supervision benchmarks, bringing unsupervised feature learning closer to real data applications.

Motivation & Objective

• To address the instability of contrastive feature learning in real-world data with high instance correlation and long-tail class distributions.
• To resolve the conflict between grouping objectives and instance-level discrimination in unsupervised representation learning.
• To prevent degeneracy in grouping-based methods by decoupling grouping and discrimination objectives.
• To improve generalization of unsupervised feature learning on datasets that deviate from idealized, class-balanced benchmarks.
• To integrate group-level structure into instance-level discrimination without explicitly imposing group-level discrimination.

Proposed method

• Introduce cross-level discrimination that combines local attraction between augmented instances and long-range repulsion against common instance groups.
• Decouple the feature space into separate branches for grouping and discrimination to avoid objective conflict.
• Use invariant mapping to encourage local feature consistency across data augmentations.
• Model instance similarity through repulsion from shared group representations, promoting distinctiveness across groups.
• Apply contrastive learning on the discrimination branch and grouping via clustering on the grouping branch.
• Leverage shared backbone features but apply grouping and discrimination objectives independently on separate feature streams.

Experimental results

Research questions

• RQ1 How can unsupervised feature learning be stabilized in data with high instance correlation and long-tail class distributions?
• RQ2 What is the role of repulsion between instances and groups in improving feature discrimination beyond local attraction?
• RQ3 Can decoupling grouping and discrimination objectives prevent degeneracy in self-supervised representation learning?
• RQ4 How does cross-level discrimination compare to standard instance-level or group-level contrastive learning on real-world data?
• RQ5 To what extent does the proposed method generalize across self-supervised and semi-supervised benchmarks?

Key findings

• The proposed method achieves significant performance gains on datasets with high instance correlation and long-tail distributions.
• It outperforms standard instance-level contrastive learning and group-level methods on multiple self-supervision benchmarks.
• The method demonstrates leading performance on semi-supervised learning benchmarks, indicating strong generalization.
• Decoupling grouping and discrimination objectives effectively prevents degeneracy and improves training stability.
• The integration of long-range repulsion against common groups enhances feature discriminability beyond local invariance.
• The approach generalizes well to real-world data distributions that deviate from idealized, balanced datasets.