[Paper Review] On the Effects of Self-supervision and Contrastive Alignment in Deep Multi-view Clustering

Daniel J. Trosten, Sigurd Løkse+2 more|arXiv (Cornell University)|Mar 17, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper introduces DeepMVC, a unified framework for deep multi-view clustering that integrates recent methods and enables systematic analysis of self-supervision and contrastive alignment. It theoretically and empirically demonstrates that contrastive alignment harms cluster separability—especially with many views—and proposes six new, superior DeepMVC instances that outperform prior methods, particularly on high-view datasets.

ABSTRACT

Self-supervised learning is a central component in recent approaches to deep multi-view clustering (MVC). However, we find large variations in the development of self-supervision-based methods for deep MVC, potentially slowing the progress of the field. To address this, we present DeepMVC, a unified framework for deep MVC that includes many recent methods as instances. We leverage our framework to make key observations about the effect of self-supervision, and in particular, drawbacks of aligning representations with contrastive learning. Further, we prove that contrastive alignment can negatively influence cluster separability, and that this effect becomes worse when the number of views increases. Motivated by our findings, we develop several new DeepMVC instances with new forms of self-supervision. We conduct extensive experiments and find that (i) in line with our theoretical findings, contrastive alignments decreases performance on datasets with many views; (ii) all methods benefit from some form of self-supervision; and (iii) our new instances outperform previous methods on several datasets. Based on our results, we suggest several promising directions for future research. To enhance the openness of the field, we provide an open-source implementation of DeepMVC, including recent models and our new instances. Our implementation includes a consistent evaluation protocol, facilitating fair and accurate evaluation of methods and components.

Motivation & Objective

• Address the lack of consistency and reproducibility in deep multi-view clustering (MVC) research due to divergent implementations, evaluation protocols, and hyperparameter choices.
• Identify and analyze the negative impact of contrastive alignment on cluster separability, especially as the number of views increases.
• Develop a unified framework (DeepMVC) that generalizes existing MVC methods and enables fair, consistent evaluation of components and models.
• Propose new MVC instances with improved self-supervision strategies that outperform state-of-the-art methods on multiple datasets.
• Promote reproducibility and progress in the field by releasing an open-source implementation with a standardized evaluation protocol.

Proposed method

• Design DeepMVC as a modular framework with five core components: view-specific encoders, single-view self-supervised learning (SV-SSL), multi-view self-supervised learning (MV-SSL), feature fusion, and clustering.
• Integrate existing MVC methods as instances of DeepMVC by mapping their components to the framework’s structure.
• Theoretically prove that contrastive alignment can reduce the number of separable clusters in representation space, with the negative effect worsening as the number of views increases.
• Develop six new DeepMVC instances using diverse self-supervision techniques: contrastive learning, mutual information maximization, autoencoding, and entropy regularization.
• Implement a consistent evaluation protocol across all models and datasets, including standardized data splits, hyperparameter search, and metrics (e.g., NMI).
• Use hyperparameter sweeps to assess robustness and identify sensitive configurations, such as high contrastive loss weights or low entropy regularization.

Experimental results

Research questions

• RQ1 How does contrastive alignment affect cluster separability in deep multi-view clustering, especially when the number of views increases?
• RQ2 Which forms of self-supervision are most effective across datasets with varying numbers of views?
• RQ3 Can a unified framework improve reproducibility and fair comparison in deep MVC research?
• RQ4 Do new DeepMVC instances with alternative self-supervision mechanisms outperform existing state-of-the-art methods?
• RQ5 How sensitive are the proposed models to hyperparameter configurations, particularly contrastive loss weight and entropy regularization?

Key findings

• Contrastive alignment significantly degrades clustering performance on datasets with many views, such as Caltech7, where performance drops to 0.09 NMI when contrastive loss weight is increased to 10.0.
• Mutual information maximization outperforms contrastive alignment on high-view datasets, achieving 0.84 NMI with InfoDDC on NoisyMNIST when entropy regularization is properly tuned.
• All evaluated methods benefit from at least one form of self-supervision, with AECoDDC achieving 0.99 NMI on NoisyMNIST and 0.49 NMI on Caltech7 under optimal reconstruction weight.
• The new DeepMVC instances are largely robust to hyperparameter changes, except when contrastive loss weight is too high (e.g., AECoDDC drops to 0.19 NMI on Caltech7) or entropy regularization is too low (e.g., InfoDDC drops from 0.94 to 0.21 NMI).
• The framework enables fair comparison: MvSCN and Multi-VAE show performance discrepancies in the new implementation, suggesting prior results may have been influenced by differing preprocessing and evaluation.
• The open-source DeepMVC implementation with standardized evaluation facilitates reproducibility and accelerates method development, as demonstrated by consistent and accurate benchmarking across multiple datasets.