[Paper Review] Deep Partial Multi-View Learning

Changqing Zhang, Yajie Cui+4 more|arXiv (Cornell University)|Nov 12, 2020

Domain Adaptation and Few-Shot Learning22 citations

TL;DR

This paper proposes Cross Partial Multi-View Networks (CPM-Nets), a deep learning framework that jointly learns complete, versatile latent representations and imputes missing views in multi-view data using adversarial training and a nonparametric classification loss. The method achieves state-of-the-art performance in classification, representation learning, and data imputation under complex view-missing scenarios, with improvements of up to 5.11% in NMI and 3.36% in ACC over baselines at 50% missing rate.

ABSTRACT

Although multi-view learning has made signifificant progress over the past few decades, it is still challenging due to the diffificulty in modeling complex correlations among different views, especially under the context of view missing. To address the challenge, we propose a novel framework termed Cross Partial Multi-View Networks (CPM-Nets), which aims to fully and flflexibly take advantage of multiple partial views. We fifirst provide a formal defifinition of completeness and versatility for multi-view representation and then theoretically prove the versatility of the learned latent representations. For completeness, the task of learning latent multi-view representation is specififically translated to a degradation process by mimicking data transmission, such that the optimal tradeoff between consistency and complementarity across different views can be implicitly achieved. Equipped with adversarial strategy, our model stably imputes missing views, encoding information from all views for each sample to be encoded into latent representation to further enhance the completeness. Furthermore, a nonparametric classifification loss is introduced to produce structured representations and prevent overfifitting, which endows the algorithm with promising generalization under view-missing cases. Extensive experimental results validate the effectiveness of our algorithm over existing state of the arts for classifification, representation learning and data imputation.

Motivation & Objective

• Address the challenge of learning robust, complete, and versatile representations from multi-view data with arbitrary and complex view-missing patterns.
• Overcome limitations of existing methods that assume complete views or rely on fixed grouping strategies, which lack flexibility under high-dimensional or incomplete view configurations.
• Enhance representation completeness by modeling view imputation as a degradation process and stabilizing it via adversarial training.
• Improve generalization and separability of learned representations through a clustering-like nonparametric classification loss.
• Enable joint exploitation of all samples and views regardless of missingness patterns, ensuring adaptability to real-world data with incomplete modalities.

Proposed method

• Formalize completeness and versatility of multi-view representations through a theoretical framework grounded in data transmission analogy, modeling representation learning as a degradation process to balance consistency and complementarity.
• Introduce an adversarial strategy where a generator network learns to reconstruct missing views from observed ones, with the discriminator enforcing realistic imputations, thereby stabilizing and improving representation quality.
• Employ a nonparametric classification loss that encourages clustering-structured representations, enhancing separability and generalization, especially under view-missing conditions.
• Use a shared encoder to map all observed views into a unified latent space, enabling joint optimization across all samples and view combinations.
• Apply a degradation strategy to simulate missing views during training, allowing the model to implicitly learn to reconstruct and encode information from incomplete inputs.
• Train the model end-to-end using a combination of reconstruction loss (for imputation), contrastive loss (for representation learning), and nonparametric classification loss (for structure and generalization).

Figure 1: Illustration of Cross Partial Multi-View Networks for classification. Given multi-view data with missing views (black blocks), the encoding 2 2 2 Since we employ a backward encoding strategy which encodes the information from multiple views into latent representations, it can be considered

Experimental results

Research questions

• RQ1 Can a deep learning framework jointly learn complete and versatile latent representations from multi-view data with arbitrary view-missing patterns?
• RQ2 How does adversarial training improve the quality of missing view imputation and, in turn, enhance the completeness and generalization of learned representations?
• RQ3 To what extent does a nonparametric classification loss improve the structure and separability of latent representations in partial multi-view learning?
• RQ4 How does the proposed CPM-Nets framework compare to state-of-the-art methods in terms of classification, representation learning, and data imputation under heavy view-missing conditions?
• RQ5 Can the model generalize effectively to naturally occurring partial multi-view datasets without requiring complete-view training data?

Key findings

• CPM-GAN outperforms all comparative methods in view imputation across all six datasets, with the lowest NRMSE across all missing rates, validating the effectiveness of the adversarial imputation strategy.
• On average, CPM-GAN improves classification performance by 3.36% in Accuracy and 5.11% in NMI over the second-best method at a 50% missing rate, demonstrating superior robustness and generalization.
• The ablation study confirms that CPM-GAN consistently outperforms CPM-without-GAN, especially under high missing rates, proving that adversarial training enhances imputation and representation quality.
• The nonparametric classification loss leads to more structured and separable representations, improving model interpretability and performance, particularly in low-data regimes.
• Even when compared to VIGAN—a method requiring a small amount of complete-view data for training—CPM-Nets achieves better clustering performance on two-view datasets (CUB and Animal), showing its superiority in fully unsupervised settings.
• The model maintains strong performance on both synthetic and real-world naturally partial multi-view datasets, confirming its adaptability to complex, real-world view-missing patterns.

Figure 2: Illustration of CPM-Nets for unsupervised learning. The latent representation learning and missing data imputation equipped with adversarial strategy (CPM-GAN) are jointly conducted to improve each other.