[Paper Review] Hierarchical Optimal Transport for Robust Multi-View Learning

Dixin Luo, Hongteng Xu+1 more|arXiv (Cornell University)|Jun 4, 2020

Domain Adaptation and Few-Shot Learning41 references4 citations

TL;DR

This paper proposes Hierarchical Optimal Transport (HOT), a robust multi-view learning method that eliminates reliance on sample correspondence and identical latent distributions across views. By using sliced Wasserstein distances as ground distances and learning optimal transport matrices, HOT explicitly models clustering structures in views, improving performance in both semi-supervised and unsupervised settings across synthetic and real-world datasets.

ABSTRACT

Traditional multi-view learning methods often rely on two assumptions: ($i$) the samples in different views are well-aligned, and ($ii$) their representations in latent space obey the same distribution. Unfortunately, these two assumptions may be questionable in practice, which limits the application of multi-view learning. In this work, we propose a hierarchical optimal transport (HOT) method to mitigate the dependency on these two assumptions. Given unaligned multi-view data, the HOT method penalizes the sliced Wasserstein distance between the distributions of different views. These sliced Wasserstein distances are used as the ground distance to calculate the entropic optimal transport across different views, which explicitly indicates the clustering structure of the views. The HOT method is applicable to both unsupervised and semi-supervised learning, and experimental results show that it performs robustly on both synthetic and real-world tasks.

Motivation & Objective

• To address the limitations of traditional multi-view learning methods that assume well-aligned samples and identical latent distributions across views.
• To develop a method that can learn meaningful representations from unaligned, distributed multi-view data, especially in privacy-sensitive settings like healthcare and finance.
• To explicitly model the clustering structure of views using optimal transport, improving representation learning and generalization.
• To enable effective semi-supervised and unsupervised multi-view learning without requiring labeled data in all views.
• To provide a more interpretable and robust alternative to co-regularization methods by learning view-level clustering via optimal transport.

Proposed method

• The method uses sliced Wasserstein distance (SWD) to measure distributional discrepancy between latent representations of different views, avoiding the need for sample correspondence.
• SWD is used as the ground distance to compute entropic optimal transport across views, forming a hierarchical optimal transport (HOT) model that captures pairwise view similarities.
• A learnable weight is introduced for each pair of views in the SWD, which is interpreted as an optimal transport plan between views, enabling clustering structure discovery.
• The method incorporates global reference representations to explicitly learn view clusters, allowing the optimal transport matrix to reflect groupings of views with similar or redundant information.
• Mini-batch stochastic gradient descent combined with the Sinkhorn algorithm enables efficient optimization of the HOT model.
• The framework supports both semi-supervised and unsupervised learning, with optional autoencoder decoders to further regularize representations via reconstruction loss.

Experimental results

Research questions

• RQ1 Can optimal transport based on sliced Wasserstein distances effectively regularize multi-view learning without requiring sample correspondence?
• RQ2 Does modeling view-level clustering via optimal transport improve performance in multi-view learning under weak assumptions?
• RQ3 How does the HOT model compare to baseline methods in semi-supervised and unsupervised settings on real-world and synthetic data?
• RQ4 Can the learned optimal transport matrices reveal meaningful groupings of views, and are these groupings interpretable in terms of information redundancy?
• RQ5 To what extent does the HOT model reduce over-regularization caused by enforcing identical latent distributions across views?

Key findings

• The proposed HOT model consistently outperforms strong baselines on multiple datasets, achieving higher classification accuracy in both semi-supervised and unsupervised settings.
• In unsupervised feature extraction, methods relying on explicit view clustering (via HOT) retain high accuracy, while those using pairwise comparisons degrade significantly.
• The optimal transport matrices learned by HOT reveal interpretable clustering structures: for example, in Caltech7, Gabor and CENTIST form one cluster, WM and GIST another, while HOG and LBP are mixtures of clusters.
• Removing views from the same cluster (e.g., Gabor or CENTRIST) causes minimal performance drop, confirming that such views carry redundant information.
• In contrast, removing a unique view like MOR from a cluster leads to severe performance degradation, validating the model’s ability to identify structurally distinct views.
• The inclusion of autoencoder reconstruction loss further improves classification accuracy, demonstrating the benefit of multi-objective regularization in the HOT framework.