[论文解读] Multiview Self-Representation Learning across Heterogeneous Views

Jie Chen, Zhu Wang+2 more|arXiv (Cornell University)|Feb 4, 2026

Domain Adaptation and Few-Shot Learning被引用 0

一句话总结

论文提出了 MSRL，一种完全无监督的迁移学习框架，通过信息传递自表示机制和分配概率分布一致性方案，在多种预训练模型的异质特征之间学习不变的表征。

ABSTRACT

Features of the same sample generated by different pretrained models often exhibit inherently distinct feature distributions because of discrepancies in the model pretraining objectives or architectures. Learning invariant representations from large-scale unlabeled visual data with various pretrained models in a fully unsupervised transfer manner remains a significant challenge. In this paper, we propose a multiview self-representation learning (MSRL) method in which invariant representations are learned by exploiting the self-representation property of features across heterogeneous views. The features are derived from large-scale unlabeled visual data through transfer learning with various pretrained models and are referred to as heterogeneous multiview data. An individual linear model is stacked on top of its corresponding frozen pretrained backbone. We introduce an information-passing mechanism that relies on self-representation learning to support feature aggregation over the outputs of the linear model. Moreover, an assignment probability distribution consistency scheme is presented to guide multiview self-representation learning by exploiting complementary information across different views. Consequently, representation invariance across different linear models is enforced through this scheme. In addition, we provide a theoretical analysis of the information-passing mechanism, the assignment probability distribution consistency and the incremental views. Extensive experiments with multiple benchmark visual datasets demonstrate that the proposed MSRL method consistently outperforms several state-of-the-art approaches.

研究动机与目标

• 使用多种具有本质上不同特征分布的预训练骨干网，从大规模无标注视觉数据中激励学习不变的表征。
• 提出一个多视图自表示学习框架，汇聚各视图的信息。
• 通过分配概率分布一致性机制实现跨视图表征的不变性。
• 对信息传递机制和多视图一致性提供理论见解。
• 在基准数据集上展示对现有方法的经验性优越性。

提出的方法

• 在每个冻结的预训练骨干网之上叠加一个单独的线性模型，以获得异质的多视图特征。
• 引入一个信息传递机制，通过基于注意力的算子自适应地聚合相邻特征。
• 将每个特征表示为其同类邻居的线性组合，以形成低维表示。
• 对聚合特征使用基于 softmax 的分类器，计算每个视图的分配概率分布。
• 通过对视图之间的分配分布取平均并施加语义伪标签损失、簇多样性损失和跨视图一致性损失来实现跨视图的一致性。
• 提供对弱邻域对齐、边界化的多视图一致性以及增量视图分析的理论分析。

Figure 1 : Framework of the MSRL model with two pretrained models. The framework consists of three main modules: a transfer learning module, a feature self-representation learning (FSRL) module and an assignment probability distribution consistency (APDC) module.

实验结果

研究问题

• RQ1 如何从无标注数据中学习跨异构预训练模型的不变表征？
• RQ2 信息传递（自表示）机制是否能有效聚合跨视图特征，产生鲁棒的低维表示？
• RQ3 分配概率分布一致性方案是否能改善跨视图对齐和多预训练骨干网的聚类性能？
• RQ4 对所提出的信息传递和多视图一致性机制有哪些理论保障？
• RQ5 在基准视觉数据集上，MSRL在无监督迁移方法中的实际表现如何？

主要发现

• MSRL在多个基准视觉数据集上持续优于若干最先进的方法。
• 信息传递机制使基于邻居的自适应特征聚合成为可能，生成低维表示。
• 分配概率分布一致性利用各视图的互补信息来强化跨视图不变性。
• 理论结果证明弱邻域对齐和有界的多视图一致性，并展示增量视图分析。
• 该框架在完全无监督的迁移设置下工作，骨干网保持冻结。
• 语义伪标签损失、簇多样性损失和跨视图一致性损失共同引导模型朝向统一的潜在分布。

Figure 2 : Clustering accuracy comparison between MSRL and the zero-shot transfer learning-based methods.