[論文レビュー] FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation

Min Tan, Junchao Ma+7 more|arXiv (Cornell University)|Mar 5, 2026

Privacy-Preserving Technologies in Data被引用数 0

ひとこと要約

FedAFD は三つのモジュールからなるフレームワークを導入する。ビリベル対立的整合、粒度意識型特徴融合、類似性ガイド付きアンサンブル蒸留を通じて、同一データを共有せずにヘテロジニアスなクライアントとアーキテクチャ間でプライバシーを保つマルチモーダルフェデレーテッド学習を実現し、IID および非 IID 設定下でクライアント個別化とサーバ全体の検索性能を改善する。

ABSTRACT

Multimodal Federated Learning (MFL) enables clients with heterogeneous data modalities to collaboratively train models without sharing raw data, offering a privacy-preserving framework that leverages complementary cross-modal information. However, existing methods often overlook personalized client performance and struggle with modality/task discrepancies, as well as model heterogeneity. To address these challenges, we propose FedAFD, a unified MFL framework that enhances client and server learning. On the client side, we introduce a bi-level adversarial alignment strategy to align local and global representations within and across modalities, mitigating modality and task gaps. We further design a granularity-aware fusion module to integrate global knowledge into the personalized features adaptively. On the server side, to handle model heterogeneity, we propose a similarity-guided ensemble distillation mechanism that aggregates client representations on shared public data based on feature similarity and distills the fused knowledge into the global model. Extensive experiments conducted under both IID and non-IID settings demonstrate that FedAFD achieves superior performance and efficiency for both the client and the server.

研究の動機と目的

• Motivate multimodal federated learning when clients hold heterogeneous modalities and tasks without sharing raw data.
• Mitigate modality and task gaps to reduce model drift between clients and the server.
• Enable effective personalization for edge clients while maintaining strong global performance.
• Propose architecture-agnostic aggregation to handle heterogeneous client models on a shared server.

提案手法

• Bi-level Adversarial Alignment (BAA) to align local and global representations within and across modalities using intra-modal and cross-modal discriminators.
• Granularity-aware Feature Fusion (GFF) to adaptively fuse local and global features at multiple levels via attention-based gating.
• Similarity-guided Ensemble Distillation (SED) to weight and distill client representations on public data into the global model based on semantic similarity.

実験結果

リサーチクエスチョン

• RQ1 How to harmonize diverse modalities and tasks in a federated setting without sharing raw data?
• RQ2 Can adversarial alignment and adaptive fusion reduce modality/task drift while preserving local personalization?
• RQ3 How to aggregate heterogeneous client knowledge effectively on a server to improve global multimodal retrieval?
• RQ4 What are the performance and convergence benefits of FedAFD under IID and non-IID data distributions?

主な発見

• FedAFD improves both client-side personalization and server-side retrieval performance compared with state-of-the-art baselines.
• The method is robust to both IID and non-IID settings, with notable improvements in non-IID scenarios.
• FedAFD requires fewer communication rounds to reach target performance, indicating faster convergence.
• Ablation studies show each component (BAA, GFF, SED) contributes to overall performance, with GFF notably boosting edge client effectiveness and SED strengthening global distillation.
• Interpretability analyses show FedAFD produces more aligned, compact feature representations across clients and server compared with purely local training.