[Paper Review] FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients

Daixun Li, Weiying Xie+4 more|arXiv (Cornell University)|Nov 16, 2023

Remote-Sensing Image Classification4 citations

TL;DR

FedDiff proposes a novel federated learning framework that integrates diffusion models for privacy-preserving, multi-modal remote sensing data fusion across multiple clients. By employing a dual-branch encoder to process hyperspectral and LiDAR data separately and leveraging cross-branch denoising connections, FedDiff achieves state-of-the-art classification accuracy (96.77% OA) with reduced communication costs, marking the first deployment of diffusion models in federated learning for heterogeneous multi-modal data.

ABSTRACT

With the rapid development of imaging sensor technology in the field of remote sensing, multi-modal remote sensing data fusion has emerged as a crucial research direction for land cover classification tasks. While diffusion models have made great progress in generative models and image classification tasks, existing models primarily focus on single-modality and single-client control, that is, the diffusion process is driven by a single modal in a single computing node. To facilitate the secure fusion of heterogeneous data from clients, it is necessary to enable distributed multi-modal control, such as merging the hyperspectral data of organization A and the LiDAR data of organization B privately on each base station client. In this study, we propose a multi-modal collaborative diffusion federated learning framework called FedDiff. Our framework establishes a dual-branch diffusion model feature extraction setup, where the two modal data are inputted into separate branches of the encoder. Our key insight is that diffusion models driven by different modalities are inherently complementary in terms of potential denoising steps on which bilateral connections can be built. Considering the challenge of private and efficient communication between multiple clients, we embed the diffusion model into the federated learning communication structure, and introduce a lightweight communication module. Qualitative and quantitative experiments validate the superiority of our framework in terms of image quality and conditional consistency.

Motivation & Objective

• To address the challenge of securely fusing heterogeneous multi-modal remote sensing data—such as hyperspectral (HSI) and LiDAR—across multiple clients without exposing raw data.
• To overcome the limitations of existing single-modality and single-client diffusion models in federated settings, particularly in handling distributed, high-dimensional, and heterogeneous data.
• To develop a lightweight, efficient communication module that enables secure feature exchange between clients while preserving data privacy.
• To enable effective feature-level fusion by exploiting complementary denoising capabilities between different modalities in a diffusion process.

Proposed method

• FedDiff employs a dual-branch encoder architecture where HSI and LiDAR data are processed independently through separate encoders to extract modality-specific features.
• The framework establishes bilateral connections between the two branches based on complementary denoising potential, enhancing feature refinement during the diffusion process.
• A lightweight communication module transmits intermediate features instead of raw data between clients, minimizing communication overhead while preserving privacy.
• The global model is updated via federated averaging using these intermediate features, enabling collaborative learning without data sharing.
• The diffusion model is trained end-to-end in a conditional generation setup, where the denoising process is guided by multi-modal features to improve reconstruction and classification.
• Frequency domain analysis is incorporated to enhance feature representation by filtering or emphasizing specific spectral-spatial components.

Experimental results

Research questions

• RQ1 Can a diffusion model be effectively adapted to a federated learning framework for multi-modal remote sensing data fusion?
• RQ2 How can complementary denoising capabilities across different modalities (e.g., HSI and LiDAR) be leveraged to improve feature representation in a federated setting?
• RQ3 What is the impact of using intermediate feature exchange instead of raw data on communication efficiency and model performance?
• RQ4 How does FedDiff compare to existing methods in terms of classification accuracy and communication cost across multi-modal remote sensing datasets?
• RQ5 Can the integration of frequency domain filtering improve the robustness and expressiveness of fused features in a federated diffusion learning setup?

Key findings

• FedDiff achieves a mean classification accuracy of 96.77% on three multi-modal remote sensing datasets, outperforming existing methods in both accuracy and communication efficiency.
• On the MUUFL dataset, FedDiff achieves an overall accuracy (OA) of 95.32%, surpassing the next-best method (MFT at 94.34%) by 0.98 percentage points.
• The framework reduces communication cost significantly by transmitting only intermediate features rather than raw data, enabling scalable deployment in distributed environments.
• Visualization results show that FedDiff produces smoother, less noisy classification maps compared to traditional methods, which exhibit salt-and-pepper noise due to reliance on spectral-only representations.
• FedDiff demonstrates superior performance in classifying challenging classes such as water bodies and building shadows, achieving 96.84% and 90.52% accuracy respectively on the MUUFL dataset.
• The integration of frequency domain filtering enhances feature representation, leading to finer-grained texture expression and improved generalization in classification tasks.