[Paper Review] Self-supervised Cross-silo Federated Neural Architecture Search

Xinle Liang, Yang Liu+4 more|arXiv (Cornell University)|Jan 28, 2021

Privacy-Preserving Technologies in Data74 references17 citations

TL;DR

This paper proposes Self-supervised Cross-silo Federated Neural Architecture Search (SS-VFNAS), a method that enables privacy-preserving, automated neural architecture search in vertical federated learning (VFL) settings. Each party first performs self-supervised representation learning on its local data, then collaboratively optimizes a shared, high-performing architecture using only overlapping samples, achieving superior performance, communication efficiency, and privacy compared to existing federated NAS approaches—especially under low data overlap.

ABSTRACT

Federated Learning (FL) provides both model performance and data privacy for machine learning tasks where samples or features are distributed among different parties. In the training process of FL, no party has a global view of data distributions or model architectures of other parties. Thus the manually-designed architectures may not be optimal. In the past, Neural Architecture Search (NAS) has been applied to FL to address this critical issue. However, existing Federated NAS approaches require prohibitive communication and computation effort, as well as the availability of high-quality labels. In this work, we present Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) for automating FL where participants hold feature-partitioned data, a common cross-silo scenario called Vertical Federated Learning (VFL). In the proposed framework, each party first conducts NAS using self-supervised approach to find a local optimal architecture with its own data. Then, parties collaboratively improve the local optimal architecture in a VFL framework with supervision. We demonstrate experimentally that our approach has superior performance, communication efficiency and privacy compared to Federated NAS and is capable of generating high-performance and highly-transferable heterogeneous architectures even with insufficient overlapping samples, providing automation for those parties without deep learning expertise.

Motivation & Objective

• To address the challenge of designing optimal neural architectures in vertical federated learning (VFL) where data is feature-partitioned and privacy is critical.
• To reduce communication and computational overhead in federated NAS by leveraging self-supervised pretraining before collaborative architecture search.
• To enable non-expert participants to automatically discover high-performance, transferable models without access to global data or labels.
• To improve model performance and privacy preservation in VFL, especially when overlapping samples are limited.
• To demonstrate that self-supervised representations from diverse data domains can effectively initialize and improve downstream VFL architectures.

Proposed method

• Each party performs self-supervised representation learning on its local, unlabeled data using contrastive learning to learn transferable features.
• Local optimal architectures are searched using NAS on the self-supervised representations, without requiring labels or raw data sharing.
• Parties collaboratively refine the local architectures in a vertical federated learning setup using only the 10% overlapping samples for supervised training.
• The framework supports integration with various NAS algorithms and enables communication-efficient, privacy-preserving architecture search.
• Differential privacy is applied to transmitted gradients to further protect model and data privacy during training.
• The method is evaluated on multi-domain medical imaging datasets (FedCheXpert, FedModelNet40) under varying overlap and privacy constraints.

Experimental results

Research questions

• RQ1 Can self-supervised pretraining on local, unlabeled data improve the quality of neural architecture search in vertical federated learning?
• RQ2 How does the proposed SS-VFNAS method perform under low data overlap (e.g., 10%) compared to standard federated NAS?
• RQ3 To what extent can self-supervised representations from different domains (e.g., MRI vs. PET) serve as effective initialization for collaborative architecture search?
• RQ4 How does the integration of differential privacy affect model accuracy and communication efficiency in SS-VFNAS?
• RQ5 Can SS-VFNAS achieve better performance and transferability than existing federated NAS methods while preserving data privacy?

Key findings

• SS c+c -VFNAS 1 and SS c+m -VFNAS 1 outperform SS-VFNAS 1 by a greater margin under 10% overlapping samples than under 100% overlap, demonstrating the method's robustness to data scarcity.
• The mean test accuracy of SS c+m -VFNAS 1 reaches 0.742 on FedCheXpert with 10% overlap, outperforming SS-VFNAS 1 (0.733) and VFNAS 1 (0.698).
• SS-VFNAS achieves significantly higher accuracy than VFNAS under the same differential privacy noise variance, with performance degradation only approaching baseline levels at noise variance 10.
• The method maintains high performance even when pretraining data comes from different domains (e.g., FedCheXpert and FedModelNet40), showing strong transferability of self-supervised representations.
• Communication efficiency is improved due to reduced need for exchanging architecture and weight parameters, especially in low-overlap scenarios.
• The integration of differential privacy with SS-VFNAS preserves model utility better than in standard VFNAS, indicating enhanced privacy-efficiency trade-offs.