[Paper Review] A Hybrid Self-Supervised Learning Framework for Vertical Federated Learning

Yuanqin He, Yan Kang+5 more|arXiv (Cornell University)|Aug 18, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes FedHSSL, a hybrid self-supervised learning framework for vertical federated learning that leverages cross-party views of aligned samples and local augmentations of unaligned samples to improve representation learning. It achieves state-of-the-art performance with a superior privacy-utility trade-off, outperforming baselines by large margins and demonstrating robustness against label inference attacks through invariant feature aggregation.

ABSTRACT

Vertical federated learning (VFL), a variant of Federated Learning (FL), has recently drawn increasing attention as the VFL matches the enterprises' demands of leveraging more valuable features to achieve better model performance. However, conventional VFL methods may run into data deficiency as they exploit only aligned and labeled samples (belonging to different parties), leaving often the majority of unaligned and unlabeled samples unused. The data deficiency hampers the effort of the federation. In this work, we propose a Federated Hybrid Self-Supervised Learning framework, named FedHSSL, that utilizes cross-party views (i.e., dispersed features) of samples aligned among parties and local views (i.e., augmentation) of unaligned samples within each party to improve the representation learning capability of the VFL joint model. FedHSSL further exploits invariant features across parties to boost the performance of the joint model through partial model aggregation. FedHSSL, as a framework, can work with various representative SSL methods. We empirically demonstrate that FedHSSL methods outperform baselines by large margins. We provide an in-depth analysis of FedHSSL regarding label leakage, which is rarely investigated in existing self-supervised VFL works. The experimental results show that, with proper protection, FedHSSL achieves the best privacy-utility trade-off against the state-of-the-art label inference attack compared with baselines. Code is available at \url{https://github.com/jorghyq2016/FedHSSL}.

Motivation & Objective

• To address data deficiency in vertical federated learning (VFL) caused by limited labeled and aligned samples.
• To improve representation learning in VFL by leveraging both cross-party views of aligned samples and local augmentations of unaligned samples.
• To enhance model generalization through partial aggregation of invariant features across parties.
• To analyze and mitigate label leakage in self-supervised VFL, a rarely studied issue in prior work.
• To achieve a better privacy-utility trade-off compared to state-of-the-art methods under label inference attacks.

Proposed method

• FedHSSL uses cross-party views—disseminated features from aligned samples across parties—as positive pairs for contrastive self-supervised learning.
• It applies local data augmentations within each party to generate additional views for unaligned samples, enhancing local representation learning.
• The framework performs partial model aggregation to extract and share invariant features across parties, improving joint model generalization.
• It integrates various self-supervised learning methods (e.g., SimSiam, BYOL, MoCo) as plug-in components, enabling flexibility and compatibility.
• A novel label leakage analysis is conducted on both pretraining and fine-tuning phases, with ISO-based protection mechanisms applied to evaluate privacy-utility trade-offs.
• The framework is evaluated using four real-world datasets (NUSWIDE, Avazu, BHI, Modelnet), with ablation studies confirming the contribution of each component.

Figure 1: The conventional VFL setting illustrated by two parties. Active party 1 owns a bottom model $f^{1}$ and a top model $g^{1}$ , while passive party 2 owns a bottom model $f^{2}$ . We call the joint VFL model composed of $f^{1}$ , $f^{2}$ , and $g^{1}$ FedSplitNN .

Experimental results

Research questions

• RQ1 Can a hybrid self-supervised learning framework effectively utilize both aligned and unaligned samples in vertical federated learning to improve model performance?
• RQ2 How does leveraging cross-party views of aligned samples compare to relying only on local augmentations in terms of representation quality?
• RQ3 To what extent does FedHSSL reduce label leakage compared to existing self-supervised VFL methods under label inference attacks?
• RQ4 What is the impact of varying amounts of aligned samples on the privacy-utility trade-off in FedHSSL?
• RQ5 Does partial aggregation of invariant features across parties significantly improve the performance of the joint VFL model?

Key findings

• FedHSSL outperforms all baselines by large margins across four datasets, with improvements ranging from 0.03 to 0.05 in main task accuracy.
• On the NUSWIDE dataset, FedHSSL-SimSiam achieved a main task accuracy of 0.833 with 40% aligned samples, significantly outperforming FedLocalSimSiam (0.533) under the same conditions.
• The calibrated averaged performance (CAP) metric showed that FedHSSL-SimSiam achieved a CAP of 0.284 on NUSWIDE and 0.349 on Modelnet, outperforming FedSplitNN and FedLocalSimSiam.
• With 40% aligned samples, FedHSSL-SimSiam achieved a main task accuracy of 0.788 on BHI, compared to 0.756 for FedLocalSimSiam, demonstrating the benefit of cross-party view utilization.
• In privacy analysis, FedHSSL achieved a better privacy-utility trade-off than baselines, with label recovery accuracy (MC attack) dropping to 0.443 on NUSWIDE when using 40% aligned samples and strong protection.
• Ablation studies confirmed that both cross-party view learning and local augmentation contribute significantly to performance, with the combination yielding the highest gains.

Figure 2: Architecture overview of three representative SSL methods. All methods comprise two encoders: an online encoder $f$ , and a target encoder $\tilde{f}$ . Gradients are not computed for the target encoder. For MoCo and BYOL, $\tilde{f}$ is the moving average of $f$ . MoCo has a queue to prov