[Paper Review] Multi-Participant Multi-Class Vertical Federated Learning

Siwei Feng, Han Yu|arXiv (Cornell University)|Jan 30, 2020

Privacy-Preserving Technologies in DataComputer Science23 references68 citations

TL;DR

MMVFL introduces a privacy-preserving framework for multi-class vertical federated learning with more than two participants, enabling label sharing from a label owner to others and performing feature selection to reduce communication overhead. It achieves comparable multi-class performance to supervised baselines.

ABSTRACT

Federated learning (FL) is a privacy-preserving paradigm for training collective machine learning models with locally stored data from multiple participants. Vertical federated learning (VFL) deals with the case where participants sharing the same sample ID space but having different feature spaces, while label information is owned by one participant. Current studies of VFL only support two participants, and mostly focus on binaryclass logistic regression problems. In this paper, we propose the Multi-participant Multi-class Vertical Federated Learning (MMVFL) framework for multi-class VFL problems involving multiple parties. Extending the idea of multi-view learning (MVL), MMVFL enables label sharing from its owner to other VFL participants in a privacypreserving manner. To demonstrate the effectiveness of MMVFL, a feature selection scheme is incorporated into MMVFL to compare its performance against supervised feature selection and MVL-based approaches. Experiment results on real-world datasets show that MMVFL can effectively share label information among multiple VFL participants and match multi-class classification performance of existing approaches.

Motivation & Objective

• Extend vertical federated learning (VFL) to multi-class problems with multiple participants.
• Enable privacy-preserving label sharing from the label owner to other participants.
• Incorporate feature selection to assess and reduce communication and computation costs.
• Provide a mechanism to evaluate feature importance per participant to improve VFL efficiency.
• Demonstrate competitive multi-class classification performance on real datasets.

Proposed method

• Model MMVFL as a multi-view learning-inspired VFL framework with a separate model for each participant.
• Use sparse learning-based unsupervised feature selection to compute feature importance per participant via an optimization that includes an l2,1 regularizer (Eq. 1).
• Adapt label-sharing by introducing pseudo-label matrices Z_k and a shared Z with Y as the true labels for the label owner (Eq. 3).
• Relax hard constraints with penalty terms (Eq. 4) and solve via alternating optimization updating W_k (Eq. 8), Z_k (Eq. 11/12), and Z (Eq. 14).
• Provide a parallel, federated training procedure (Algorithm 1) where participants update local W_k, Z_k, and a central Z.
• Discuss convergence (non-increasing objective) and time complexity dominated by the slowest participant, O((max_k d_k)^3) per iteration.

Experimental results

Research questions

• RQ1 Can MMVFL enable effective multi-class VFL with more than two participants while preserving privacy?
• RQ2 Does label sharing from the label owner to other participants improve multi-class classification performance in VFL?
• RQ3 Is feature importance-based feature selection feasible and beneficial in MMVFL to reduce communication and computation?

Key findings

• MMVFL achieves label sharing without exposing raw data and yields competitive multi-class performance compared to supervised baselines.
• On Handwritten data, MMVFL outperformed the two baselines by up to about 2.31 percentage points on average across participants.
• On Caltech7, MMVFL performance is close to or slightly below the supervised baselines on average, illustrating competitive results in a privacy-preserving setting.
• Feature importance-based dropping of less informative features can reduce communication and computation while maintaining accuracy in many cases.
• MVL-based supervision (supMVLFL) can improve performance over non-MVL baselines, and MMVFL can achieve similar gains without sharing raw data.