[Paper Review] Private Multi-Task Learning: Formulation and Applications to Federated Learning

Shengyuan Hu, Zhiwei Steven Wu+1 more|arXiv (Cornell University)|Aug 30, 2021

Privacy-Preserving Technologies in Data38 references4 citations

TL;DR

This paper proposes Private Mean-Regularized Multi-Task Learning (PMTL), a novel framework that enables client-level privacy in federated learning via billboard privacy (BP), ensuring each client's data remains private while allowing personalized models to depend on their own data. The method achieves better privacy-utility trade-offs than global baselines across FEMNIST, CelebA, and StackOverflow benchmarks, outperforming private global models and local training under strong privacy constraints.

ABSTRACT

Many problems in machine learning rely on multi-task learning (MTL), in which the goal is to solve multiple related machine learning tasks simultaneously. MTL is particularly relevant for privacy-sensitive applications in areas such as healthcare, finance, and IoT computing, where sensitive data from multiple, varied sources are shared for the purpose of learning. In this work, we formalize notions of client-level privacy for MTL via joint differential privacy (JDP), a relaxation of differential privacy for mechanism design and distributed optimization. We then propose an algorithm for mean-regularized MTL, an objective commonly used for applications in personalized federated learning, subject to JDP. We analyze our objective and solver, providing certifiable guarantees on both privacy and utility. Empirically, we find that our method provides improved privacy/utility trade-offs relative to global baselines across common federated learning benchmarks.

Motivation & Objective

• To address the lack of formal privacy guarantees in multi-task learning (MTL) for privacy-sensitive domains like healthcare, finance, and IoT.
• To formalize client-level privacy in MTL using billboard privacy (BP), a relaxation of differential privacy that allows task-specific models to depend on their own data.
• To develop a scalable, communication-efficient solver for mean-regularized MTL that provides certifiable $(\epsilon,\delta)$-billboard privacy.
• To empirically evaluate the privacy-utility trade-off of the proposed method against global baselines and local training in standard federated learning benchmarks.
• To demonstrate that private MTL can retain the accuracy benefits of MTL while offering stronger privacy than training a single global model.

Proposed method

• Proposes Private Mean-Regularized MTL, a framework that jointly optimizes a global model and client-specific models using a mean-regularization term to encourage similarity across tasks.
• Applies billboard privacy (BP) to ensure that the output models for all tasks except one are insensitive to changes in any single client’s data, enabling client-level privacy.
• Uses a communication-efficient solver based on FedAvg, adapted to handle the mean-regularized MTL objective with differential privacy guarantees.
• Introduces a convergence analysis for both convex and nonconvex objectives, extending to settings with partial client participation.
• Employs noise injection in the global model update step to satisfy $(\epsilon,\delta)$-billboard privacy, with privacy budget controlled via noise scale.
• Supports model personalization via fine-tuning after training, which incurs no additional privacy cost under the BP framework.

Experimental results

Research questions

• RQ1 Can multi-task learning be made private at the client level while preserving model utility?
• RQ2 How can billboard privacy be formally applied to mean-regularized MTL to ensure that client data remains protected?
• RQ3 What is the trade-off between privacy (measured by $\epsilon$) and utility (measured by test accuracy) in private MTL compared to private global models?
• RQ4 Does private MTL outperform local training and private global baselines in federated learning benchmarks under strong privacy constraints?
• RQ5 Can the proposed method achieve better privacy-utility trade-offs than existing methods like PP-SGD in personalized federated learning?

Key findings

• On the FEMNIST benchmark, PMTL achieves test accuracy of 0.681 under $\epsilon = 2.0$, significantly outperforming private FedAvg and FedProx.
• On CelebA, PMTL reaches 0.818 test accuracy at $\epsilon = 2.0$, surpassing both private global models and local training.
• On StackOverflow, PMTL achieves 0.324 test accuracy at $\epsilon = 0.1$, outperforming private FedAvg and FedProx, which drop below 0.31.
• For $\epsilon < 1$, PP-SGD yields worse privacy-utility trade-offs than PMTL, indicating PMTL’s superior privacy efficiency.
• Even under strong privacy ($\epsilon = 0.1$), PMTL with fine-tuning consistently outperforms pure local training on non-convex problems like FEMNIST and CelebA.
• The convergence analysis confirms that the solver maintains convergence guarantees under both convex and nonconvex settings, even with partial client participation.