[Paper Review] Ditto: Fair and Robust Federated Learning Through Personalization

Tian Li, Shengyuan Hu+2 more|arXiv (Cornell University)|Dec 8, 2020

Privacy-Preserving Technologies in Data73 references218 citations

TL;DR

Ditto introduces a global-regularized multi-task learning approach for federated learning that personalizes per-device models while retaining global objectives, yielding improved accuracy, robustness to data/model poisoning, and fairness across devices.

ABSTRACT

Fairness and robustness are two important concerns for federated learning systems. In this work, we identify that robustness to data and model poisoning attacks and fairness, measured as the uniformity of performance across devices, are competing constraints in statistically heterogeneous networks. To address these constraints, we propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits, and develop a scalable solver for it. Theoretically, we analyze the ability of Ditto to achieve fairness and robustness simultaneously on a class of linear problems. Empirically, across a suite of federated datasets, we show that Ditto not only achieves competitive performance relative to recent personalization methods, but also enables more accurate, robust, and fair models relative to state-of-the-art fair or robust baselines.

Motivation & Objective

• Address fairness and robustness trade-offs in statistically heterogeneous federated networks.
• Propose a simple yet scalable personalization framework (Ditto) that regularizes personalized models toward a global model.
• Provide convergence guarantees and show Ditto matches or surpasses state-of-the-art baselines on accuracy, robustness, and fairness.
• Analyze how personalization can inherently improve robustness and fairness in FL across diverse datasets and attacks.

Proposed method

• Formulate Ditto as a bi-level objective: for each device k, minimize F_k(v_k) + (lambda/2) ||v_k - w^*||^2, where w^* is the optimal global model for the aggregated objective G(F_1(w), ..., F_K(w)).
• Use an alternating optimization algorithm (Algorithm 1) where the server updates the global model w^t via standard FL solvers, and each device updates its personalized model v_k by inexactly solving min_v_k F_k(v_k) + (lambda/2)||v_k - w^t||^2 with local updates.
• Maintain modularity by allowing any G(·) solver (e.g., FedAvg) to be used for the global objective, preserving privacy and communication properties.
• Provide convergence guarantees: if F_k are strongly convex and smooth and w^t converges to w^*, then v_k^t converges to v_k^* with a related rate (Theorem 1).
• Analyze Ditto’s fairness/robustness trade-offs on a class of linear problems, deriving optimal lambda* and showing Pareto benefits.

Experimental results

Research questions

• RQ1 Can personalized federated learning via global-regularized multi-task learning improve accuracy, robustness, and fairness simultaneously in heterogeneous FL settings?
• RQ2 How does the Ditto regularization parameter lambda balance local personalization versus global consistency to enhance fairness and defend against training-time attacks?
• RQ3 What convergence properties does the Ditto solver exhibit under common FL practices (limited device participation, local updates)?
• RQ4 Do the fairness and robustness benefits of Ditto generalize beyond convex settings to non-convex models and real-world FL benchmarks?

Key findings

• Ditto achieves competitive or superior accuracy compared with personalization methods and outperforms several robust or fair baselines on diverse FL datasets.
• Across attacks and datasets, Ditto improves test accuracy by approximately 6% (absolute) over the strongest robust baseline on average.
• Ditto reduces variance of test accuracy across devices by about 10% on average, indicating improved fairness.
• Ditto provides robustness without sacrificing much fairness, and in some cases improves both fairness and robustness over state-of-the-art baselines.
• A theoretical analysis on linear problems identifies an optimal lambda* that yields the most accurate, fair, and robust solution within Ditto’s space.