[Paper Review] Locally Private Distributed Reinforcement Learning

Hajime Ono, Tsubasa Takahashi|arXiv (Cornell University)|Jan 31, 2020

Privacy-Preserving Technologies in Data27 references19 citations

TL;DR

This paper proposes Private Gradient Collection (PGC), a novel framework for locally differentially private distributed reinforcement learning (DRL) that enables robust policy learning across private environments. By injecting noise into gradients via Laplace or PRS mechanisms to satisfy local differential privacy (LDP), PGC allows agents to contribute updates without exposing private environment details, achieving strong privacy guarantees while maintaining effective learning performance under LDP constraints.

ABSTRACT

We study locally differentially private algorithms for reinforcement learning to obtain a robust policy that performs well across distributed private environments. Our algorithm protects the information of local agents' models from being exploited by adversarial reverse engineering. Since a local policy is strongly being affected by the individual environment, the output of the agent may release the private information unconsciously. In our proposed algorithm, local agents update the model in their environments and report noisy gradients designed to satisfy local differential privacy (LDP) that gives a rigorous local privacy guarantee. By utilizing a set of reported noisy gradients, a central aggregator updates its model and delivers it to different local agents. In our empirical evaluation, we demonstrate how our method performs well under LDP. To the best of our knowledge, this is the first work that actualizes distributed reinforcement learning under LDP. This work enables us to obtain a robust agent that performs well across distributed private environments.

Motivation & Objective

• To address privacy leakage in distributed reinforcement learning where local policies may inadvertently expose private environment information.
• To design a DRL framework that ensures local differential privacy (LDP) for gradients reported by agents in private environments.
• To enable robust policy learning across diverse private environments without compromising agent-level privacy.
• To evaluate the trade-off between privacy (controlled by ε) and learning efficiency in a practical DRL setting.

Proposed method

• Proposes the Private Gradient Collection (PGC) framework, where local agents train models in private environments and report LDP-protected noisy gradients to a central aggregator.
• Employs two randomized mechanisms—Laplace and PRS (Poisson Rejection Sampling)—to inject noise into gradients, ensuring local differential privacy.
• Uses asynchronous advantage actor-critic (A3C) as the base algorithm, modifying gradient reporting to preserve privacy while maintaining learning stability.
• Applies a buffering mechanism in PRS to improve gradient stability, reducing noise-induced training instability.
• The central aggregator updates the global model using the reported noisy gradients, enabling robust policy learning across environments.
• Employs a privacy budget ε to control the level of differential privacy, with lower ε indicating stronger privacy.

Experimental results

Research questions

• RQ1 Can distributed reinforcement learning be made robust to privacy leakage from local policies trained in private environments?
• RQ2 How can local differential privacy (LDP) be effectively enforced on gradients in a DRL setting without requiring raw data transfer?
• RQ3 What is the impact of different noise injection mechanisms (Laplace vs. PRS) on learning efficiency and privacy-utility trade-offs in DRL?
• RQ4 How does the choice of privacy budget ε affect the success rate and convergence speed of the learning process under LDP?

Key findings

• PGC-A3C with the PRS mechanism achieves a higher success ratio (0.95) at ε=2 compared to Laplace (0.90), indicating better early learning performance under strong privacy constraints.
• The Laplace mechanism achieves a relative AUC of 0.965 at ε=10, approaching the non-private baseline (AUC=1.0), indicating strong learning efficiency under relaxed privacy.
• The PRS mechanism achieves the best relative AUC (0.862) at ε=2, outperforming Laplace at this privacy level, suggesting it is more effective for moderate privacy requirements.
• With ε=1, the PRS mechanism achieves a success ratio of 0.85, while Laplace achieves 0.80, showing PRS's advantage in low-privacy regimes.
• The PRS mechanism with |B|=100 buffering improves training stability, enabling better learning performance than without buffering, especially under high noise.
• The FST (first success time) median for PRS at ε=2 is 7,549, significantly lower than Laplace’s 20,238.5, indicating faster convergence under this setting.