[Paper Review] Local Differentially Private Regret Minimization in Reinforcement Learning.

Evrard Garcelon, Vianney Perchet+2 more|arXiv (Cornell University)|Oct 15, 2020

Privacy-Preserving Technologies in Data26 references4 citations

TL;DR

This paper proposes a local differentially private (LDP) reinforcement learning algorithm for finite-horizon Markov Decision Processes that achieves sublinear regret while protecting user data. By leveraging LDP on the user side, the method ensures privacy with a multiplicative regret cost compared to non-private settings.

ABSTRACT

Reinforcement learning algorithms are widely used in domains where it is desirable to provide a personalized service. In these domains it is common that user data contains sensitive information that needs to be protected from third parties. Motivated by this, we study privacy in the context of finite-horizon Markov Decision Processes (MDPs) by requiring information to be obfuscated on the user side. We formulate this notion of privacy for RL by leveraging the local differential privacy (LDP) framework. We present an optimistic algorithm that simultaneously satisfies LDP requirements, and achieves sublinear regret. We also establish a lower bound for regret minimization in finite-horizon MDPs with LDP guarantees. These results show that while LDP is appealing in practical applications, the setting is inherently more complex. In particular, our results demonstrate that the cost of privacy is multiplicative when compared to non-private settings.

Motivation & Objective

• To address privacy concerns in personalized reinforcement learning where user data contains sensitive information.
• To formalize privacy in RL using the local differential privacy (LDP) framework, ensuring obfuscation at the user level.
• To design an optimistic algorithm that satisfies LDP constraints while achieving sublinear regret.
• To establish a theoretical lower bound on regret for LDP-protected RL in finite-horizon MDPs.
• To quantify the inherent cost of privacy in terms of regret growth compared to non-private settings.

Proposed method

• Formulates privacy in finite-horizon MDPs using the local differential privacy (LDP) framework, requiring user-side data obfuscation before transmission.
• Designs an optimistic reinforcement learning algorithm that incorporates LDP constraints into the learning process, ensuring privacy at the data collection stage.
• Introduces a mechanism to add calibrated noise to user observations at the client side to satisfy LDP guarantees.
• Employs an optimistic value iteration or similar planning component to balance exploration and exploitation under privacy constraints.
• Analyzes regret bounds under LDP, deriving both an upper bound for the proposed algorithm and a lower bound for the problem class.
• Uses theoretical analysis to compare the regret scaling under LDP to the non-private case, revealing a multiplicative cost.

Experimental results

Research questions

• RQ1 Can we design a reinforcement learning algorithm that ensures local differential privacy while maintaining sublinear regret in finite-horizon MDPs?
• RQ2 What is the fundamental trade-off between privacy and regret performance in LDP-protected RL?
• RQ3 How does the regret of LDP-protected RL scale compared to non-private RL settings?
• RQ4 Can we establish a theoretical lower bound on regret for LDP-protected finite-horizon MDPs?
• RQ5 What is the multiplicative cost of privacy in terms of regret growth in this setting?

Key findings

• The proposed algorithm achieves sublinear regret under local differential privacy constraints, demonstrating that private RL is feasible in finite-horizon MDPs.
• A theoretical lower bound on regret is established for LDP-protected finite-horizon MDPs, showing inherent limitations due to privacy.
• The cost of privacy is multiplicative compared to non-private settings, meaning regret scales with a factor that grows with the privacy budget.
• The analysis confirms that LDP imposes a significant but quantifiable performance penalty in terms of regret growth.
• The results show that while LDP is practical for real-world deployment, it fundamentally increases the complexity of regret minimization in RL.