[Paper Review] Privacy-Preserving Contextual Bandits.

Awni Hannun, Brian Knott+2 more|arXiv (Cornell University)|Oct 11, 2019

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes a privacy-preserving contextual bandit algorithm that enables multiple parties to collaboratively train a model without sharing sensitive features, rewards, or actions. By combining secure multi-party computation with an epsilon-greedy exploration mechanism, the method ensures differential privacy while maximizing cumulative reward in a distributed setting.

ABSTRACT

Contextual bandits are online learners that, given an input, select an arm and receive a reward for that arm. They use the reward as a learning signal and aim to maximize the total reward over the inputs. Contextual bandits are commonly used to solve recommendation or ranking problems. This paper considers a learning setting in which multiple parties aim to train a contextual bandit together in a private way: the parties aim to maximize the total reward but do not want to share any of the relevant information they possess with the other parties. Specifically, multiple parties have access to (different) features that may benefit the learner but that cannot be shared with other parties. One of the parties pulls the arm but other parties may not learn which arm was pulled. One party receives the reward but the other parties may not learn the reward value. This paper develops a privacy-preserving contextual bandit algorithm that combines secure multi-party computation with a differential private mechanism based on epsilon-greedy exploration in contextual bandits.

Motivation & Objective

• To enable multiple parties to collaboratively train a contextual bandit model without exposing private features, actions, or rewards.
• To preserve differential privacy during exploration in a multi-party setting where data cannot be shared.
• To ensure that no single party learns which arm was pulled or the reward value, even when they contribute features.
• To maintain high learning performance (total reward) while enforcing strict privacy constraints across distributed parties.
• To integrate secure multi-party computation with differential privacy in a contextual bandit framework for real-world recommendation and ranking applications.

Proposed method

• The algorithm uses secure multi-party computation (MPC) to compute model updates without revealing inputs or outputs across parties.
• Each party contributes features to the model, but no party learns the features of others or the selected arm.
• An epsilon-greedy exploration strategy is applied, with exploration decisions perturbed using a differentially private mechanism to protect action selection.
• The reward is collected by one party but is not disclosed to others; only a differentially private signal is used for model updates.
• The method ensures that the model learns from all parties' data while satisfying (epsilon, delta)-differential privacy guarantees.
• The system is designed to scale to multiple parties with heterogeneous, non-shareable feature sets, maintaining privacy and utility.

Experimental results

Research questions

• RQ1 How can multiple parties train a contextual bandit model collaboratively without sharing private features?
• RQ2 What mechanisms ensure that no party learns which arm was pulled or the reward value?
• RQ3 Can differential privacy be effectively combined with secure multi-party computation in a contextual bandit setting?
• RQ4 How does the privacy-preserving mechanism affect the cumulative reward compared to non-private baselines?
• RQ5 What is the trade-off between privacy (epsilon) and learning performance in this multi-party setting?

Key findings

• The proposed method achieves high cumulative reward while preserving differential privacy across all parties.
• The use of secure multi-party computation prevents any party from learning sensitive information about others’ features, actions, or rewards.
• The integration of epsilon-greedy exploration with differential privacy ensures that exploration decisions are private and statistically protected.
• The algorithm maintains model utility even when features are non-shareable and parties are untrusted.
• The system scales to multiple parties with heterogeneous data, ensuring privacy without sacrificing learning performance.
• Empirical results confirm that the method provides strong privacy guarantees while achieving competitive learning outcomes.