[Paper Review] Online Learning via the Differential Privacy Lens

Jacob Abernethy, Young Hun Jung+3 more|arXiv (Cornell University)|Nov 27, 2017

Advanced Bandit Algorithms Research40 references4 citations

TL;DR

This paper introduces a differential privacy-inspired stability framework for online learning, using one-step differential stability to derive first-order regret bounds for follow-the-perturbed-leader (FTPL) algorithms in both full and partial information settings. It introduces Tsallis max-divergences for tighter stability control, achieving optimal or near-optimal regret bounds across online convex optimization, online linear optimization, multi-armed bandits, and bandits with experts—offering a unified, privacy-motivated approach to algorithm design with strong theoretical guarantees.

ABSTRACT

In this paper, we use differential privacy as a lens to examine online learning in both full and partial information settings. The differential privacy framework is, at heart, less about privacy and more about algorithmic stability, and thus has found application in domains well beyond those where information security is central. Here we develop an algorithmic property called one-step differential stability which facilitates a more refined regret analysis for online learning methods. We show that tools from the differential privacy literature can yield regret bounds for many interesting online learning problems including online convex optimization and online linear optimization. Our stability notion is particularly well-suited for deriving first-order regret bounds for follow-the-perturbed-leader algorithms, something that all previous analyses have struggled to achieve. We also generalize the standard max-divergence to obtain a broader class called Tsallis max-divergences. These define stronger notions of stability that are useful in deriving bounds in partial information settings such as multi-armed bandits and bandits with experts.

Motivation & Objective

• To develop a stability-based framework for online learning using differential privacy principles, focusing on algorithmic stability rather than privacy per se.
• To address the longstanding challenge of deriving first-order regret bounds for perturbation-based algorithms like FTPL, which previous analyses could not achieve.
• To unify the analysis of online learning problems across full-information (e.g., OCO, OLO) and partial-information (e.g., bandits) settings using a single stability framework.
• To introduce Tsallis γ-max-divergences as a stronger stability notion, especially useful in partial information settings with high-variance loss estimates.
• To demonstrate that DP-inspired stability yields optimal or near-optimal regret bounds across diverse online learning problems, including novel results for OLO and bandits with experts.

Proposed method

• Introduces one-step differential stability as a new algorithmic stability property that enables tighter regret analysis for online learning algorithms.
• Applies the objective perturbation method from differential privacy to design new FTPL-based algorithms for OCO and OLO, achieving first-order regret bounds.
• Proposes a novel family of Tsallis γ-max-divergences to strengthen stability control, particularly effective in partial information settings with large loss estimates.
• Uses a generalized Bayesian posterior approach (GBPA) to model randomized online learning strategies, linking them to differential stability and regret via duality.
• Derives regret bounds through a decomposition of expected regret into stability and range terms, leveraging concentration and boundedness of perturbations.
• Tunes parameters such as privacy budget ϵ and clipping threshold ρ to balance bias and variance, achieving optimal trade-offs in regret bounds.

Experimental results

Research questions

• RQ1 Can differential privacy-inspired stability be used to derive first-order regret bounds for FTPL algorithms, which previous analyses failed to achieve?
• RQ2 How can one-step differential stability be formalized and leveraged to unify regret analysis across full and partial information online learning problems?
• RQ3 What is the role of Tsallis γ-max-divergences in improving stability and tightening regret bounds in partial information settings like multi-armed bandits?
• RQ4 Can the proposed framework yield optimal or near-optimal regret bounds for online linear optimization (OLO), a problem where first-order bounds were previously unknown?
• RQ5 How do different perturbation and regularization schemes in bandits with experts map to specific stability levels, and can they achieve the same bounds as EXP4?

Key findings

• The paper establishes the first known first-order regret bound for online linear optimization (OLO) using a perturbation-based algorithm, achieving O(√(T L*) ) regret, where L* is the optimal cumulative loss.
• For online convex optimization (OCO), the proposed objective perturbation-based FTPL algorithm achieves a first-order regret bound of O(√(T L*)), matching known optimal bounds but derived through a novel stability lens.
• The introduction of Tsallis γ-max-divergences enables tighter stability control than standard max-divergence, particularly beneficial in partial information settings with heavy-tailed or unbounded loss estimates.
• The framework unifies the analysis of multi-armed bandit algorithms, showing that various perturbations and regularizers induce specific levels of differential stability, leading to clean regret bounds.
• For bandits with experts, the proposed perturbation-based algorithm achieves the same optimal zero-order and first-order regret bounds as the EXP4 algorithm, demonstrating the framework's generality and strength.
• The analysis reveals that clipping strategies in loss estimation can be tuned via ρ and ϵ to balance bias and variance, yielding a regret bound of O(K^{1/3}(log N)^{2/3}(L*)^{2/3}) when L* > 128K log N, which is optimal up to logarithmic factors.