[Paper Review] New Potential-Based Bounds for Prediction with Expert Advice

Vladimir A. Kobzar, Robert V. Kohn+1 more|arXiv (Cornell University)|Nov 5, 2019

Advanced Bandit Algorithms Research24 references4 citations

TL;DR

This paper introduces new potential-based bounds for online prediction with expert advice by leveraging sub- and supersolutions of partial differential equations (PDEs) derived from optimal control theory. It provides tighter regret bounds for finite-horizon games with any number of experts, achieving optimal leading-order terms for two and three experts, and improves upon prior state-of-the-art bounds in identified regimes through closed-form PDE solutions.

ABSTRACT

This work addresses the classic machine learning problem of online prediction with expert advice. We consider the finite-horizon version of this zero-sum, two-person game. Using verification arguments from optimal control theory, we view the task of finding better lower and upper bounds on the value of the game (regret) as the problem of finding better sub- and supersolutions of certain partial differential equations (PDEs). These sub- and supersolutions serve as the potentials for player and adversary strategies, which lead to the corresponding bounds. To get explicit bounds, we use closed-form solutions of specific PDEs. Our bounds hold for any given number of experts and horizon; in certain regimes (which we identify) they improve upon the previous state of the art. For two and three experts, our bounds provide the optimal leading order term.

Motivation & Objective

• To develop tighter, non-asymptotic regret bounds for the finite-horizon expert problem using potential functions.
• To extend potential-based methods beyond upper bounds by also deriving lower bounds via PDE sub- and supersolutions.
• To identify regimes where the new bounds improve upon existing state-of-the-art results.
• To provide explicit, closed-form bounds for any number of experts and horizon using solutions to specific PDEs.
• To unify and generalize prior PDE-based approaches for optimal strategies in online learning.

Proposed method

• Formulate the expert problem as a zero-sum game and use verification arguments from optimal control to derive sub- and supersolutions of PDEs as potentials.
• Use the value function of the game as a solution to a Hamilton-Jacobi-Bellman PDE, and construct bounds via sub- and supersolutions.
• Apply closed-form solutions of linear and nonlinear PDEs (e.g., heat equation and its nonlinear analog) to generate explicit bounds.
• Derive upper bounds using the max potential and lower bounds using symmetric potentials, with error terms controlled via higher-order PDE derivatives.
• Use numerical integration and random walk approximations to compute and validate bounds for comparison.
• Establish that the bounds are tight up to the leading-order term for N=2 and N=3 experts.

Experimental results

Research questions

• RQ1 Can sub- and supersolutions of PDEs be systematically used to derive both upper and lower bounds on regret in the expert problem?
• RQ2 What classes of potential functions yield tighter non-asymptotic regret bounds for general N and finite T?
• RQ3 How do PDE-based bounds compare to known asymptotic and non-asymptotic results, especially for small N?
• RQ4 Can closed-form solutions of PDEs lead to explicit, computable regret bounds that improve upon prior work?
• RQ5 In which parameter regimes do the new bounds achieve the optimal leading-order term for N=2 and N=3?

Key findings

• For two experts, the new bounds achieve the optimal leading-order term in regret, matching the known non-asymptotic optimal strategy.
• For three experts, the bounds also achieve the optimal leading-order term, improving upon previous state-of-the-art bounds.
• The method provides explicit, closed-form regret bounds for any number of experts and finite horizon T, derived from solutions to specific PDEs.
• The upper bound error term is shown to be O(N log |t|) under appropriate PDE parameterization, improving over prior potential-based methods.
• The lower bound derived from the symmetric potential matches the known lower bound from the i.i.d. Gaussian regime, confirming tightness in the asymptotic limit.
• The framework unifies and extends prior PDE-based approaches, enabling both upper and lower bounds via sub- and supersolutions.