[Paper Review] Second-order Quantile Methods for Experts and Combinatorial Games

Wouter M. Koolen, Tim van Erven|arXiv (Cornell University)|Feb 27, 2015

Advanced Bandit Algorithms Research30 references20 citations

TL;DR

This paper introduces Squint and Component iProd, novel online learning algorithms that combine second-order regret bounds (dependent on variance) with quantile bounds (dependent on prior knowledge of expert quality), using a hierarchical prior over learning rates. The key contribution is the first regret bounds that simultaneously achieve both improvements, with efficient implementations for uniform and log-uniform priors.

ABSTRACT

We aim to design strategies for sequential decision making that adjust to the difficulty of the learning problem. We study this question both in the setting of prediction with expert advice, and for more general combinatorial decision tasks. We are not satisfied with just guaranteeing minimax regret rates, but we want our algorithms to perform significantly better on easy data. Two popular ways to formalize such adaptivity are second-order regret bounds and quantile bounds. The underlying notions of 'easy data', which may be paraphrased as "the learning problem has small variance" and "multiple decisions are useful", are synergetic. But even though there are sophisticated algorithms that exploit one of the two, no existing algorithm is able to adapt to both. In this paper we outline a new method for obtaining such adaptive algorithms, based on a potential function that aggregates a range of learning rates (which are essential tuning parameters). By choosing the right prior we construct efficient algorithms and show that they reap both benefits by proving the first bounds that are both second-order and incorporate quantiles.

Motivation & Objective

• To design adaptive online learning algorithms that perform significantly better on easy data, defined as data with low variance or multiple useful experts.
• To unify two complementary adaptive learning paradigms: second-order regret bounds (variance-based) and quantile bounds (prior-based expert selection).
• To develop a method that learns the optimal learning rate through a prior over learning rates, enabling simultaneous adaptation to both data variance and expert quality.

Proposed method

• Proposes a potential function that aggregates over a range of learning rates, using a prior distribution over the learning rate parameter η.
• Derives regret bounds by integrating over η using Bayesian mixtures, with the prior encoding assumptions about the optimal learning rate.
• Applies the method to the expert setting (Squint) and to combinatorial games (Component iProd), using a component-wise aggregation strategy.
• Uses exponential grid discretization for learning rates to balance computational cost and regret overhead.
• Employs uniform and log-uniform priors to achieve efficient algorithms with tight bounds, including Clr = ln ln T for the log-uniform case.
• Introduces a novel potential function that combines mix loss and variance terms, enabling second-order quantile bounds.

Experimental results

Research questions

• RQ1 Can an online learning algorithm simultaneously adapt to low-variance data and to the presence of multiple high-performing experts?
• RQ2 Is it possible to design a single algorithm that achieves both second-order regret bounds and quantile bounds?
• RQ3 How can the learning rate be adaptively learned in a way that improves performance on both easy and hard data regimes?
• RQ4 What prior distributions over learning rates yield efficient algorithms with optimal regret guarantees?
• RQ5 Can the method be generalized to combinatorial decision problems beyond expert advice?

Key findings

• Squint achieves a regret bound of the form RK_T ≺ √(V^K_T * (Clr - ln π(K))), where Clr = ln ln T for the log-uniform prior, combining variance and quantile benefits.
• The algorithm with the log-uniform prior achieves Clr = ln ln T, effectively treating ln ln T as constant, thus combining second-order and quantile adaptivity.
• For the uniform prior, the algorithm is efficient (K operations per round) and achieves Clr = ln V^K_T, still combining both adaptivity types.
• Component iProd extends the method to combinatorial games, achieving second-order quantile bounds with a regret bound of R^v_T ≤ (4/√3)√(V^v_T * (△²(v||π) + K ln⌈1 + log₂T⌉)) + 4△²(v||π) + K max{4 ln⌈1 + log₂T⌉, 1}.
• The method shows that learning the optimal learning rate via a prior over η enables both variance and quantile adaptation, with theoretical guarantees that hold in stochastic and bounded-loss settings.
• The approach is robust to loss normalization and can be extended to matrix-valued prediction and switching experts, suggesting broad applicability.