[Paper Review] MetaGrad: Multiple Learning Rates in Online Learning

Tim van Erven, Wouter M. Koolen|arXiv (Cornell University)|Apr 29, 2016

Advanced Bandit Algorithms Research39 references20 citations

TL;DR

MetaGrad is an adaptive online learning algorithm that dynamically combines multiple learning rates using a master algorithm based on empirical performance, enabling logarithmic regret for a broad class of functions—including exp-concave, strongly convex, and non-curvature functions like unregularized hinge loss—without prior tuning. Its key contribution is achieving fast rates across diverse function types through a non-monotonic, performance-weighted learning rate selection strategy.

ABSTRACT

In online convex optimization it is well known that certain subclasses of objective functions are much easier than arbitrary convex functions. We are interested in designing adaptive methods that can automatically get fast rates in as many such subclasses as possible, without any manual tuning. Previous adaptive methods are able to interpolate between strongly convex and general convex functions. We present a new method, MetaGrad, that adapts to a much broader class of functions, including exp-concave and strongly convex functions, but also various types of stochastic and non-stochastic functions without any curvature. For instance, MetaGrad can achieve logarithmic regret on the unregularized hinge loss, even though it has no curvature, if the data come from a favourable probability distribution. MetaGrad's main feature is that it simultaneously considers multiple learning rates. Unlike previous methods with provable regret guarantees, however, its learning rates are not monotonically decreasing over time and are not tuned based on a theoretically derived bound on the regret. Instead, they are weighted directly proportional to their empirical performance on the data using a tilted exponential weights master algorithm.

Motivation & Objective

• To develop an adaptive online learning method that automatically achieves fast regret rates across diverse function classes without manual tuning.
• To extend beyond previous adaptive methods that only interpolate between strongly convex and general convex functions.
• To handle functions without curvature, such as the unregularized hinge loss, under favorable data distributions.
• To design a learning rate selection mechanism that is not monotonically decreasing and not based on theoretical regret bounds.
• To provide a unified framework that simultaneously adapts to multiple function types, including stochastic and non-stochastic settings.

Proposed method

• MetaGrad maintains a master algorithm that combines multiple 'slaves', each using a different learning rate η.
• Each slave performs online gradient descent with its own learning rate η, tracking surrogate losses based on the gradient and parameter updates.
• The master algorithm uses tilted exponential weights to assign higher weights to learning rates that perform better empirically on the data.
• The learning rates are selected from an exponentially spaced grid with a heavy-tailed prior, enabling coverage of a wide range of rates.
• The full version maintains a d×d covariance matrix, while the diagonal version only tracks the diagonal elements for computational efficiency.
• The regret bound is derived by combining the master's regret with slave-specific surrogate regret bounds, leading to a composite bound that adapts to the data's structure.

Experimental results

Research questions

• RQ1 Can an online learning algorithm achieve logarithmic regret for unregularized hinge loss, which lacks curvature, under favorable data distributions?
• RQ2 Is it possible to design an adaptive method that simultaneously achieves fast rates across multiple function classes, including exp-concave, strongly convex, and non-curvature functions?
• RQ3 Can a learning rate selection strategy be designed that is not monotonically decreasing and does not rely on theoretical regret bounds?
• RQ4 What is the trade-off between adaptivity and computational cost when extending to broader function classes?
• RQ5 Can the regret bound be simultaneously tight for both general convex and easier function types without prior knowledge?

Key findings

• MetaGrad achieves a regret bound of O(√T ln ln T) and O(√(V^u_T d ln T) + d ln T) for any u ∈ U, where V^u_T measures the variance of parameter-distance-weighted gradients.
• For favorable data distributions, MetaGrad achieves logarithmic regret O(ln T) on the unregularized hinge loss, despite the absence of curvature.
• The diagonal version of MetaGrad achieves O(d ln(D²G²T)) regret, matching the theoretical expectations for dimension dependence.
• The full version of MetaGrad achieves a regret bound that depends on the rank of the accumulated gradient matrix, enabling faster rates when the data exhibit low-dimensional structure.
• The method's performance is robust to unknown data distributions, as the regret bound holds for any u ∈ U and adapts to the data's intrinsic difficulty.
• The algorithm maintains strong theoretical guarantees without requiring prior knowledge of the function class, making it broadly applicable across online learning tasks.