[Paper Review] Excess Risk Bounds for Exponentially Concave Losses

Mehrdad Mahdavi, Rong Jin|arXiv (Cornell University)|Jan 18, 2014

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper establishes high-probability excess risk bounds of $O(d\log n/n)$ for learning with exponentially concave loss functions in both batch and online settings. It achieves this by leveraging local Radamacher complexities in batch learning and a modified online Newton method with martingale Bernstein inequalities and peeling techniques in online learning, under mild assumptions on the loss function.

ABSTRACT

The overarching goal of this paper is to derive excess risk bounds for learning from exp-concave loss functions in passive and sequential learning settings. Exp-concave loss functions encompass several fundamental problems in machine learning such as squared loss in linear regression, logistic loss in classification, and negative logarithm loss in portfolio management. In batch setting, we obtain sharp bounds on the performance of empirical risk minimization performed in a linear hypothesis space and with respect to the exp-concave loss functions. We also extend the results to the online setting where the learner receives the training examples in a sequential manner. We propose an online learning algorithm that is a properly modified version of online Newton method to obtain sharp risk bounds. Under an additional mild assumption on the loss function, we show that in both settings we are able to achieve an excess risk bound of $O(d\log n/n)$ that holds with a high probability.

Motivation & Objective

• To derive sharp excess risk bounds for learning from exponentially concave loss functions in both batch and sequential settings.
• To extend fast rate generalization bounds beyond strong convexity to the broader class of exp-concave losses.
• To achieve high-probability excess risk bounds, improving upon existing expectation-based bounds in online learning.
• To analyze the performance of empirical risk minimization in linear hypothesis spaces under exp-concave losses.
• To explore the possibility of reducing dependence on dimension $d$ via sparsity in future work.

Proposed method

• Uses empirical risk minimization in the batch setting over a bounded linear hypothesis space $\mathscr{W}$ with $\|\mathbf{w}\| \leq R$.
• Applies local Radamacher complexity analysis tailored to exp-concave functions to derive sharp convergence rates in the batch case.
• Proposes a modified online Newton method with adaptive regularization and Hessian-based updates to control regret and excess risk.
• Employs Bernstein inequality for martingales and a peeling process to handle the stochastic nature of online updates and control error accumulation.
• Introduces a novel decomposition of the excess risk involving terms related to Hessian approximation error, gradient noise, and curvature deviation.
• Uses a dual analysis framework combining matrix trace inequalities and determinant bounds to control the growth of the Hessian inverse in online updates.

Experimental results

Research questions

• RQ1 Can sharp $O(d\log n/n)$ excess risk bounds be achieved in high probability for exp-concave losses in the batch learning setting?
• RQ2 Can the online Newton method be modified to achieve high-probability excess risk bounds of $O(d\log n/n)$ rather than just expectation-based bounds?
• RQ3 What additional assumptions on the loss function are necessary to achieve such fast rates in both settings?
• RQ4 Is the $\log n$ factor in the excess risk bound unavoidable, or can it be removed with refined analysis?
• RQ5 Can the dependence on dimension $d$ be reduced to $s\log d$ for sparse solutions using techniques from sparse recovery?

Key findings

• The paper establishes a high-probability excess risk bound of $O(d\log n/n)$ for empirical risk minimization in the batch setting under exp-concave losses.
• In the online setting, the modified online Newton method achieves the same $O(d\log n/n)$ excess risk bound with high probability.
• The analysis relies on a peeling process and Bernstein inequality for martingales to control the deviation of stochastic gradients.
• The bound holds under a mild additional assumption on the loss function, beyond exp-concavity and Lipschitz continuity.
• The derived bound is significantly faster than the standard $O(1/\sqrt{n})$ rate for general convex Lipschitz losses.
• The paper identifies $d\log n/n$ as a potential tight bound, with open questions on removing $\log n$ or reducing $d$ to $s\log d$ for sparse solutions.