[Paper Review] From Stochastic Mixability to Fast Rates

Nishant A. Mehta, Robert C. Williamson|ANU Open Research (Australian National University)|Jun 14, 2014

Machine Learning and Algorithms24 references3 citations

TL;DR

This paper establishes that stochastic mixability—a bridge between statistical and online learning—implies fast $O(1/n)$ convergence rates for empirical risk minimization (ERM) in both finite and VC-type function classes. Using a direct proof based on the Cramér-Chernoff method and Kemperman's solution to the general moment problem, it shows that stochastic mixability yields exact oracle inequalities with leading constant 1, offering new insight into the geometry of fast rates and suggesting a characterization of effective convexity in learning problems.

ABSTRACT

Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution $\mathsf{P}$ and returns a hypothesis $f$ chosen from a fixed class $\mathcal{F}$ with small loss $\ell$. In the parametric setting, depending upon $(\ell, \mathcal{F},\mathsf{P})$ ERM can have slow $(1/\sqrt{n})$ or fast $(1/n)$ rates of convergence of the excess risk as a function of the sample size $n$. There exist several results that give sufficient conditions for fast rates in terms of joint properties of $\ell$, $\mathcal{F}$, and $\mathsf{P}$, such as the margin condition and the Bernstein condition. In the non-statistical prediction with expert advice setting, there is an analogous slow and fast rate phenomenon, and it is entirely characterized in terms of the mixability of the loss $\ell$ (there being no role there for $\mathcal{F}$ or $\mathsf{P}$). The notion of stochastic mixability builds a bridge between these two models of learning, reducing to classical mixability in a special case. The present paper presents a direct proof of fast rates for ERM in terms of stochastic mixability of $(\ell,\mathcal{F}, \mathsf{P})$, and in so doing provides new insight into the fast-rates phenomenon. The proof exploits an old result of Kemperman on the solution to the general moment problem. We also show a partial converse that suggests a characterization of fast rates for ERM in terms of stochastic mixability is possible.

Motivation & Objective

• To establish a direct connection between stochastic mixability and fast convergence rates in statistical learning.
• To provide a new proof of fast rates for ERM using the Cramér-Chernoff method and Kemperman’s solution to the general moment problem.
• To show that stochastic mixability implies exact oracle inequalities with leading constant 1 for finite and VC-type classes.
• To investigate whether stochastic mixability characterizes the fast-rate phenomenon, including a partial converse suggesting non-unique minimizers as a sign of failure.
• To explore the relationship between stochastic mixability and the Bernstein condition, and to assess the possibility of equivalence under bounded loss assumptions.

Proposed method

• The authors use the Cramér-Chernoff method to control tail probabilities of the excess risk, leveraging the stochastic mixability condition.
• They apply Kemperman’s solution to the general moment problem to derive sharp bounds on the moment-generating function of the excess loss.
• The analysis is localized by restricting attention to functions with excess risk at least $ ( heta n)^{-1/(2- heta)} $, enabling extension from finite to VC-type classes.
• A key technical step involves bounding the optimal value of a moment problem instance under stochastic mixability, ensuring fast rate control.
• The proof structure allows separate control of large deviation probabilities for individual functions in the class, enabling generalization to VC-type classes via $$-nets.
• The approach is extended to nonparametric classes by considering polynomial metric entropy, though full extension remains an open problem.

Experimental results

Research questions

• RQ1 Does stochastic mixability imply fast $O(1/n)$ rates for ERM in finite function classes?
• RQ2 Can the fast rate result under stochastic mixability be extended to VC-type classes?
• RQ3 Is there a partial converse: does failure of stochastic mixability imply non-unique minimizers and slow rates?
• RQ4 How is stochastic mixability related to the Bernstein condition, and are they equivalent under bounded losses?
• RQ5 Can the bounded loss assumption be removed to extend the results to more general loss distributions?

Key findings

• Stochastic mixability implies an exact oracle inequality with leading constant 1 for finite classes, yielding a fast rate of $ O(1/n) $.
• For finite classes with $ N $ functions and bounded loss $ V $, the excess risk is bounded by $ \frac{6(\log(1/\delta) + \log N)}{(η_0 n)^{1/(2-\kappa)}} $ with high probability.
• The result extends to VC-type classes by localizing the analysis to functions with excess risk at least $ (η_0 n)^{-1/(2-\kappa)} $.
• A partial converse is established: if stochastic mixability fails, the risk minimizer is not unique in a way that leads to slow rates, indicating a geometric obstruction.
• The Bernstein condition implies stochastic mixability under bounded losses, suggesting a deeper connection between the two conditions.
• The worst-case random variables for stochastic mixability are those with low probability of large gains and high probability of small losses, which may require additional moment conditions to rule out.