[Paper Review] Supervised Learning with General Risk Functionals

Liu Leqi, Audrey Huang+2 more|arXiv (Cornell University)|Jun 27, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper presents the first uniform convergence guarantees for estimating the cumulative distribution function (CDF) of loss distributions in supervised learning, enabling simultaneous generalization bounds across all Hölder risk functionals and all hypotheses. It introduces a novel gradient-based optimization method for distortion risks—subsuming mean, CVaR, and cumulative prospect theory risks—with theoretical convergence guarantees, validated on both tabular and deep learning settings.

ABSTRACT

Standard uniform convergence results bound the generalization gap of the expected loss over a hypothesis class. The emergence of risk-sensitive learning requires generalization guarantees for functionals of the loss distribution beyond the expectation. While prior works specialize in uniform convergence of particular functionals, our work provides uniform convergence for a general class of Hölder risk functionals for which the closeness in the Cumulative Distribution Function (CDF) entails closeness in risk. We establish the first uniform convergence results for estimating the CDF of the loss distribution, yielding guarantees that hold simultaneously both over all Hölder risk functionals and over all hypotheses. Thus licensed to perform empirical risk minimization, we develop practical gradient-based methods for minimizing distortion risks (widely studied subset of Hölder risks that subsumes the spectral risks, including the mean, conditional value at risk, cumulative prospect theory risks, and others) and provide convergence guarantees. In experiments, we demonstrate the efficacy of our learning procedure, both in settings where uniform convergence results hold and in high-dimensional settings with deep networks.

Motivation & Objective

• To address the lack of generalization guarantees for risk-sensitive learning beyond expected loss.
• To establish uniform convergence for CDF estimation over a hypothesis class, enabling simultaneous bounds across all Hölder risk functionals.
• To develop a practical, gradient-based optimization method for distortion risks—widely used in risk-sensitive learning—alongside theoretical convergence guarantees.
• To validate the method empirically in both low-dimensional and high-dimensional deep learning settings.

Proposed method

• Proposes a new complexity measure, permutation complexity, tailored for CDF estimation in hypothesis classes.
• Establishes uniform convergence of the empirical CDF to the true CDF using Rademacher complexity and VC dimension, extended via permutation complexity.
• Introduces a general framework for Hölder risk functionals, where closeness in CDF implies closeness in risk, enabling unified generalization bounds.
• Develops an empirical distortion risk minimization (EDRM) algorithm that reweights losses based on CDF values for gradient-based optimization.
• Derives convergence guarantees for the EDRM algorithm under smoothness and Lipschitz assumptions on the risk functional.
• Applies the method to distortion risks such as CVaR, spectral risks, and cumulative prospect theory, with implementation in deep networks.

Experimental results

Research questions

• RQ1 Can uniform convergence be established for CDF estimation across a hypothesis class, ensuring generalization for all Hölder risk functionals simultaneously?
• RQ2 How can risk-sensitive objectives like CVaR or spectral risks be optimized efficiently using gradient-based methods with theoretical guarantees?
• RQ3 What new complexity measures are necessary and sufficient for uniform CDF estimation in supervised learning?
• RQ4 To what extent do the proposed generalization and optimization methods hold in high-dimensional deep learning settings?
• RQ5 How do the proposed methods compare to standard ERM in terms of robustness to distribution shifts, noisy labels, and class imbalance?

Key findings

• The paper establishes the first uniform convergence result for CDF estimation, with bounds derived from Rademacher complexity, VC dimension, and a novel permutation complexity measure.
• Generalization guarantees hold simultaneously for all Hölder risk functionals and all hypotheses in the model class, enabling risk-agnostic generalization.
• The proposed EDRM algorithm achieves convergence to a stationary point under standard smoothness and Lipschitz conditions, with a convergence rate of O(1/√T).
• Empirical results show that the method improves robustness in settings with class imbalance and noisy labels, particularly for CVaR and spectral risk objectives.
• In ImageNet experiments, models optimized via the proposed method show improved worst-case performance (e.g., lower CVaR) across varying α levels, indicating better tail-risk control.