[Paper Review] Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statistics

Kaan Gökcesu, Hakan Gökcesu|arXiv (Cornell University)|Aug 28, 2021

Statistical Methods and InferenceMathematics24 references33 citations

TL;DR

The paper generalizes the Huber loss using a log-exp transform to blend robustness of absolute loss with the fast convergence of quadratic loss and provides an algorithm to efficiently minimize it.

ABSTRACT

We propose a generalized formulation of the Huber loss. We show that with a suitable function of choice, specifically the log-exp transform; we can achieve a loss function which combines the desirable properties of both the absolute and the quadratic loss. We provide an algorithm to find the minimizer of such loss functions and show that finding a centralizing metric is not that much harder than the traditional mean and median.

Motivation & Objective

• Motivate robust learning by integrating robustness with fast convergence.
• Propose a generalized Huber loss that unifies various loss forms via a log-exp transform.
• Establish mathematical properties of the generalized loss, including smoothing and asymptotic behavior.
• Develop an efficient algorithm to minimize the generalized loss and analyze centralizing metrics.
• Discuss practical considerations and extensions for high-dimensional, multivariate settings.

Proposed method

• Define a generalized loss framework using a monotone auxiliary function f and its inverse g to create a smoothed absolute-like loss.
• Instantiate a specific smooth, strictly convex loss L_M(x) via the log-exp transform: L_M(x)= (1/a) log(e^{ax}+e^{-ax}+b).
• Prove convexity properties of the log-exp loss and its asymptotic behavior (L_M -> |x| at infinity; L_M -> quadratic near 0).
• Derive first and second derivatives of L_M to understand optimization landscape.
• Present Algorithm 1 for finding the centralizing sample pair minimizing a cumulative loss, and Algorithm 2 for achieving an ε-optimal solution with complexity analyses.
• Provide corollaries detailing convergence to absolute and quadratic losses and discuss conditions for convexity (b>=0).

Experimental results

Research questions

• RQ1 How can a loss function simultaneously achieve robustness to outliers and fast convergence like quadratics?
• RQ2 Can a generalized Huber-like loss be constructed via log-exp transforms to interpolate between |x| and x^2?
• RQ3 What are the convexity and smoothness properties of the proposed generalized loss, and how can they be leveraged for efficient minimization?
• RQ4 What algorithms efficiently compute centralizing estimators under the generalized loss, and what are their complexities?
• RQ5 How does the generalized loss behave asymptotically compared to traditional loss functions (absolute and quadratic)?

Key findings

• A generalized Huber loss using a log-exp transform yields a smooth, strictly convex loss with robust and fast-converging properties.
• The loss L_M(x)= (1/a) log(e^{ax}+e^{-ax}+b) converges to |x| as |x|→∞ and to a quadratic form near 0, ensuring robustness and fast local convergence.
• The paper proves convexity conditions (b≥0) and derives explicit expressions for derivatives, enabling optimization.
• An O(N log N) algorithm (Algorithm 1) finds the centralizing minimizer for a cumulative loss over N samples; an ε-optimal variant (Algorithm 2) has runtime O(N log(D/ε)).
• The framework encompasses and generalizes common losses (Huber, Pseudo-Huber, log-cosh) and yields a flexible approach to centralizing metrics beyond mean/median.