[Paper Review] Tilted Empirical Risk Minimization

Tian Li, Ahmad Beirami+2 more|arXiv (Cornell University)|Jul 2, 2020

Statistical Methods and Inference77 references4 citations

TL;DR

This paper introduces Tilted Empirical Risk Minimization (TERM), a unified framework that generalizes empirical risk minimization (ERM) using a hyperparameter 'tilt' to flexibly control the influence of individual losses. By adjusting the tilt, TERM enhances robustness to outliers, improves fairness across subgroups, reduces variance for better generalization, and handles class imbalance—outperforming or matching specialized methods across diverse machine learning applications with a single, adaptable objective.

ABSTRACT

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly. While many methods aim to address these problems individually, in this work, we explore them through a unified framework -- tilted empirical risk minimization (TERM). In particular, we show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to ERM using a hyperparameter called the tilt. We provide several interpretations of the resulting framework: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to a superquantile method. We develop batch and stochastic first-order optimization methods for solving TERM, and show that the problem can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. TERM is not only competitive with existing solutions tailored to these individual problems, but can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.

Motivation & Objective

• To address limitations of standard ERM, which optimizes average performance and can be sensitive to outliers, unfair subgroup treatment, or poor generalization.
• To unify disparate challenges—outlier robustness, fairness, class imbalance, and variance reduction—under a single, flexible optimization framework.
• To demonstrate that TERM can simultaneously address multiple issues (e.g., fairness and robustness) where existing methods are limited to one.
• To provide theoretical and empirical justification for TERM’s effectiveness across diverse machine learning applications.

Proposed method

• TERM extends ERM using a tilt hyperparameter t, defining the objective as R̃(t;θ) = (1/t) log( (1/N) Σ e^{t f(xi;θ)} ), which generalizes ERM at t=0.
• For t < 0, TERM suppresses large losses, reducing outlier influence and improving robustness; for t > 0, it amplifies large losses, promoting min-max or variance-reduction behavior.
• The framework is interpreted as a smooth approximation to superquantile (conditional value-at-risk) methods, linking it to robust optimization.
• Batch and stochastic first-order optimization methods are developed to efficiently solve TERM, enabling practical deployment.
• The tilt parameter t is tuned adaptively across experiments, with negative values used for robustness and fairness, and positive values for variance reduction or imbalance handling.
• TERM is applied across multiple tasks: robust regression, fair PCA, imbalanced classification, and variance reduction, using consistent optimization and hyperparameter tuning.

Experimental results

Research questions

• RQ1 Can a single, unified framework effectively replace multiple specialized methods for handling outliers, fairness, and class imbalance in machine learning?
• RQ2 How does the tilt hyperparameter t control the trade-off between average performance and worst-case or subgroup-specific performance?
• RQ3 To what extent can TERM improve generalization through variance reduction compared to standard ERM?
• RQ4 Can TERM simultaneously address multiple challenges, such as fairness and robustness, where existing methods are limited?
• RQ5 How does TERM compare empirically to state-of-the-art methods in robust learning, fairness, and imbalanced classification?

Key findings

• TERM with t = -2 effectively suppresses outliers in regression and classification, outperforming Huber loss and CRR under high noise levels (e.g., 80% corruption).
• TERM achieves competitive or superior performance to specialized methods in fair PCA, reducing subgroup disparity while maintaining model accuracy.
• In imbalanced classification, TERM with adaptive t-tuning matched or exceeded performance of focal loss and LearnReweight, especially under high noise and imbalance.
• TERM with positive t values improved generalization by reducing variance, as shown in linear SVM experiments with tuned decision thresholds.
• The framework successfully addressed multiple challenges simultaneously—e.g., fairness and robustness—demonstrating its flexibility beyond single-purpose solutions.
• Empirical results show that TERM is robust to hyperparameter choice and can be efficiently optimized using stochastic first-order methods.