[Paper Review] Optimizing Black-box Metrics with Adaptive Surrogates

Qijia Jiang, Olaoluwa Adigun+3 more|arXiv (Cornell University)|Feb 20, 2020

Machine Learning and Algorithms31 references4 citations

TL;DR

This paper proposes a method to optimize black-box machine learning metrics—such as F-measure or Precision@K—by modeling them as monotonic functions of a small set of differentiable surrogate losses. Using finite-difference and local linear interpolation to estimate gradients in the surrogate space, the approach performs inexact projected gradient descent, achieving convergence under smoothness assumptions and matching performance of known-metric methods even when the metric's mathematical form is unknown.

ABSTRACT

We address the problem of training models with black-box and hard-to-optimize metrics by expressing the metric as a monotonic function of a small number of easy-to-optimize surrogates. We pose the training problem as an optimization over a relaxed surrogate space, which we solve by estimating local gradients for the metric and performing inexact convex projections. We analyze gradient estimates based on finite differences and local linear interpolations, and show convergence of our approach under smoothness assumptions with respect to the surrogates. Experimental results on classification and ranking problems verify the proposal performs on par with methods that know the mathematical formulation, and adds notable value when the form of the metric is unknown.

Motivation & Objective

• To address the challenge of optimizing non-decomposable, hard-to-optimize evaluation metrics in machine learning when no closed-form gradient is available.
• To enable training models with black-box metrics by expressing them as monotonic functions of a small number of easy-to-optimize surrogate losses.
• To develop a gradient-based optimization framework that works without requiring the mathematical form of the metric, leveraging a small validation set with true labels.
• To ensure convergence to a stationary point under smoothness assumptions on the unknown metric function.
• To empirically validate that the method performs as well as metric-aware baselines and offers gains when the metric is truly unknown.

Proposed method

• The method formulates the optimization problem in a K-dimensional surrogate space, where K << d, with d being the number of model parameters.
• It models the black-box metric M as an unknown monotonic function ψ: ℝ₊ᴷ → ℝ of K surrogate losses ℓ(θ).
• Gradients of ψ with respect to the surrogates are estimated using finite differences and local linear interpolation over perturbations of the model parameters.
• Projected gradient descent is performed in the surrogate space using inexact projections, which are computed by solving a convex subproblem in the original parameter space.
• The algorithm uses adaptive step-sizes and perturbation-based gradient estimation, with convergence guarantees under smoothness assumptions on ψ.
• The approach is applied to classification, ranking, and proxy label problems, using a small validation set to guide metric estimation.

Experimental results

Research questions

• RQ1 Can we optimize black-box metrics in machine learning when the metric’s mathematical form is unknown or non-differentiable?
• RQ2 How can we effectively estimate gradients for a black-box metric without access to its analytical derivative?
• RQ3 Can we achieve convergence to a stationary point in the optimization of a black-box metric using surrogate-based gradient estimation?
• RQ4 Does the proposed method perform competitively with metric-aware training approaches when the metric’s form is known?
• RQ5 Can the method provide significant gains in performance when the true metric is a black-box and no prior formulation is available?

Key findings

• The proposed method achieves test G-mean of 0.803 on the F-measure task, comparable to the best baseline methods, even when the metric is unknown.
• With only 10 perturbations, the method achieved a macro F-measure of 0.796 on the Business dataset, close to the 0.796 obtained with 1000 perturbations.
• For the KDD Cup 2008 ranking task, the method used minibatches of size 100 and a fixed σ = 1.5, achieving stable performance across runs.
• The gradient estimation error from local linear interpolation showed only mild dependence on K, with mean squared error increasing slowly as K increased from 2 to 10.
• The method outperformed baselines in proxy label learning scenarios where training labels were noisy and the true metric was unknown.
• Theoretical analysis shows convergence to a stationary point under smoothness assumptions on the unknown function ψ mapping surrogates to the metric.