[Paper Review] Convergence Properties of Stochastic Hypergradients

Riccardo Grazzi, Massimiliano Pontil+1 more|arXiv (Cornell University)|Nov 13, 2020

Advanced Topology and Set Theory19 references6 citations

TL;DR

This paper proposes a stochastic inexact descent (SID) method for computing hypergradients in bilevel optimization via approximate implicit differentiation (AID), replacing deterministic solvers with stochastic ones. It provides a mean square error bound for the hypergradient estimator that is agnostic to the choice of stochastic solvers, proving convergence under contraction assumptions and demonstrating improved accuracy in practice, especially with decreasing step sizes.

ABSTRACT

Bilevel optimization problems are receiving increasing attention in machine learning as they provide a natural framework for hyperparameter optimization and meta-learning. A key step to tackle these problems is the efficient computation of the gradient of the upper-level objective (hypergradient). In this work, we study stochastic approximation schemes for the hypergradient, which are important when the lower-level problem is empirical risk minimization on a large dataset. The method that we propose is a stochastic variant of the approximate implicit differentiation approach in (Pedregosa, 2016). We provide bounds for the mean square error of the hypergradient approximation, under the assumption that the lower-level problem is accessible only through a stochastic mapping which is a contraction in expectation. In particular, our main bound is agnostic to the choice of the two stochastic solvers employed by the procedure. We provide numerical experiments to support our theoretical analysis and to show the advantage of using stochastic hypergradients in practice.

Motivation & Objective

• To develop a stochastic variant of approximate implicit differentiation (AID) for hypergradient computation in bilevel optimization.
• To provide a mean square error bound for the hypergradient estimator that is independent of the specific stochastic solvers used for the lower-level problem and linear system.
• To extend the convergence analysis of stochastic fixed-point iterations to handle bilevel problems with contraction mappings.
• To empirically validate the effectiveness of stochastic hypergradients in practice, especially with decreasing step sizes.
• To enable scalable hyperparameter optimization and meta-learning by replacing expensive deterministic solvers with stochastic alternatives.

Proposed method

• Proposes a stochastic inexact descent (SID) algorithm that uses unbiased stochastic approximations for both the lower-level problem and the linear system arising in AID.
• Employs two stochastic solvers: one for solving the lower-level fixed-point equation and another for solving the linear system in the hypergradient computation.
• Uses a contraction mapping assumption on the lower-level mapping Φ(w,λ) to ensure convergence of the stochastic fixed-point iterations.
• Derives a mean square error bound for the hypergradient estimator that depends on the contraction rate and the variance of the stochastic approximations, independent of the specific solvers used.
• Applies a general convergence analysis for stochastic fixed-point iterations, extending SGD convergence to strongly convex and contraction-based problems.
• Employs decreasing step sizes in the Stoch dec variant to ensure convergence to the true hypergradient, while constant stepsizes may fail to converge in expectation.

Experimental results

Research questions

• RQ1 Can a stochastic variant of approximate implicit differentiation (AID) achieve convergence to the true hypergradient in bilevel optimization?
• RQ2 What is the mean square error bound for the hypergradient estimator when both the lower-level problem and the linear system are solved stochastically?
• RQ3 How does the choice of stochastic solver (e.g., constant vs. decreasing step sizes) affect the convergence behavior of the hypergradient estimator?
• RQ4 Can the convergence analysis of stochastic fixed-point iterations be extended to the bilevel setting with contraction mappings?
• RQ5 Does using stochastic solvers in AID lead to better practical performance compared to deterministic AID in large-scale settings?

Key findings

• The proposed SID method provides a mean square error bound for the hypergradient estimator that is agnostic to the choice of stochastic solvers used for the lower-level problem and the linear system.
• The Stoch dec variant of SID converges in expectation to the true hypergradient at a rate of O(1/(γ_λ + t)), where t is the number of iterations.
• The Stoch const variant may not converge to the true hypergradient in expectation, as it can approach a neighborhood of the solution but not necessarily converge to it.
• Numerical experiments show that SID, especially with decreasing stepsizes, outperforms the deterministic AID method in terms of hypergradient approximation accuracy.
• The convergence analysis extends the SGD framework to stochastic fixed-point iterations under contraction assumptions, providing theoretical support for the method.
• The method enables scalable hyperparameter optimization by replacing expensive deterministic solvers with stochastic approximations, maintaining theoretical guarantees.