[Paper Review] On the Iteration Complexity of Hypergradient Computation

Riccardo Grazzi, Luca Franceschi+2 more|arXiv (Cornell University)|Jun 29, 2020

Model Reduction and Neural Networks29 references22 citations

TL;DR

This paper provides a unified theoretical analysis of hypergradient computation in bilevel optimization, comparing iterative differentiation (ITD) and approximate implicit differentiation (AID) under contraction mapping assumptions. It establishes iteration complexity bounds and finds that AID using conjugate gradients offers superior computational efficiency and approximation accuracy, especially when the lower-level map is contractive.

ABSTRACT

We study a general class of bilevel problems, consisting in the minimization of an upper-level objective which depends on the solution to a parametric fixed-point equation. Important instances arising in machine learning include hyperparameter optimization, meta-learning, and certain graph and recurrent neural networks. Typically the gradient of the upper-level objective (hypergradient) is hard or even impossible to compute exactly, which has raised the interest in approximation methods. We investigate some popular approaches to compute the hypergradient, based on reverse mode iterative differentiation and approximate implicit differentiation. Under the hypothesis that the fixed point equation is defined by a contraction mapping, we present a unified analysis which allows for the first time to quantitatively compare these methods, providing explicit bounds for their iteration complexity. This analysis suggests a hierarchy in terms of computational efficiency among the above methods, with approximate implicit differentiation based on conjugate gradient performing best. We present an extensive experimental comparison among the methods which confirm the theoretical findings.

Motivation & Objective

• To provide a unified theoretical framework for analyzing the iteration complexity of hypergradient approximation methods in bilevel optimization.
• To compare the computational efficiency and approximation quality of iterative differentiation (ITD) and approximate implicit differentiation (AID) methods.
• To establish explicit iteration complexity bounds for ITD and AID under the assumption that the lower-level fixed-point map is a contraction.
• To evaluate the practical performance of these methods on hyperparameter optimization, meta-learning, and equilibrium models (EQM).
• To identify conditions under which AID with conjugate gradients outperforms ITD in terms of accuracy and memory efficiency.

Proposed method

• Formulates a general bilevel problem where the upper-level objective depends on the solution to a parametric fixed-point equation defined by a contraction mapping.
• Analyzes two main hypergradient approximation strategies: iterative differentiation (ITD) using forward or reverse mode automatic differentiation, and approximate implicit differentiation (AID) via solving linear systems with conjugate gradients.
• Derives theoretical iteration complexity bounds for both ITD and AID under the contraction assumption, enabling direct quantitative comparison.
• Applies the conjugate gradient method to solve the normal equations in AID when the Jacobian of the fixed-point map is non-symmetric.
• Employs spectral projection to enforce the contraction condition (||A|| < 1) in equilibrium models, ensuring convergence and stability.
• Conducts extensive experiments on hyperparameter optimization, meta-learning, and equilibrium models (EQM) to validate theoretical findings.

Experimental results

Research questions

• RQ1 How do the iteration complexities of ITD and AID compare under the contraction mapping assumption for the lower-level fixed-point problem?
• RQ2 What is the relative approximation accuracy and computational efficiency of AID with conjugate gradients versus ITD in bilevel optimization?
• RQ3 How does enforcing the contraction condition (||A|| < 1) affect the stability and performance of hypergradient approximation methods in equilibrium models?
• RQ4 In what settings does ITD outperform AID, particularly when the contraction assumption is violated?
• RQ5 To what extent does spectral projection or regularization enforce the contraction condition and improve method stability?

Key findings

• AID based on conjugate gradients achieves the best trade-off between approximation accuracy and iteration complexity, outperforming ITD in terms of computational efficiency when the contraction assumption holds.
• Theoretical iteration complexity bounds show that AID converges faster than ITD under the same error tolerance, especially when the Hessian of the lower-level problem is well-conditioned.
• When the contraction condition is enforced via spectral projection (||A|| < 1), all methods converge stably; however, AID with conjugate gradients is significantly faster and more memory-efficient than ITD.
• Without enforcing the contraction condition, AID methods (especially memory-less variants) become unstable across a wide range of learning rates, while ITD remains more robust.
• Projecting the spectral norm of the transition matrix A onto [0, 1−ε] acts as a strong regularizer, improving generalization and stability, consistent with prior findings.
• Empirical results confirm that AID with conjugate gradients is the preferred method for large-scale bilevel problems when the lower-level map is contractive, due to superior convergence and lower memory footprint.