[Paper Review] A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic

Mingyi Hong, Hoi To Wai+2 more|arXiv (Cornell University)|Jul 10, 2020

Adaptive Dynamic Programming Control62 references52 citations

TL;DR

Introduces a two-timescale stochastic approximation (TTSA) algorithm for bilevel optimization with an unconstrained, strongly convex inner problem and a smooth outer objective, and derives convergence rates; applies TTSA to a two-timescale natural actor-critic policy optimization with shown rates.

ABSTRACT

This paper analyzes a two-timescale stochastic algorithm framework for bilevel optimization. Bilevel optimization is a class of problems which exhibit a two-level structure, and its goal is to minimize an outer objective function with variables which are constrained to be the optimal solution to an (inner) optimization problem. We consider the case when the inner problem is unconstrained and strongly convex, while the outer problem is constrained and has a smooth objective function. We propose a two-timescale stochastic approximation (TTSA) algorithm for tackling such a bilevel problem. In the algorithm, a stochastic gradient update with a larger step size is used for the inner problem, while a projected stochastic gradient update with a smaller step size is used for the outer problem. We analyze the convergence rates for the TTSA algorithm under various settings: when the outer problem is strongly convex (resp.~weakly convex), the TTSA algorithm finds an $\mathcal{O}(K^{-2/3})$-optimal (resp.~$\mathcal{O}(K^{-2/5})$-stationary) solution, where $K$ is the total iteration number. As an application, we show that a two-timescale natural actor-critic proximal policy optimization algorithm can be viewed as a special case of our TTSA framework. Importantly, the natural actor-critic algorithm is shown to converge at a rate of $\mathcal{O}(K^{-1/4})$ in terms of the gap in expected discounted reward compared to a global optimal policy.

Motivation & Objective

• Motivate and formalize bilevel optimization where the inner problem is strongly convex and the outer problem is smooth.
• Propose a single-loop TTSA algorithm that updates inner and outer variables on different timescales.
• Establish convergence rates for TTSA under strong, convex, and weakly convex outer objectives.
• Provide surrogate gradient construction for the outer objective using the inner solution via implicit differentiation.
• Demonstrate application to reinforcement learning via a two-timescale natural actor-critic PPO framework.

Proposed method

• Formulate TTSA that updates y on a larger step size and x on a smaller step size, ensuring y tracks y*(x) as x changes.
• Use a gradient surrogate for the outer objective based on y, specifically overline{∇}_x f(x,y) = ∇_x f(x,y) − ∇_{xy}^2 g(x,y) [∇_{yy}^2 g(x,y)]^{-1} ∇_y f(x,y).
• Provide stochastic gradient and Hessian/Jacobian estimates with controlled bias and variance (Assumptions 3, 7).
• Propose a gradient estimator h_f^k constructed from random samples to approximate overline{∇}_x f while leveraging the strongly convex inner problem.
• Analyze coupled inequalities and tracking error Δ_y^k to establish convergence rates for outer and inner recursions.

Experimental results

Research questions

• RQ1 Can a single-loop TTSA algorithm achieve convergence for bilevel problems with a strongly convex inner problem and a smooth outer objective?
• RQ2 What are the convergence rates for TTSA under strongly convex outer, convex outer, and weakly convex outer settings?
• RQ3 How does the two-timescale dynamic influence tracking error and overall convergence in practice?
• RQ4 Can TTSA be effectively applied to reinforcement learning frameworks such as actor-critic methods?
• RQ5 What surrogate gradient formulations enable practical computation of the outer objective gradient in TTSA?

Key findings

• TTSA achieves O(K_max^{-2/3})-optimality for strongly convex outer objectives with diminishing stepsizes.
• TTSA achieves O(K_max^{-2/5})-stationarity for weakly convex outer objectives.
• For convex outer objectives, TTSA achieves O(K_max^{-1/4})-outer-rate and O(K_max^{-1/2})-inner-rate with appropriate stepsize choices.
• A surrogate gradient based on implicit differentiation enables unbiased-ish estimation with controlled bias/variance.
• Application to two-timescale natural actor-critic PPO shows convergence rate O(K^{-1/4}) in terms of regret with respect to the optimal policy.