[Paper Review] SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation

Bo Dai, Albert Shaw+6 more|arXiv (Cornell University)|Dec 29, 2017

Adaptive Dynamic Programming Control120 citations

TL;DR

SBEED reformulates the Bellman equation into a primal-dual saddle-point problem with smoothing, enabling convergent reinforcement learning using nonlinear function approximators such as neural networks. It provides convergence guarantees and favorable empirical results in continuous control tasks.

ABSTRACT

When function approximation is used, solving the Bellman optimality equation with stability guarantees has remained a major open problem in reinforcement learning for decades. The fundamental difficulty is that the Bellman operator may become an expansion in general, resulting in oscillating and even divergent behavior of popular algorithms like Q-learning. In this paper, we revisit the Bellman equation, and reformulate it into a novel primal-dual optimization problem using Nesterov's smoothing technique and the Legendre-Fenchel transformation. We then develop a new algorithm, called Smoothed Bellman Error Embedding, to solve this optimization problem where any differentiable function class may be used. We provide what we believe to be the first convergence guarantee for general nonlinear function approximation, and analyze the algorithm's sample complexity. Empirically, our algorithm compares favorably to state-of-the-art baselines in several benchmark control problems.

Motivation & Objective

• Address instability and divergence of Bellman-based methods with nonlinear function approximators.
• Introduce a smoothed Bellman operator to enable stable optimization.
• Develop a primal-dual objective that avoids the double-sample issue and supports off-policy learning.
• Provide convergence guarantees and sample complexity analysis for nonlinear function approximation.
• Demonstrate empirical performance on benchmark control problems.
• Extend to both continuous and discrete action spaces and unify value function estimation with policy optimization.

Proposed method

• Reformulate the Bellman equation as a smoothed, entropy-regularized max via Nesterov smoothing, yielding a contraction operator with a unique fixed point.
• Derive a primal-dual objective that couples the value function V, policy π, and a dual variable ν (or ρ), enabling optimization without the non-smooth max operator.
• Transform the squared Bellman error into a saddle-point problem using Fenchel duality to avoid the double-sample issue.
• Introduce a two-player (minimax) objective Lη(V,π;ρ) that trades off squared Bellman residuals and a variance-canceling dual term.
• Develop a stochastic mirror descent algorithm (SBEED) to update nonlinear function approximators for V and π, with solvable dual updates.
• Provide theoretical guarantees on convergence to stationary points, generalization bounds, and an explicit error decomposition including smoothing bias and approximation error.

Experimental results

Research questions

• RQ1 Can nonlinear function approximators yield convergent off-policy RL when solving the Bellman equation?
• RQ2 Does smoothing the Bellman operator and adopting a primal-dual formulation ensure stability and convergence with neural networks?
• RQ3 How does the proposed SBEED framework fare in sample efficiency and robustness compared to state-of-the-art baselines on continuous control tasks?
• RQ4 What is the impact of smoothing parameter on bias-variance trade-off and overall error in practice?
• RQ5 Can the method handle both continuous and discrete action spaces within a unified objective?

Key findings

• SBEED provides convergence guarantees for general nonlinear function approximation in off-policy RL.
• The smoothed Bellman operator remains a contraction, ensuring a unique fixed point Vλ* for λ>0.
• A tractable primal-dual formulation avoids the double-sample issue and enables stochastic gradient updates.
• The algorithm achieves stable learning with neural networks and shows favorable empirical performance on continuous control benchmarks.
• An explicit error decomposition highlights smoothing bias, approximation error, and statistical error, with convergence to V* as λ→0 and data grows.
• SBEED unifies value estimation and policy optimization and supports both multi-step bootstrapping and eligibility traces.