[Paper Review] Inexact cuts for Deterministic and Stochastic Dual Dynamic Programming applied to convex nonlinear optimization problems

Vincent Guigues|arXiv (Cornell University)|Jul 4, 2017

Risk and Portfolio Optimization18 references3 citations

TL;DR

This paper introduces Inexact Dual Dynamic Programming (IDDP) and Inexact Stochastic Dual Dynamic Programming (ISDDP) for solving convex nonlinear dynamic programming problems where subproblems are solved with bounded error. It proves convergence to approximate solutions when errors vanish, establishing theoretical foundations for using inexact cuts in DDP and SDDP variants.

ABSTRACT

We introduce an extension of Dual Dynamic Programming (DDP) to solve convex nonlinear dynamic programming equations. We call Inexact DDP (IDDP) this extension which applies to situations where some or all primal and dual subproblems to be solved along the iterations of the method are solved with a bounded error. We show that any accumulation point of the sequence of decisions is an approximate solution to the dynamic programming equations. When these errors tend to zero as the number of iterations goes to infinity, we show that IDDP solves the dynamic programming equations. We extend the analysis to stochastic convex nonlinear dynamic programming equations, introducing Inexact Stochastic Dual Dynamic Programming (ISDDP), an inexact variant of SDDP corresponding to the situation where some or all problems to be solved in the forward and backward passes of SDDP are solved approximately. We also show the almost sure convergence of ISDDP for vanishing errors.

Motivation & Objective

• Address the gap in theoretical analysis for DDP and SDDP when subproblems are solved inexactly, which is common in nonlinear problems.
• Develop inexact cuts for value functions in convex nonlinear optimization that bound the error between the true value function and the cut at a given point.
• Provide conditions under which ε-optimal dual solutions remain bounded, enabling stable cut computation.
• Extend the analysis to stochastic settings, introducing ISDDP with convergence guarantees under vanishing errors.
• Enable practical acceleration of DDP and SDDP by allowing early, less accurate solutions in early iterations, especially for first-stage problems.

Proposed method

• Propose inexact cuts for the value function Q(x) of a convex optimization problem, derived from ε-optimal primal and dual solutions.
• Derive analytical bounds ε₀ on the error Q(𝐱̄) − C(𝐱̄) between the true value function and the inexact cut, ensuring ε₀ → 0 as ε → 0.
• Provide two formulations: one for problems where x appears only in the objective (Proposition 2.2), and a general case with constraints (Propositions 2.7–2.8).
• Introduce a bound on the norm of ε-optimal dual solutions (Proposition 3.1), crucial for stability and error control.
• Establish convergence of inexact cuts to the true value function when εᵏ → 0 along a sequence of iterations (Propositions 4.5–4.6).
• Extend the framework to stochastic settings via ISDDP, proving almost sure convergence under vanishing errors (Theorem 5.3).

Experimental results

Research questions

• RQ1 Can DDP and SDDP be extended to handle inexact solutions of subproblems while preserving convergence to the dynamic programming solution?
• RQ2 What error bounds can be derived for inexact cuts of value functions in convex nonlinear programs when primal and dual solutions are ε-optimal?
• RQ3 Under what conditions are ε-optimal dual solutions uniformly bounded, ensuring numerical stability in cut generation?
• RQ4 Does ISDDP converge almost surely when subproblem errors vanish over iterations in a stochastic setting?
• RQ5 Can inexact variants of DDP and SDDP be practically accelerated by solving early-stage subproblems less accurately and increasing precision over time?

Key findings

• Inexact cuts for value functions are derived with explicit error bounds ε₀, which vanish as the subproblem solution error ε → 0.
• For problems where x appears only in the objective, tighter error bounds are derived under additional assumptions (Propositions 2.3 and 2.5).
• A uniform bound on the norm of ε-optimal dual solutions is provided, ensuring that inexact cuts remain well-conditioned (Proposition 3.1).
• When εᵏ → 0 along a sequence of iterations, the distance between inexact cuts and the true value function at the corresponding points converges to zero (Propositions 4.5 and 4.6).
• ISDDP converges almost surely to the optimal solution when subproblem errors vanish, even when cuts are computed inexactly (Theorem 5.3).
• The theoretical framework supports practical acceleration strategies, such as solving early iterations with lower accuracy, without sacrificing convergence.