[Paper Review] Approximate Dynamic Programming with Feasibility Guarantees

Alexander Engelmann, Maísa Beraldo Bandeira+1 more|arXiv (Cornell University)|Jun 9, 2023

Optimal Power Flow Distribution4 citations

TL;DR

This paper proposes a novel approximate dynamic programming (ADP) scheme for tree-structured optimization problems that guarantees a feasible solution in a single backward-forward pass, even with nonconvex constraints. By combining value function approximations with inner approximations of feasible sets via set projections, the method ensures one-shot feasibility while incorporating cost information, generalizing prior approaches in power systems and optimal control.

ABSTRACT

Safe and economic operation of networked systems is often challenging. Optimization-based schemes are frequently considered, since they achieve near-optimality while ensuring safety via the explicit consideration of constraints. In applications, these schemes, however, often require solving large-scale optimization problems. Iterative techniques from distributed optimization are frequently proposed for complexity reduction. Yet, they achieve feasibility only asymptotically, which induces a substantial computational burden. This work presents an approximate dynamic programming scheme, which is guaranteed to deliver a feasible solution in "one shot", i.e., in one backward-forward iteration over all subproblems provided they are coupled by a tree structure. Our proposed scheme generalizes methods from seemingly disconnected domains such as power systems and optimal control. We demonstrate its efficacy for problems with nonconvex constraints via numerical examples from both domains.

Motivation & Objective

• To address the challenge of achieving feasibility in large-scale, hierarchical optimization of networked systems without iterative refinement.
• To develop a method that guarantees feasibility in a single pass, overcoming the asymptotic feasibility of distributed optimization and the curse of dimensionality in classic dynamic programming.
• To unify and generalize existing approaches from power systems and optimal control by incorporating cost information and nonconvex constraints.
• To enable practical deployment in real-time applications where computational delay and feasibility are critical.
• To provide formal feasibility guarantees for approximate dynamic programming in tree-structured problems with general constraint sets.

Proposed method

• The method uses extended real-valued value functions that encode both cost and feasibility information, enabling recursive backward propagation from leaves to root.
• It introduces inner approximations of the feasible sets of subproblems, ensuring that solutions remain feasible in the parent problems.
• Set projections are used to compute these inner approximations, allowing the propagation of feasibility constraints through the tree structure.
• The algorithm performs one backward pass to compute approximated value functions and feasible sets, followed by a forward pass to compute a feasible solution.
• Cost information is preserved either exactly or approximately via piecewise-quadratic or polyhedral approximations in the value functions.
• The framework is compatible with computational geometry tools for efficient projection computation, especially for polytopic constraints in DC OPF and LinDistFlow models.

Experimental results

Research questions

• RQ1 Can a one-shot feasible solution be guaranteed for tree-structured optimization problems with nonconvex constraints using approximate dynamic programming?
• RQ2 How can cost information be integrated into feasibility-preserving ADP schemes without sacrificing computational tractability?
• RQ3 To what extent can the proposed method generalize existing approaches in power systems and optimal control that rely on feasibility cuts or flexibility aggregation?
• RQ4 What is the role of inner approximations and set projections in ensuring feasibility while maintaining suboptimality bounds?
• RQ5 How does the proposed ADP scheme compare to classical distributed optimization in terms of convergence speed and feasibility guarantees?

Key findings

• The proposed ADP scheme guarantees feasibility of the solution in a single backward-forward pass, even for nonconvex constraints, by using inner approximations of feasible sets.
• For DC Optimal Power Flow (DC OPF), the method achieves exact feasibility via polytope projections, with the value function being piecewise-quadratic and nondifferentiable.
• The method generalizes existing flexibility aggregation techniques in power systems, unifying them under a common ADP framework with formal feasibility guarantees.
• Numerical examples demonstrate efficacy on problems with nonconvex constraints, showing that the approach maintains feasibility while reducing computational burden compared to iterative schemes.
• The framework enables method transfer between domains such as power systems and optimal control by abstracting the tree-structured problem into a common ADP formulation.
• Theoretical analysis confirms that the domain of the value function corresponds exactly to the projection of the joint feasible set, ensuring consistency across the hierarchy.