[Paper Review] Optimal control in Markov decision processes via distributed optimization

Jie Fu, Shuo Han+1 more|arXiv (Cornell University)|Mar 24, 2015

Formal Methods in Verification19 references3 citations

TL;DR

This paper proposes a distributed optimization framework for optimal control in large-scale Markov decision processes (MDPs) with temporal logic constraints, leveraging decomposition to reduce computational complexity. By reformulating the problem as a sparse linear program and applying the block splitting algorithm based on ADMM, the method enables scalable, parallelizable synthesis of policies that maximize satisfaction probabilities of complex temporal logic specifications, achieving near-optimal average rewards in robotic motion planning examples.

ABSTRACT

Optimal control synthesis in stochastic systems with respect to quantitative temporal logic constraints can be formulated as linear programming problems. However, centralized synthesis algorithms do not scale to many practical systems. To tackle this issue, we propose a decomposition-based distributed synthesis algorithm. By decomposing a large-scale stochastic system modeled as a Markov decision process into a collection of interacting sub-systems, the original control problem is formulated as a linear programming problem with a sparse constraint matrix, which can be solved through distributed optimization methods. Additionally, we propose a decomposition algorithm which automatically exploits, if exists, the modular structure in a given large-scale system. We illustrate the proposed methods through robotic motion planning examples.

Motivation & Objective

• To address the scalability limitations of centralized optimal control synthesis in large MDPs with temporal logic constraints.
• To develop a distributed framework that decomposes large MDPs into smaller sub-problems while preserving global optimality.
• To enable efficient, parallelizable solution of large-scale MDP control problems using distributed optimization techniques.
• To handle both discounted-reward and average-reward objectives in ergodic MDPs through approximation and decomposition.
• To demonstrate the method’s effectiveness on robotic motion planning with complex temporal logic specifications.

Proposed method

• Decomposes a large MDP into interacting subsystems based on modular structure, especially exploiting planar graph properties for efficient decomposition.
• Reformulates the optimal control problem as a linear program with a sparse constraint matrix to enable distributed solution.
• Applies the block splitting algorithm, a variant of ADMM, to solve the decomposed problem in a distributed and parallel manner.
• Uses proximal operators and projection steps to update variables in each iteration, ensuring convergence to a feasible and optimal solution.
• Implements an exchange and averaging mechanism to synchronize variables across subsystems and enforce coupling constraints.
• Approximates average reward via discounted reward with γ = 0.98 to improve convergence and feasibility in ergodic MDPs.

Experimental results

Research questions

• RQ1 Can a decomposition-based distributed optimization method effectively scale optimal control synthesis in large MDPs with temporal logic constraints?
• RQ2 How can modular structure in large MDPs be automatically exploited to reduce computational complexity?
• RQ3 To what extent does the distributed block splitting algorithm maintain feasibility and convergence compared to centralized methods?
• RQ4 Can the approximation of average reward via discounted reward be effectively used in distributed synthesis for ergodic MDPs?
• RQ5 How does the method perform in practice on robotic motion planning tasks with complex temporal logic objectives?

Key findings

• The distributed synthesis algorithm terminated after 14,284 iterations with an optimal discounted reward of 0.9998, corresponding to an average reward of 0.02 after scaling by (1−γ).
• The infeasibility measure of the final solution was 0.016, indicating good feasibility despite the average-reward constraint.
• The method successfully handled a 14-state deterministic Büchi automaton (DBA) derived from a complex temporal logic specification involving four regions.
• The block splitting algorithm enabled concurrent solution of sub-problems with penalty-based coordination, improving convergence over two-stage methods.
• The approach demonstrated scalability and effectiveness on a robotic motion planning case study with concurrent reachability and recurrence objectives.
• The decomposition algorithm efficiently exploited the modular and loosely coupled structure of the underlying MDP, reducing problem size and enabling parallel computation.