[Paper Review] Weakly Time-Coupled Approximation of Markov Decision Processes

Negar Soheili, Selvaprabu Nadarajah+1 more|arXiv (Cornell University)|Mar 13, 2026

Risk and Portfolio Optimization0 citations

TL;DR

The paper introduces a weakly time-coupled approximation (WTCA) for finite-horizon MDPs that relaxes temporal coupling, enabling horizon-length independent computation and tighter bounds than ALP, while being competitive with PO under equal budgets.

ABSTRACT

Finite-horizon Markov decision processes (MDPs) with high-dimensional exogenous uncertainty and endogenous states arise in operations and finance, including the valuation and exercise of Bermudan and real options, but face a scalability barrier as computational complexity grows with the horizon. A common approximation represents the value function using basis functions, but methods for fitting weights treat cross-stage optimization differently. Least squares Monte Carlo (LSM) fits weights via backward recursion and regression, avoiding joint optimization but accumulating error over the horizon. Approximate linear programming (ALP) and pathwise optimization (PO) jointly fit weights to produce upper bounds, but temporal coupling causes computational complexity to grow with the horizon. We show this coupling is an artifact of the approximation architecture, and develop a weakly time-coupled approximation (WTCA) where cross-stage dependence is independent of horizon. For any fixed basis function set, the WTCA upper bound is tighter than that of ALP and looser than that of PO, and converges to the optimal policy value as the basis family expands. We extend parallel deterministic block coordinate descent to the stochastic MDP setting exploiting weak temporal coupling. Applied to WTCA, weak coupling yields computational complexity independent of the horizon. Within equal time budget, solving WTCA accommodates more exogenous samples or basis functions than PO, yielding tighter bounds despite PO being tighter for fixed samples and basis functions. On Bermudan option and ethanol production instances, WTCA produces tighter upper bounds than PO and LSM in every instance tested, with near-optimal policies at longer horizons.

Motivation & Objective

• Motivate approximation methods for high-dimensional, finite-horizon MDPs with irreversible decision implications.
• Analyze how temporal coupling in existing methods (ALP and PO) affects computational complexity and bound quality.
• Introduce WTCA to relax cross-stage coupling while preserving useful upper-bound properties.
• Develop a parallel stochastic block coordinate descent algorithm to exploit WTCA's weak coupling.
• Demonstrate empirical advantages of WTCA over PO and LSM on Bermudan option and ethanol production problems.

Proposed method

• Formalize MDP with endogenous and exogenous states and basis-function value approximations.
• Cast ALP and PO within a unified stochastic optimization framework to define temporal coupling.
• Show that PO and ALP are fully time-coupled (κ(F)=T) and formulate WTCA by replacing the dual supremum with an exogenous expectation.
• Define WTCA as a sum of stagewise Bellman deviations penalized in expectation, preserving local coupling (κ=2).
• Propose Parallel Stochastic Block Coordinate Descent (PS-BCD) that updates stage-wise blocks in parallel exploiting WTCA’s weak coupling.
• Provide theoretical comparisons: WTCA is a relaxation of ALP and upper-bounded by ALP, while PO remains potentially tighter for fixed bases.

Figure 1: Convergence of upper and lower bounds for WTCA (left) and PO (right) in the instance with $\mathbf{T=36}$ , $\mathbf{N=8}$ , and $\mathbf{w^{I}=100}$ .

Experimental results

Research questions

• RQ1 How does temporal coupling in ALP and PO affect computational complexity as the horizon grows?
• RQ2 Can we design an MDP approximation that preserves upper-bound guarantees while making computation horizon-independent?
• RQ3 Does WTCA provide tighter upper bounds than ALP and LSM under practical computation budgets?
• RQ4 Can parallel block coordinate descent efficiently solve WTCA without significant information loss?
• RQ5 How do WTCA and PO compare in terms of bound tightness and policy quality on practical irreversible decision problems?

Key findings

• WTCA yields an upper bound that, for a fixed basis set, is tighter than ALP but looser than PO.
• WTCA converges to the optimal value as the basis-function set expands.
• PS-BCD solves WTCA with horizon-length independent computation, enabling more samples or basis functions within a fixed budget.
• Under equal time budgets, WTCA can outperform PO in bound tightness due to better exploitation of parallelism, while maintaining comparable policy quality.
• WTCA and PO both outperform LSM in upper bound tightness and policy quality across tested Bermudan option and ethanol production instances.
• Empirical results show WTCA provides tighter bounds than PO and LSM in every tested instance, with WTCA yielding near-optimal policies at longer horizons.

Figure 2: Endogenous state transitions in ethanol production (Guthrie 2009 , Yang et al. 2024 , 2025 ) .