[Paper Review] Reoptimization Nearly Solves Weakly Coupled Markov Decision Processes

Nicolas Gast, Bruno Gaujal+1 more|arXiv (Cornell University)|Nov 3, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces the LP-update policy for finite-horizon weakly coupled Markov decision processes (MDPs), a generalization of restless bandits with multiple actions and constraints. By solving a linear program at each time step and applying a rounding procedure, the policy achieves asymptotic optimality at rate 𝒪(1/√N), improving to 𝒪(1/N) under non-degeneracy and e⁻ᴼ⁽ᴺ⁾ under perfect rounding, with theoretical guarantees and efficient implementation.

ABSTRACT

We propose a new policy, called the LP-update policy, to solve finite horizon weakly-coupled Markov decision processes. The latter can be seen as multi-constraint multi-action bandits, and generalize the classical restless bandit problems. Our solution is based on re-solving periodically a relaxed version of the original problem, that can be cast as a linear program (LP). When the problem is made of $N$ statistically identical sub-components, we show that the LP-update policy becomes asymptotically optimal at rate $O(T^2/\sqrt{N})$. This rate can be improved to $O(T/\sqrt{N})$ if the problem satisfies some ergodicity property and to $O(1/N)$ if the problem is non-degenerate. The definition of non-degeneracy extends the same notion for restless bandits. By using this property, we also improve the computational efficiency of the LP-update policy. We illustrate the performance of our policy on randomly generated examples, as well as a generalized applicant screening problem, and show that it outperforms existing heuristics.

Motivation & Objective

• To address the lack of asymptotically optimal policies for weakly coupled MDPs with multiple actions and constraints, generalizing beyond classical restless bandits.
• To close the gap in theoretical guarantees for multi-action, multi-constraint MDPs where existing policies lack asymptotic optimality proofs.
• To develop a scalable, efficient policy that leverages linear programming and rounding for large-scale stochastic sequential decision-making.
• To establish convergence rates for the proposed policy under various structural assumptions, including non-degeneracy and perfect rounding.
• To provide a practical and theoretically grounded method for real-world applications such as generalized applicant screening.

Proposed method

• The LP-update policy solves a time-dependent linear program (LP) at each decision epoch t, using the current state configuration M^(N)(t) and remaining time horizon T−t to compute an optimal fractional action distribution y*(t).
• The policy computes a feasible integer action count vector Y^(N)(t) via floor-based rounding: Y^(N)_{s,a}(t) = N⁻¹⌊Ny*_{s,a}(t)⌋ for a≠0, with passive actions adjusted to satisfy resource constraints.
• The policy uses the rounded action counts to transition the system to the next time step, ensuring feasibility under global resource constraints.
• Theoretical analysis leverages dynamic programming principles and the principle of optimality to compare the policy’s value with the relaxed LP value V_rel.
• A Lipschitz-continuity argument is used to bound the deviation between the policy’s performance and the LP relaxation, establishing 𝒪(1/√N) optimality.
• An advanced rounding procedure is introduced to improve convergence rates under non-degenerate and perfectly roundable problems.

Experimental results

Research questions

• RQ1 Can a policy be designed for weakly coupled MDPs with multiple actions and constraints that achieves asymptotic optimality as N→∞?
• RQ2 What convergence rate can be theoretically guaranteed for such a policy, and how does it depend on structural properties like non-degeneracy or perfect rounding?
• RQ3 How does the LP-update policy compare to existing policies in terms of performance and computational efficiency in large-scale settings?
• RQ4 Can the rounding procedure be optimized to improve convergence beyond 𝒪(1/√N) under specific problem conditions?
• RQ5 Is the LP-update policy implementable efficiently while maintaining theoretical performance guarantees?

Key findings

• The LP-update policy achieves asymptotic optimality at rate 𝒪(1/√N) for any weakly coupled MDP with statistically identical arms.
• Under the non-degeneracy condition, the convergence rate improves to 𝒪(1/N), extending a known property from two-action restless bandits.
• If the problem admits a perfect rounding, the convergence rate becomes exponentially fast, e⁻ᴼ⁽ᴺ⁾, significantly outperforming the general case.
• The policy is proven to be 𝒪(1/√N)-optimal, with matching lower bounds showing this rate is tight in general.
• An efficient implementation is enabled by the proposed rounding procedure, which maintains theoretical guarantees while improving practical performance.
• The method is validated on a generalized applicant screening problem, demonstrating strong empirical performance consistent with theoretical predictions.