[Paper Review] Online Reinforcement Learning of Optimal Threshold Policies for Markov Decision Processes

Arghyadip Roy, Vivek S. Borkar+2 more|arXiv (Cornell University)|Dec 21, 2019

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes SALMUT, a structure-aware online reinforcement learning algorithm that exploits the ordered multi-threshold structure of optimal policies in Markov Decision Processes. By learning only threshold parameters via a two-timescale stochastic approximation scheme, SALMUT achieves faster convergence and reduced computational and storage complexity compared to classical RL methods, with asymptotic convergence to the optimal policy proven theoretically and validated through simulations.

ABSTRACT

To overcome the curses of dimensionality and modeling of Dynamic Programming (DP) methods to solve Markov Decision Process (MDP) problems, Reinforcement Learning (RL) methods are adopted in practice. Contrary to traditional RL algorithms which do not consider the structural properties of the optimal policy, we propose a structure-aware learning algorithm to exploit the ordered multi-threshold structure of the optimal policy, if any. We prove the asymptotic convergence of the proposed algorithm to the optimal policy. Due to the reduction in the policy space, the proposed algorithm provides remarkable improvements in storage and computational complexities over classical RL algorithms. Simulation results establish that the proposed algorithm converges faster than other RL algorithms.

Motivation & Objective

• To address the curse of dimensionality and modeling in Dynamic Programming for MDPs by leveraging structural properties of optimal policies.
• To reduce the policy search space by focusing only on ordered multi-threshold policies, thereby improving computational and storage efficiency.
• To develop an online learning algorithm that asymptotically converges to the optimal threshold policy without requiring prior knowledge of transition probabilities.
• To demonstrate faster convergence and lower complexity than classical RL algorithms through theoretical analysis and simulation.

Proposed method

• The algorithm uses a two-timescale stochastic approximation framework, updating value functions on a fast timescale and threshold parameters on a slow timescale.
• Threshold parameters are updated using gradient estimates of the average reward with respect to the thresholds, derived via simultaneous perturbation stochastic approximation.
• The method enforces monotonicity in value function differences to preserve the threshold structure during learning.
• It leverages the unimodal property of average reward with respect to threshold parameters, enabling efficient gradient-based optimization.
• The algorithm avoids full policy enumeration by restricting learning to threshold parameters, significantly reducing the search space.
• Policy evaluation and improvement are performed iteratively using sampled transitions, with value function updates based on sampled rewards and state transitions.

Experimental results

Research questions

• RQ1 Can exploiting the known multi-threshold structure of optimal policies lead to faster convergence in online reinforcement learning for MDPs?
• RQ2 How can the policy search space be reduced effectively while maintaining asymptotic convergence to the optimal policy?
• RQ3 What is the impact of structure-aware learning on computational and storage complexity compared to classical RL algorithms?
• RQ4 Does gradient-based threshold update using stochastic approximation yield stable and convergent learning in the presence of sampling noise?
• RQ5 Can the unimodal property of average reward with respect to threshold parameters be leveraged to design efficient online learning algorithms?

Key findings

• The SALMUT algorithm asymptotically converges to the optimal threshold policy under standard assumptions on step-sizes and sampling.
• The algorithm achieves faster convergence than classical Q-learning and PDS-based methods due to the reduced policy search space.
• Storage and computational complexity are significantly reduced by learning only threshold parameters instead of full value functions over all state-action pairs.
• The average reward is proven to be unimodal with respect to each threshold parameter, enabling efficient gradient-based optimization.
• Simulation results confirm that SALMUT converges faster than baseline RL algorithms in a finite-buffer multi-server queue with multiple customer classes.
• The method maintains structural consistency (e.g., monotonicity of value function differences) throughout learning, ensuring the threshold policy remains valid at all iterations.