[Paper Review] Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm

Yuchao Dong|arXiv (Cornell University)|Aug 4, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper introduces a randomized optimal stopping framework in continuous time by embedding entropy regularization into the reward functional, transforming the problem into a solvable HJB equation. It establishes convergence rates for policy iteration and demonstrates that the temperature parameter λ balances learning speed and bias, with numerical results validating the approach on American put options using a tailored reinforcement learning algorithm.

ABSTRACT

In this paper, we study the optimal stopping problem in the so-called exploratory framework, in which the agent takes actions randomly conditioning on current state and an entropy-regularized term is added to the reward functional. Such a transformation reduces the optimal stopping problem to a standard optimal control problem. We derive the related HJB equation and prove its solvability. Furthermore, we give a convergence rate of policy iteration and the comparison to classical optimal stopping problem. Based on the theoretical analysis, a reinforcement learning algorithm is designed and numerical results are demonstrated for several models.

Motivation & Objective

• To extend the exploratory framework of reinforcement learning to continuous-time optimal stopping problems.
• To reformulate the optimal stopping problem as a regularized optimal control problem using entropy regularization.
• To derive and prove the solvability of the associated Hamilton-Jacobi-Bellman (HJB) equation for the American put option.
• To analyze the trade-off between convergence rate and bias in the choice of the temperature parameter λ.
• To design and validate a reinforcement learning algorithm based on theoretical insights, with numerical experiments on financial models.

Proposed method

• Introduce a randomized policy framework where actions are chosen probabilistically based on a temperature-adjusted softmax over value estimates.
• Modify the reward functional by adding a regularization term proportional to the differential entropy of the policy, promoting exploration.
• Derive the HJB equation for the value function under this exploratory formulation, proving its solvability via viscosity solution theory and Schauder estimates.
• Establish bounds on the difference between the regularized value function and the classical optimal value function, showing it scales as O(λ log(1/λ)).
• Design a policy iteration algorithm with convergence rate analysis, showing linear convergence under appropriate conditions.
• Implement a reinforcement learning algorithm using neural networks to approximate the value function and policy, with soft stopping rules to ensure differentiability.

Figure 1 : Value functions $u^{\lambda}$ and $u^{*}$ at $t=0$

Experimental results

Research questions

• RQ1 How can the exploratory framework from continuous-time reinforcement learning be adapted to optimal stopping problems in continuous time?
• RQ2 What is the impact of the temperature parameter λ on the trade-off between convergence speed and bias in the solution?
• RQ3 Can the HJB equation derived for the regularized optimal stopping problem be proven to have a classical solution?
• RQ4 How does the regularized value function V^λ relate to the classical optimal value function V* in terms of approximation error?
• RQ5 What is the convergence rate of the policy iteration algorithm in this randomized optimal stopping setting?

Key findings

• The HJB equation for the regularized optimal stopping problem is shown to have a classical solution, with continuous derivatives established via local Schauder estimates.
• The difference between the regularized value function V^λ and the classical optimal value function V* is bounded by O(λ log(1/λ)), indicating a controlled bias.
• The policy iteration algorithm converges linearly, with the convergence rate dependent on the choice of the temperature parameter λ.
• Numerical results demonstrate that increasing λ improves convergence speed but increases the bias in the solution, confirming the trade-off.
• The proposed reinforcement learning algorithm successfully learns optimal stopping strategies for the American put option, with soft stopping rules enabling gradient-based optimization.
• The theoretical bound on the expected discounted payoff loss is approximately O(λ log(1/λ)), which quantifies the wealth cost of using randomized policies.

(a) $\{u^{\lambda}>h\}$