[論文レビュー] The Pontryagin maximum principle and $Q$-functions in rough environments

Estepan Ashkarian, Prakash Chakraborty+2 more|arXiv (Cornell University)|Jan 8, 2026

Stability and Controllability of Differential Equations被引用数 0

ひとこと要約

The paper derives the Pontryagin maximum principle and infinitesimal Q/q-functions for relaxed control of noisy rough differential equations, using spike variation perturbations to enable policy improvement under entropic costs.

ABSTRACT

We derive the Pontryagin maximum principle and $Q$-functions for the relaxed control of noisy rough differential equations. Our main tool is the development of a novel differentiation procedure along `spike variation' perturbations of the optimal state-control pair. We then exploit our development of the infinitesimal $Q$-function (also known as the $q$-function) to derive a policy improvement algorithm for settings with entropic cost constraints.

研究の動機と目的

• Motivate reinforcement learning in general noisy and non-Markovian environments captured by rough paths.
• Develop a Pontryagin maximum principle for relaxed controls in rough differential equations.
• Define and utilize infinitesimal Q-function (q-function) to relate PMP to Q-learning in continuous time.
• Propose entropy-regularized policy improvement and Gibbs-form policies in open-loop and closed-loop settings.

提案手法

• Model state dynamics as rough differential equations with relaxed control and rough noise.
• Introduce and implement a spike variation perturbation framework to derive the PMP in the rough setting.
• Define and analyze the infinitesimal q-function to connect PMP with Q-learning concepts in continuous time.
• Establish a rough viscosity framework for value functions and HJB-type equations under rough input.
• Derive open-loop Gibbs-form policies under entropy terms and devise policy improvement via rough flow transformations.
• Provide a computational angle for policy improvement and discuss viscosity/optimality principles in rough dynamics.

実験結果

リサーチクエスチョン

• RQ1 How can the Pontryagin maximum principle be formulated for relaxed controls in rough differential equations?
• RQ2 What is the appropriate infinitesimal analog of the Q-function (the q-function) in rough environments, and how can it be derived?
• RQ3 How does entropy regularization influence optimal policies in continuous-time, noisy, non-Markovian settings?
• RQ4 Can policy improvement be justified and implemented when the dynamics are driven by rough paths and the control space is probabilistic (relaxed)?

主な発見

• A Pontryagin maximum principle is derived for relaxed controls in rough differential equations using a novel spike variation differentiation method.
• The infinitesimal q-function is constructed and shown to relate the PMP to a Hamiltonian-like object in the rough setting.
• Entropy terms lead to Gibbs-form optimal open-loop policies and link the q-function to explicit policy representations.
• Policy improvement in both open-loop and closed-loop settings is analyzed via transformation along rough flows, enabling a rough viscosity HJB framework.
• The framework encompasses Gaussian processes and fractional Brownian motion as natural applications, illustrating broad applicability.
• The work contributes foundational tools for numerical analysis and reinforcement learning in rough, non-Markovian environments.