[Paper Review] Online Markov Decision Processes with Terminal Law Constraints

Bianca Marin Moreno, Margaux Brégère+2 more|arXiv (Cornell University)|Jan 12, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

This paper introduces a reset-free periodic framework for online MDPs with unknown dynamics and adversarial losses, defines periodic policies, and provides algorithms with sublinear periodic regret for multi-agent settings.

ABSTRACT

Traditional reinforcement learning usually assumes either episodic interactions with resets or continuous operation to minimize average or cumulative loss. While episodic settings have many theoretical results, resets are often unrealistic in practice. The infinite-horizon setting avoids this issue but lacks non-asymptotic guarantees in online scenarios with unknown dynamics. In this work, we move towards closing this gap by introducing a reset-free framework called the periodic framework, where the goal is to find periodic policies: policies that not only minimize cumulative loss but also return the agents to their initial state distribution after a fixed number of steps. We formalize the problem of finding optimal periodic policies and identify sufficient conditions under which it is well-defined for tabular Markov decision processes. To evaluate algorithms in this framework, we introduce the periodic regret, a measure that balances cumulative loss with the terminal law constraint. We then propose the first algorithms for computing periodic policies in two multi-agent settings and show they achieve sublinear periodic regret of order $ ilde O(T^{3/4})$. This provides the first non-asymptotic guarantees for reset-free learning in the setting of $M$ homogeneous agents, for $M > 1$.

Motivation & Objective

• Formalize periodic policies in reset-free online MDPs with unknown dynamics.
• Propose a periodic regret metric that accounts for terminal distribution constraints.
• Develop algorithms for computing periodic policies in multi-agent, adversarial settings.
• Establish non-asymptotic guarantees for periodic regret in M>1 agent scenarios.

Proposed method

• Define periodic policy as one with rho P_pi = rho, ensuring return to initial distribution after N steps.
• Introduce the convex RL framework to handle general convex losses over state-action distributions.
• Develop a bonus-based exploration Mirror Descent algorithm (MDPP-K) to handle unknown transitions and adversarial losses.
• Use a constrained MDP formulation with terminal-law constraints transformed into feasible inequalities via bonuses.
• Provide two frameworks: Framework 1 with known rho_t and Framework 2 with estimated rho_t and limited resets, both yielding periodic regret bounds.
• Analyze and prove a non-asymptotic periodic regret bound of order tilde-O(T^{3/4}) under appropriate conditions.

Experimental results

Research questions

• RQ1 What conditions ensure the existence of periodic policies in online MDPs with unknown dynamics?
• RQ2 How to define and minimize a periodic regret that balances cumulative loss and terminal distribution constraints?
• RQ3 Can we design online algorithms with provable non-asymptotic guarantees for reset-free learning with multiple homogeneous agents?
• RQ4 How does unknown transition dynamics impact feasibility and how can bonuses ensure exploration and constraint satisfaction?

Key findings

• Propose periodic policy concept and contraction Assumption 2 ensuring ergodicity toward rho.
• Introduce periodic regret R_T combining cumulative loss differences and terminal-distribution deviations.
• MDPP-K algorithm achieves a sublinear periodic regret of order tilde-O(T^{3/4}) for M>1 agents.
• Provide two frameworks (known rho_t and unknown rho_t with limited resets) with corresponding regret guarantees.
• Demonstrate feasibility and high-probability bounds for the constrained MD problem under adversarial losses.