[Paper Review] A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization

Donghao Ying, Yuhao Ding+1 more|arXiv (Cornell University)|Oct 17, 2021

Reinforcement Learning in Robotics28 references4 citations

TL;DR

This paper proposes a dual approach for entropy-regularized constrained Markov decision processes (CMDPs) under soft-max parameterization, leveraging smooth Lagrangian duality to enable accelerated convergence. It introduces a projected accelerated gradient descent for dual variables and natural policy gradient in the inner loop, achieving a global convergence rate of $\widetilde{\mathcal{O}}(1/T)$ for both optimality gap and constraint violation.

ABSTRACT

We study entropy-regularized constrained Markov decision processes (CMDPs) under the soft-max parameterization, in which an agent aims to maximize the entropy-regularized value function while satisfying constraints on the expected total utility. By leveraging the entropy regularization, our theoretical analysis shows that its Lagrangian dual function is smooth and the Lagrangian duality gap can be decomposed into the primal optimality gap and the constraint violation. Furthermore, we propose an accelerated dual-descent method for entropy-regularized CMDPs. We prove that our method achieves the global convergence rate $\widetilde{\mathcal{O}}(1/T)$ for both the optimality gap and the constraint violation for entropy-regularized CMDPs. A discussion about a linear convergence rate for CMDPs with a single constraint is also provided.

Motivation & Objective

• To establish the theoretical effectiveness of entropy regularization in constrained MDPs from an optimization perspective.
• To analyze the smoothness of the Lagrangian dual function under the Slater condition and exploratory initial distribution.
• To develop a globally convergent algorithm for entropy-regularized CMDPs with provable convergence rates.
• To extend the analysis to single-constraint CMDPs, showing linear convergence under a bisection-based dual method.

Proposed method

• Leverages entropy regularization to induce smoothness in the Lagrangian dual function under the soft-max policy parameterization.
• Uses a projected accelerated gradient descent method to update dual variables (Lagrangian multipliers).
• Employs the natural policy gradient method in the inner loop to optimize the primal policy for each dual iterate.
• Establishes a sandwich inequality to bound the duality gap using the discounted entropy regularization term.
• Applies strong duality and the performance difference lemma to relate primal optimality gap and constraint violation to the dual gap.
• Considers the regularized problem as a warm-start for solving the standard CMDP when the regularization weight $\tau$ is small.

Experimental results

Research questions

• RQ1 Can entropy regularization induce smoothness in the Lagrangian dual function of CMDPs under soft-max parameterization?
• RQ2 What convergence rate can be achieved by a dual-descent method for entropy-regularized CMDPs?
• RQ3 Does the presence of entropy regularization enable faster convergence compared to standard CMDP methods?
• RQ4 Can the duality gap be bounded using the entropy regularization term, and how does it affect the approximation of the standard CMDP?
• RQ5 Is a linear convergence rate achievable for single-constraint CMDPs using a bisection-based dual method?

Key findings

• The Lagrangian dual function is smooth under the Slater condition and exploratory initial distribution, enabling accelerated optimization.
• An $\mathcal{O}(\varepsilon)$ error in the dual optimality gap leads to an $\mathcal{O}(\sqrt{\varepsilon})$ error in both the primal optimality gap and constraint violation.
• The proposed accelerated dual-descent method achieves a global convergence rate of $\widetilde{\mathcal{O}}(1/T)$ for both the optimality gap and constraint violation.
• For CMDPs with a single constraint, a bisection-based dual method achieves a linear convergence rate.
• The duality gap is upper-bounded by $\tau \log|\mathcal{A}| / (1 - \gamma)$, implying that small $\tau$ yields small duality gap and good approximation to the standard CMDP.
• The entropy-regularized solution serves as a warm-start for solving the original CMDP, especially when $\tau = \mathcal{O}(\epsilon)$.