[Paper Review] Direct Behavior Specification via Constrained Reinforcement Learning

Julien Roy, Roger Girgis+3 more|arXiv (Cornell University)|Dec 22, 2021

Advanced Software Engineering Methodologies4 citations

TL;DR

This paper proposes using Constrained Reinforcement Learning (CRL) with Lagrangian methods to directly specify desired agent behaviors via frequency-of-event constraints, avoiding the trial-and-error reward engineering typical in RL. The approach enables scalable, stable training across multiple constraints—demonstrated on complex navigation tasks in video game-like environments—achieving feasible, high-performing policies without extensive hyperparameter tuning.

ABSTRACT

The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most often, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied RL projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods to automatically weigh each of these behavioral constraints. Specifically, we investigate how CMDPs can be adapted to solve goal-based tasks while adhering to several constraints simultaneously. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

Motivation & Objective

• To address the impracticality of reward engineering in real-world RL applications, where manual reward shaping is error-prone and requires extensive hyperparameter tuning.
• To demonstrate that Constrained Markov Decision Processes (CMDPs) can serve as a more intuitive and scalable alternative for specifying behavioral preferences in RL.
• To develop a stable, scalable algorithm that can jointly satisfy multiple behavioral constraints while maintaining performance on the primary task.
• To evaluate the framework on realistic, complex environments resembling video game NPC design, where behavioral fidelity is critical.
• To show that bootstrap constraints can help avoid trivial policies when multiple constraints make the main task infeasible to solve.

Proposed method

• Formulate behavioral preferences as indicator functions that track the frequency of specific events (e.g., looking at a marker, avoiding lava).
• Integrate these behavioral objectives as constraints in a CMDP framework, with each constraint having a threshold on event frequency.
• Use a Lagrangian relaxation approach with multiplier normalization to stabilize training and balance constraint enforcement.
• Adapt SAC (Soft Actor-Critic) with Lagrangian multipliers to jointly optimize the main task reward and constraint penalties.
• Introduce a success constraint as a bootstrap mechanism to guide the agent toward feasible policies early in training.
• Train the agent in both a controlled Arena environment and a complex OpenWorld environment with dynamic terrain and interactive objects.

Experimental results

Research questions

• RQ1 Can behavioral constraints expressed as event frequency thresholds be effectively used to shape agent behavior without reward engineering?
• RQ2 How does the performance of CRL compare to traditional reward engineering when multiple constraints are applied simultaneously?
• RQ3 Can Lagrangian CRL with multiplier normalization stabilize training and improve convergence when enforcing many constraints?
• RQ4 Does the use of a bootstrap success constraint help the agent escape trivial, constraint-satisfying policies and achieve better main task performance?
• RQ5 Can the proposed framework scale to complex, realistic environments resembling video game NPC behavior?

Key findings

• Reward engineering required extensive hyperparameter search: only 2 out of 49 experiments with two constraints and none of 343 experiments with three constraints yielded good-performing feasible policies.
• The proposed CRL framework with Lagrangian multipliers and normalization successfully satisfied all constraints while solving the navigation task in the Arena environment, even under multiple constraints.
• In the OpenWorld environment, the agent successfully completed the navigation task with four behavioral constraints (on-ground, not in lava, looking at marker, above energy limit) after up to 50M training steps, demonstrating scalability.
• Introducing a bootstrap success constraint significantly improved policy quality by guiding the agent toward feasible and high-performing solutions, avoiding trivial constraint-satisfying behaviors.
• The method achieved stable training and consistent constraint satisfaction across environments, outperforming reward engineering in both reliability and design efficiency.
• The framework is easily implementable on existing policy gradient codebases and shows strong potential for industrial RL applications in game AI and beyond.