[Paper Review] Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act

Alexis Jacq, Johan Ferret+2 more|arXiv (Cornell University)|Mar 16, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces lazy-MDPs, a novel reinforcement learning formalism that extends standard MDPs by adding a 'lazy' action to defer decisions to a default policy, while penalizing non-lazy actions to encourage minimal intervention. The method enables agents to learn when and how to act, resulting in interpretable policies that highlight critical states, achieving competitive or improved performance—especially in hard exploration tasks—while controlling only a small fraction of states.

ABSTRACT

Traditionally, Reinforcement Learning (RL) aims at deciding how to act optimally for an artificial agent. We argue that deciding when to act is equally important. As humans, we drift from default, instinctive or memorized behaviors to focused, thought-out behaviors when required by the situation. To enhance RL agents with this aptitude, we propose to augment the standard Markov Decision Process and make a new mode of action available: being lazy, which defers decision-making to a default policy. In addition, we penalize non-lazy actions in order to encourage minimal effort and have agents focus on critical decisions only. We name the resulting formalism lazy-MDPs. We study the theoretical properties of lazy-MDPs, expressing value functions and characterizing optimal solutions. Then we empirically demonstrate that policies learned in lazy-MDPs generally come with a form of interpretability: by construction, they show us the states where the agent takes control over the default policy. We deem those states and corresponding actions important since they explain the difference in performance between the default and the new, lazy policy. With suboptimal policies as default (pretrained or random), we observe that agents are able to get competitive performance in Atari games while only taking control in a limited subset of states.

Motivation & Objective

• To address the limitation of traditional RL, which focuses only on 'how to act,' by introducing a new paradigm that also considers 'when to act.'
• To enhance interpretability in RL by identifying states where the agent diverges from a default policy, signaling critical decision points.
• To enable efficient learning on top of suboptimal or pretrained policies by learning sparse control over them.
• To study the theoretical properties of lazy-MDPs, including value functions and optimality conditions.
• To empirically validate that lazy-MDPs yield policies that are both effective and interpretable, especially in sparse-reward environments.

Proposed method

• Augment a standard MDP by extending the action space to include a 'lazy' action that defers decision-making to a default policy.
• Modify the reward function to penalize non-lazy actions (via a hyperparameter η), encouraging agents to minimize control frequency.
• Formulate value functions and optimality conditions for lazy-MDPs, showing that optimal policies depend on the default policy and the control penalty.
• Train DQN agents in the lazy-MDP framework using either random or pretrained suboptimal policies as defaults.
• Use statewise differences between the learned policy and the default policy to interpret which states require active control.
• Evaluate performance across Atari games using human-normalized median scores and control frequency metrics, comparing to standard DQN and warm-restart baselines.

Experimental results

Research questions

• RQ1 Can a formalism that learns when to act, rather than just how to act, lead to more interpretable and effective reinforcement learning policies?
• RQ2 How does the frequency of control taking affect performance in both dense- and sparse-reward environments?
• RQ3 Can learning in lazy-MDPs outperform standard training or warm restarts when using a pretrained policy as a default?
• RQ4 To what extent can lazy-MDPs improve exploration in hard-reward tasks by reducing reliance on active control?
• RQ5 How do theoretical properties of lazy-MDPs, such as value functions and optimality, depend on the default policy and control penalty?

Key findings

• Policies trained in lazy-MDPs achieve up to 200% of the DQN baseline score in Frostbite while taking control in less than 10% of states, demonstrating improved exploration.
• In Gravitar, agents maintained nearly 100% of the DQN score with less than 10% control frequency, showing robustness to low intervention rates.
• With a suboptimal DQN agent as default (at 50% performance), lazy-MDPs enabled faster and more stable learning than warm restarts, with no asymptotic performance gain but better sample efficiency.
• When the control penalty η was tuned per game, lazy-MDPs achieved significant performance gains over training from scratch, indicating strong potential for hyperparameter sensitivity.
• The method produced interpretable policies that highlight critical states—those where control is necessary—offering insight into task structure without post-hoc explanation methods.
• In Zaxxon and PrivateEye, agents achieved over 80% and 100% of the DQN baseline score, respectively, with less than 30% control frequency, confirming effectiveness in sparse-reward settings.