[Paper Review] Situational Awareness by Risk-Conscious Skills

Daniel J. Mankowitz, Aviv Tamar+1 more|arXiv (Cornell University)|Oct 10, 2016

Reinforcement Learning in Robotics28 references3 citations

TL;DR

This paper introduces SARiCoS, a novel hierarchical reinforcement learning algorithm that learns risk-aware skills by optimizing a risk-sensitive objective in a Probabilistic Goal Semi-Markov Decision Process (PG-SMDP). It enables agents to exhibit human-like situational awareness—such as time-wasting in soccer—by learning skills with distinct risk attitudes, effectively mitigating reward-based model misspecification and achieving superior performance in both winning and losing game states.

ABSTRACT

Hierarchical Reinforcement Learning has been previously shown to speed up the convergence rate of RL planning algorithms as well as mitigate feature-based model misspecification (Mankowitz et. al. 2016a,b, Bacon 2015). To do so, it utilizes hierarchical abstractions, also known as skills -- a type of temporally extended action (Sutton et. al. 1999) to plan at a higher level, abstracting away from the lower-level details. We incorporate risk sensitivity, also referred to as Situational Awareness (SA), into hierarchical RL for the first time by defining and learning risk aware skills in a Probabilistic Goal Semi-Markov Decision Process (PG-SMDP). This is achieved using our novel Situational Awareness by Risk-Conscious Skills (SARiCoS) algorithm which comes with a theoretical convergence guarantee. We show in a RoboCup soccer domain that the learned risk aware skills exhibit complex human behaviors such as `time-wasting' in a soccer game. In addition, the learned risk aware skills are able to mitigate reward-based model misspecification.

Motivation & Objective

• To address the lack of risk sensitivity in hierarchical reinforcement learning (H-RL), particularly in mitigating reward-based model misspecification.
• To integrate situational awareness (SA), defined as risk attitude dependent on time or space, into H-RL through a risk-sensitive objective function.
• To develop a framework that enables learning of Risk-Aware Skills (RASs) with distinct risk attitudes, such as risk-averse or risk-seeking behaviors.
• To design an algorithm that learns both the inter-skill policy and the risk-aware parameters (RAPs) of RASs in a PG-SMDP.
• To demonstrate that risk-aware skill learning leads to complex, human-like behaviors such as time-wasting in dynamic environments like RoboCup soccer.

Proposed method

• The paper proposes a Probabilistic Goal Semi-Markov Decision Process (PG-SMDP) to model tasks with time- and risk-dependent objectives.
• It introduces Risk-Aware Skills (RASs), which are parameterized options with an additional Risk-Aware Parameter (RAP) to control risk attitude.
• The SARiCoS algorithm optimizes a risk-sensitive objective, such as Conditional Value-at-Risk (CVaR), to learn both the inter-skill policy and the RAPs of RASs.
• The algorithm is theoretically guaranteed to converge to a locally optimal solution under standard assumptions of policy gradient methods.
• The framework allows for flexible risk criteria, including exponential risk, CVaR, and VaR, enabling adaptation to various risk preferences.
• The method is evaluated in a RoboCup soccer environment, where it learns to switch between attacking and time-wasting behaviors based on game state and risk sensitivity.

Experimental results

Research questions

• RQ1 Can risk-sensitive objectives in hierarchical RL induce complex, human-like behaviors such as time-wasting in a soccer game?
• RQ2 Can risk-aware skill learning mitigate reward-based model misspecification, where incorrect reward shaping leads to suboptimal policies?
• RQ3 Does optimizing for a risk-sensitive objective—such as CVaR—enable agents to learn more robust and contextually appropriate strategies than standard expected return maximization?
• RQ4 Can the same framework support different risk attitudes (e.g., risk-averse vs. risk-seeking) depending on the task state, such as time remaining in a game?
• RQ5 Is there a theoretical convergence guarantee for learning risk-aware skills in a hierarchical RL setting with PG-SMDP?

Key findings

• In the winning scenario, the SARiCoS agent learned to time-waste by dribbling slowly and standing on the ball, achieving an average episode length of 142.3 ± 1.5 steps.
• The SARiCoS agent scored only 1.3 ± 0.6 goals per episode in the winning scenario but prevented the opponent from capturing the ball only 7.3 ± 0.6 times per 100 episodes, indicating effective risk-averse behavior.
• In the losing scenario, the SARiCoS agent achieved 74.3 ± 6.5 goals per episode and an average reward of 6.3 ± 0.2, significantly outperforming the Expected Return (ER) baseline, which scored only 1.7 ± 1.2 goals and had a negative average reward of -0.3 ± 0.1.
• The SARiCoS agent successfully passed the performance threshold β in the losing scenario due to its risk-sensitive objective, which pushed it to seek high-impact rewards like goal scoring, while the ER agent got stuck in a local optimum.
• The SARiCoS agent demonstrated robustness to reward-based model misspecification by avoiding suboptimal behaviors such as excessive dribbling from afar, which trapped the ER agent in a low-reward loop.
• The algorithm achieved theoretical convergence to a locally optimal solution, validating its stability and practicality for risk-aware hierarchical decision-making.