[Paper Review] Machine Learning Techniques for Stackelberg Security Games: a Survey

Giuseppe De Nittis, Francesco Trovò|arXiv (Cornell University)|Sep 29, 2016

Reinforcement Learning in Robotics19 references3 citations

TL;DR

This survey presents machine learning techniques for Stackelberg Security Games (SSGs), focusing on modeling boundedly rational attackers using human behavior, estimating unknown attacker payoffs via regret-based methods, and applying online learning with Whittle index and POMDP-based planning. It introduces GMOP, an algorithm using Gibbs sampling and Monte Carlo Tree Search to improve belief state sampling for optimal protector strategies in resource conservation games.

ABSTRACT

The present survey aims at presenting the current machine learning techniques employed in security games domains. Specifically, we focused on papers and works developed by the Teamcore of University of Southern California, which deepened different directions in this field. After a brief introduction on Stackelberg Security Games (SSGs) and the poaching setting, the rest of the work presents how to model a boundedly rational attacker taking into account her human behavior, then describes how to face the problem of having attacker's payoffs not defined and how to estimate them and, finally, presents how online learning techniques have been exploited to learn a model of the attacker.

Motivation & Objective

• To address the challenge of modeling attackers who deviate from perfect rationality by incorporating human behavior into SSGs.
• To estimate unknown attacker payoffs when they are not directly observable, using defender regret as a proxy.
• To develop online learning methods that adapt defender strategies in real time using historical observations and belief updates.
• To improve decision-making in dynamic, partially observable environments such as wildlife protection and anti-poaching operations.
• To propose a novel online planning algorithm (GMOP) that leverages Gibbs sampling for accurate belief state representation in POMDPs.

Proposed method

• Model boundedly rational attackers using the SUQR and SHARP frameworks, which incorporate adaptive utility functions reflecting human decision-making biases.
• Estimate attacker payoffs by analyzing defender regret, using a regret-based solution to infer utility structures without direct payoff information.
• Formulate the defender's problem as a restless multi-armed bandit (RMAB) problem and apply Whittle index theory to derive indexable policies.
• Propose GMOP, an online planning algorithm that uses Gibbs sampling to draw exact samples from the true belief state, improving upon particle filter approximations.
• Construct a special POMDP with known count states and time-invariant utility states, enabling direct belief sampling via Gibbs sampling.
• Integrate Gibbs sampling with Monte Carlo Tree Search (MCTS) to plan optimal defender strategies under uncertainty, using sampled belief states for action selection.

Experimental results

Research questions

• RQ1 How can attacker behavior be modeled realistically by incorporating bounded rationality and human decision-making biases in SSGs?
• RQ2 What methods can be used to estimate attacker payoffs when they are not directly observable or defined?
• RQ3 How can online learning techniques be applied to dynamically adapt defender strategies in repeated security games with incomplete information?
• RQ4 What is the impact of using exact belief state sampling (via Gibbs sampling) versus approximate methods (e.g., particle filters) on defender strategy quality in POMDP-based planning?
• RQ5 Can Whittle index-based policies be effectively applied to restless bandit formulations in security games to balance exploration and exploitation?

Key findings

• The SUQR and SHARP models successfully capture bounded rationality in attackers, improving the realism and performance of defender strategies in SSGs.
• Regret-based methods enable estimation of attacker payoffs without requiring explicit knowledge of utility functions, allowing for effective strategy computation.
• The Whittle index approach provides a sufficient condition for indexability in restless bandit formulations, enabling efficient online learning in dynamic security games.
• GMOP outperforms particle filter-based sampling by using Gibbs sampling to draw exact samples from the true belief state, leading to more accurate and stable policy learning.
• The integration of Gibbs sampling with MCTS in GMOP results in better action selection and improved expected utility in partially observable security games.
• The special POMDP structure with known count states and fixed utility states allows for efficient belief updates and direct sampling, making online planning feasible and scalable.