[Paper Review] PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning

Junfeng Guo, Ang Li+1 more|arXiv (Cornell University)|Feb 8, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

PolicyCleanse proposes a novel backdoor detection and mitigation framework for multi-agent competitive reinforcement learning, leveraging reward degradation under adversarial action sequences to identify Trojan triggers. It achieves superior detection accuracy and mitigates backdoors with a 3%+ improvement in winning rate over baselines.

ABSTRACT

While real-world applications of reinforcement learning are becoming popular, the security and robustness of RL systems are worthy of more attention and exploration. In particular, recent works have revealed that, in a multi-agent RL environment, backdoor trigger actions can be injected into a victim agent (a.k.a. Trojan agent), which can result in a catastrophic failure as soon as it sees the backdoor trigger action. To ensure the security of RL agents against malicious backdoors, in this work, we propose the problem of Backdoor Detection in a multi-agent competitive reinforcement learning system, with the objective of detecting Trojan agents as well as the corresponding potential trigger actions, and further trying to mitigate their Trojan behavior. In order to solve this problem, we propose PolicyCleanse that is based on the property that the activated Trojan agents accumulated rewards degrade noticeably after several timesteps. Along with PolicyCleanse, we also design a machine unlearning-based approach that can effectively mitigate the detected backdoor. Extensive experiments demonstrate that the proposed methods can accurately detect Trojan agents, and outperform existing backdoor mitigation baseline approaches by at least 3% in winning rate across various types of agents and environments.

Motivation & Objective

• To address the growing security risk of backdoor attacks in multi-agent competitive reinforcement learning (CRL), where attackers inject triggers via opponent actions.
• To detect Trojan agents that fail catastrophically when exposed to specific action sequences from an attacker.
• To mitigate detected backdoors through a machine unlearning-based retraining approach.
• To overcome the challenge of unknown, variable-length continuous action triggers in CRL environments.
• To ensure robustness against adaptive attacks that attempt to evade detection by slowing failure speed.

Proposed method

• PolicyCleanse detects backdoors by optimizing a reversed reward policy to identify action sequences that cause rapid reward degradation in the victim agent.
• It uses a binary search over expected cumulative rewards to localize the trigger action sequence, iteratively narrowing the threshold range.
• The method employs parallelized policy optimization with random initializations to increase detection success rate across diverse environments.
• Once a pseudo-trigger is identified, the victim policy is retrained on a mixed dataset of benign and pseudo-triggered trajectories to unlearn the Trojan behavior.
• A machine unlearning-based mitigation strategy is applied, which outperforms existing baselines in winning rate.
• The approach is evaluated under adaptive attacks where the attacker modifies the failure mechanism to evade detection, simulating real-world evasion attempts.

Experimental results

Research questions

• RQ1 Can backdoor triggers in competitive RL be detected by observing abnormal reward degradation when the victim agent encounters specific opponent actions?
• RQ2 How effective is PolicyCleanse in detecting backdoors across diverse CRL environments and agent architectures?
• RQ3 To what extent does the proposed mitigation strategy restore the victim agent’s performance compared to existing baselines?
• RQ4 How does PolicyCleanse perform under adaptive attacks that slow down the failure response to evade detection?
• RQ5 Can the detection mechanism distinguish between natural agent behaviors and artificial backdoor triggers?

Key findings

• PolicyCleanse successfully detects all Trojan agents across four different competitive environments and multiple agent architectures with high accuracy.
• The method outperforms existing backdoor mitigation baselines by at least 3% in winning rate across all evaluated environments.
• Adaptive attacks that attempt to evade detection by slowing failure speed reduce attack efficacy by at least 19.6% compared to standard BackdooRL.
• The detected pseudo-triggers are distinct from benign actions and not artifacts of natural agent behavior, as confirmed by ablation and visualization.
• The mitigation strategy based on unlearning effectively restores agent performance, with the victim agent regaining strong competitive capability.
• Even when adaptive attacks bypass detection, the resulting failure speed is significantly slower, reducing stealth and real-world viability.