[论文解读] A Search-Based Testing Approach for Deep Reinforcement Learning Agents

Amirhossein Zolfagharian, Manel Abdellatif+3 more|arXiv (Cornell University)|Jun 15, 2022

Adversarial Robustness in Machine Learning被引用 7

一句话总结

本文提出STARLA，一种基于搜索的深度强化学习（DRL）智能体测试方法，利用机器学习模型引导的遗传算法，高效检测DRL策略中的功能故障。通过应用状态抽象以降低状态空间复杂度，STARLA在有限测试预算下，相比随机测试更有效地识别出故障episode，显著提升了故障检测效率。

ABSTRACT

Deep Reinforcement Learning (DRL) algorithms have been increasingly employed during the last decade to solve various decision-making problems such as autonomous driving and robotics. However, these algorithms have faced great challenges when deployed in safety-critical environments since they often exhibit erroneous behaviors that can lead to potentially critical errors. One way to assess the safety of DRL agents is to test them to detect possible faults leading to critical failures during their execution. This raises the question of how we can efficiently test DRL policies to ensure their correctness and adherence to safety requirements. Most existing works on testing DRL agents use adversarial attacks that perturb states or actions of the agent. However, such attacks often lead to unrealistic states of the environment. Their main goal is to test the robustness of DRL agents rather than testing the compliance of agents' policies with respect to requirements. Due to the huge state space of DRL environments, the high cost of test execution, and the black-box nature of DRL algorithms, the exhaustive testing of DRL agents is impossible. In this paper, we propose a Search-based Testing Approach of Reinforcement Learning Agents (STARLA) to test the policy of a DRL agent by effectively searching for failing executions of the agent within a limited testing budget. We use machine learning models and a dedicated genetic algorithm to narrow the search towards faulty episodes. We apply STARLA on Deep-Q-Learning agents which are widely used as benchmarks and show that it significantly outperforms Random Testing by detecting more faults related to the agent's policy. We also investigate how to extract rules that characterize faulty episodes of the DRL agent using our search results. Such rules can be used to understand the conditions under which the agent fails and thus assess its deployment risks.

研究动机与目标

• 为解决在安全关键环境中测试DRL智能体所面临的挑战，因为功能故障可能导致严重故障。
• 克服现有对抗性测试方法的局限性，这些方法生成不现实的状态，且侧重于鲁棒性而非策略合规性。
• 开发一种高效、黑盒的测试方法，无需访问内部模型即可检测DRL策略中的功能故障。
• 实现从故障episode中提取可解释规则，以支持风险评估和策略改进。
• 通过状态抽象和机器学习引导的遗传搜索缩小搜索空间，降低测试成本。

提出的方法

• 使用专门设计的遗传算法演化可能触发DRL智能体功能故障的状态与动作序列。
• 训练一个机器学习模型以预测给定episode是否为故障，从而引导遗传算法向状态空间中高故障概率区域搜索。
• 应用状态抽象技术将相似状态分组，降低状态空间维度，提升模型的可学习性。
• 该方法采用黑盒测试范式，仅需访问环境和智能体策略，无需内部模型参数或训练数据。
• 基于抽象后的状态空间设计定制化的遗传算子，以提升搜索效率和收敛性。
• 对故障episode进行分析，提取可解释规则，描述智能体失败的条件。

实验结果

研究问题

• RQ1 在有限测试预算下，基于搜索的方法是否能优于随机测试，更有效地检测DRL智能体的功能故障？
• RQ2 状态抽象在降低状态空间复杂度的同时，是否仍能有效保持故障检测能力？
• RQ3 基于episode特征训练的机器学习模型是否能有效引导遗传搜索，使其聚焦于故障高发区域？
• RQ4 从搜索结果中可提取出何种可解释规则，以表征DRL智能体的失败条件？
• RQ5 所识别的故障episode在多大程度上可用于通过重训练改进智能体策略？

主要发现

• 在相同测试预算下，STARLA检测到的功能故障数量显著多于随机测试，证明其在故障发现方面具有更高的效率。
• 状态抽象的使用降低了状态空间复杂度，使故障预测模型能更有效地训练，并提升了搜索收敛性。
• 基于episode特征训练的机器学习模型成功引导遗传算法聚焦于高故障概率区域，显著提高了识别故障episode的可能性。
• 从故障episode中成功提取出可解释规则，揭示了智能体失败的具体环境与状态条件。
• 该方法在故障检测率和搜索效率方面均优于基线随机测试，尤其在存在罕见但关键故障模式的环境中表现更优。
• 结果表明，所识别的故障episode可为重训练智能体以提升策略鲁棒性和安全性提供重要价值。