[论文解读] Autonomous Penetration Testing using Reinforcement Learning

Jonathon Schwartz, Hanna Kurniawati|arXiv (Cornell University)|May 15, 2019

Advanced Malware Detection Techniques参考文献 37被引用 62

一句话总结

本论文通过构建快速模拟器，评估Q-learning（表格与神经网络）在不依赖环境模型的情况下，在模拟网络中寻找攻击路径，从而将模型无关强化学习应用于自动化渗透测试。

ABSTRACT

Penetration testing (pentesting) involves performing a controlled attack on a computer system in order to assess it's security. Although an effective method for testing security, pentesting requires highly skilled practitioners and currently there is a growing shortage of skilled cyber security professionals. One avenue for alleviating this problem is automate the pentesting process using artificial intelligence techniques. Current approaches to automated pentesting have relied on model-based planning, however the cyber security landscape is rapidly changing making maintaining up-to-date models of exploits a challenge. This project investigated the application of model-free Reinforcement Learning (RL) to automated pentesting. Model-free RL has the key advantage over model-based planning of not requiring a model of the environment, instead learning the best policy through interaction with the environment. We first designed and built a fast, low compute simulator for training and testing autonomous pentesting agents. We did this by framing pentesting as a Markov Decision Process with the known configuration of the network as states, the available scans and exploits as actions, the reward determined by the value of machines on the network. We then used this simulator to investigate the application of model-free RL to pentesting. We tested the standard Q-learning algorithm using both tabular and neural network based implementations. We found that within the simulated environment both tabular and neural network implementations were able to find optimal attack paths for a range of different network topologies and sizes without having a model of action behaviour. However, the implemented algorithms were only practical for smaller networks and numbers of actions. Further work is needed in developing scalable RL algorithms and testing these algorithms in larger and higher fidelity environments.

研究动机与目标

• 推动自动化渗透测试，以应对网络安全专家短缺。
• 研究模型无关强化学习，作为渗透测试中基于模型规划的替代方案。
• 开发一个快速、低计算成本的模拟器，将渗透测试框架化为一个马尔可夫决策过程。
• 评估Q-learning（表格型和神经网络）在发现最优攻击路径中的作用。
• 识别在更大规模、更高保真环境中的可扩展性限制与未来方向。

提出的方法

• 设计一个用于自动化渗透测试的快速模拟器，网络配置作为状态。
• 将渗透测试表述为一个马尔可夫决策过程，扫描和利用作为行动。
• 应用模型无关强化学习，特别是Q-learning，在没有行动行为模型的情况下学习策略。
• 在模拟器中比较表格型和神经网络实现的Q-learning。
• 评估在不同网络拓扑和规模下的性能。
• 讨论对于更大行动空间和网络的实际可行性限制。

实验结果

研究问题

• RQ1 模型无关强化学习是否能够在没有预定义环境模型的情况下学习到最优的渗透测试策略？
• RQ2 表格型和神经网络基础的Q-learning方法是否能扩展到自主渗透测试中的现实网络规模？
• RQ3 在可扩展性和保真度方面，当前的渗透测试强化学习方法有哪些局限？
• RQ4 模拟器框架如何支持对不同网络配置和攻击的评估？

主要发现

• 在所检验的模拟环境中，表格型和神经网络Q-learning均可识别出最优攻击路径。
• 基于RL的代理在不建模行动行为的情况下学习到有效策略。
• 对于较小的网络和动作数，结果有希望，但在更大、更复杂的设置中仍然体现出可扩展性和实际性挑战。
• 研究突出需要可扩展的RL算法和更高保真度的测试环境。