[论文解读] Open-Sourced Reinforcement Learning Environments for Surgical Robotics

Florian Richter, Ryan K. Orosco+1 more|arXiv (Cornell University)|Mar 5, 2019

Reinforcement Learning in Robotics参考文献 36被引用 64

一句话总结

本文介绍 dVRL，这是首个开源的用于 Vinci 研究套件的 RL 环境，便于原型化 RL 算法并将学习策略转移到真实机器人。它演示了 DDPG+HER 结合行为克隆，并展示了吸取/清除碎片任务的 sim-to-real 转移成功。

ABSTRACT

Reinforcement Learning (RL) is a machine learning framework for artificially intelligent systems to solve a variety of complex problems. Recent years has seen a surge of successes solving challenging games and smaller domain problems, including simple though non-specific robotic manipulation and grasping tasks. Rapid successes in RL have come in part due to the strong collaborative effort by the RL community to work on common, open-sourced environment simulators such as OpenAI's Gym that allow for expedited development and valid comparisons between different, state-of-art strategies. In this paper, we aim to start the bridge between the RL and the surgical robotics communities by presenting the first open-sourced reinforcement learning environments for surgical robots, called dVRL[3]{dVRL available at https://github.com/ucsdarclab/dVRL}. Through the proposed RL environments, which are functionally equivalent to Gym, we show that it is easy to prototype and implement state-of-art RL algorithms on surgical robotics problems that aim to introduce autonomous robotic precision and accuracy to assisting, collaborative, or repetitive tasks during surgery. Learned policies are furthermore successfully transferable to a real robot. Finally, combining dVRL with the over 40+ international network of da Vinci Surgical Research Kits in active use at academic institutions, we see dVRL as enabling the broad surgical robotics community to fully leverage the newest strategies in reinforcement learning, and for reinforcement learning scientists with no knowledge of surgical robotics to test and develop new algorithms that can solve the real-world, high-impact challenges in autonomous surgery.

研究动机与目标

• 通过为 da Vinci Research Kit (dVRL) 提供开源 RL 环境，连接 RL 与手术机器人学。
• 证明在仿真中学习的策略能有效转移到真实的 dVRK 系统。
• 展示现代 RL 技术（DDPG、HER）如何解决手术风格的到达与拾取任务。
• 使非领域专家能够在真实的手术环境中实验 RL。
• 强调通过标准化的 RL 接口实现广泛社区协作的潜力。

提出的方法

• 环境继承 OpenAI Gym API，并在同步模式下使用 V-REP 进行物理仿真。
• 使用单个 PSM 臂的末端执行器空间控制和与工具无关的动作，以实现工具可转移性。
• 两项面向目标的任务：PSM Reach（将末端执行器移动到目标）和 PSM Pick（抓取对象并移动到目标）。
• 使用结合 HER 的 DDPG 来学习策略，并通过行为克隆增强来提升探索和样本效率。
• 在仿真中学习的策略通过正向运动学和 DH-参数知情的工具集成转移到真实的 dVRK。

实验结果

研究问题

• RQ1 在轻量级仿真器中学习的 RL 策略是否能够在最少调整的情况下转移到真实的 dVRK？
• RQ2 末端执行器空间策略是否能在不同的 EndoWrists 和工具之间泛化？
• RQ3 现代 RL 技术（DDPG+HER、BC）在手术机器人环境中的到达和拾取任务上的表现如何？
• RQ4 在自主或辅助手术子任务的仿真到真实转移方面有哪些实际收益？

主要发现

• 在仿真中，DDPG+HER 结合行为克隆解决了 PSM Reach 和 PSM Pick。
• 以 50 Hz 转移到真实 dVRK 的策略在吸力和碎片清除场景中的任务成功完成。
• 当给定 1000 仿真步骤时，PSM Reach 策略实现了 100% 成功率（相比训练中的 100 步）。
• 使用 Docker 容器并行化仿真可提高训练效率（更快的 rollout 时长）。
• 吸 tool 和 LND 能执行学习得到的策略，实现自主清除碎片并在重现的腹腔场景中将任务交接给助手。