[논문 리뷰] Open-Sourced Reinforcement Learning Environments for Surgical Robotics

Florian Richter, Ryan K. Orosco외 1명|arXiv (Cornell University)|2019. 03. 05.

Reinforcement Learning in Robotics참고 문헌 36인용 수 64

한 줄 요약

이 논문은 da Vinci Research Kit용 최초의 오픈 소스 RL 환경인 dVRL을 도입하여 RL 알고리즘의 빠른 프로토타이핑과 학습된 정책의 실제 로봇으로의 전이를 가능하게 한다. 또한 DDPG+HER를 행동 복제와 함께 활용하고 흡입 및 파편 제거 작업에 대한 시뮬레이션-현실 간 전이가 성공적으로 이루어짐을 보여준다.

ABSTRACT

Reinforcement Learning (RL) is a machine learning framework for artificially intelligent systems to solve a variety of complex problems. Recent years has seen a surge of successes solving challenging games and smaller domain problems, including simple though non-specific robotic manipulation and grasping tasks. Rapid successes in RL have come in part due to the strong collaborative effort by the RL community to work on common, open-sourced environment simulators such as OpenAI's Gym that allow for expedited development and valid comparisons between different, state-of-art strategies. In this paper, we aim to start the bridge between the RL and the surgical robotics communities by presenting the first open-sourced reinforcement learning environments for surgical robots, called dVRL[3]{dVRL available at https://github.com/ucsdarclab/dVRL}. Through the proposed RL environments, which are functionally equivalent to Gym, we show that it is easy to prototype and implement state-of-art RL algorithms on surgical robotics problems that aim to introduce autonomous robotic precision and accuracy to assisting, collaborative, or repetitive tasks during surgery. Learned policies are furthermore successfully transferable to a real robot. Finally, combining dVRL with the over 40+ international network of da Vinci Surgical Research Kits in active use at academic institutions, we see dVRL as enabling the broad surgical robotics community to fully leverage the newest strategies in reinforcement learning, and for reinforcement learning scientists with no knowledge of surgical robotics to test and develop new algorithms that can solve the real-world, high-impact challenges in autonomous surgery.

연구 동기 및 목표

• Bridge RL and surgical robotics by providing open-source RL environments for the da Vinci Research Kit (dVRL).
• Demonstrate that policies learned in simulation transfer effectively to the real dVRK system.
• Showcase how modern RL techniques (DDPG, HER) can solve surgical-style reach and pick tasks.
• Enable non-domain experts to experiment with RL in a realistic surgical context.
• Highlight the potential for broad community collaboration through standardized RL interfaces.

제안 방법

• Environments inherit OpenAI Gym API and use V-REP for physics simulation in synchronous mode.
• End-effector space control with a single PSM arm and tool-agnostic actions to enable tool transferability.
• Two goal-oriented tasks: PSM Reach (move end-effector to a goal) and PSM Pick (grasp object and move to goal).
• DDPG with HER is used to learn policies, with behavioral cloning augmentation to improve exploration and sample efficiency.
• Policies learned in simulation are transferred to the real dVRK using forward kinematics and DH-parameter-informed tool integration.

실험 결과

연구 질문

• RQ1 Can RL policies learned in a lightweight simulator transfer to the real dVRK with minimal adaptation?
• RQ2 Do end-effector space policies generalize across different EndoWrists and tools?
• RQ3 How do modern RL techniques (DDPG+HER, BC) perform on reach and pick tasks in a surgical robotics setting?
• RQ4 What are the practical gains in sim-to-real transfer for autonomous or assisted surgical subtasks?

주요 결과

• DDPG+HER with behavioral cloning solved both PSM Reach and PSM Pick in simulation.
• Policies transferred to the real dVRK at 50 Hz demonstrated successful task completion in suction and debris removal scenarios.
• The PSM Reach policy achieved 100% success when given 1000 simulation steps (vs. 100 steps in training).
• Parallelized simulation using Docker containers enhances training efficiency (faster rollout times).
• The suction tool and LND could execute learned policies, enabling autonomous debris removal and handoff to the assistant in a recreated abdominal scene.