[Paper Review] Open-Sourced Reinforcement Learning Environments for Surgical Robotics

Florian Richter, Ryan K. Orosco+1 more|arXiv (Cornell University)|Mar 5, 2019

Reinforcement Learning in Robotics36 references64 citations

TL;DR

The paper introduces dVRL, the first open-sourced RL environments for the da Vinci Research Kit, enabling easy prototyping of RL algorithms and transfer of learned policies to real robots. It demonstrates DDPG+HER with behavioral cloning and shows successful sim-to-real transfer for suction and debris removal tasks.

ABSTRACT

Reinforcement Learning (RL) is a machine learning framework for artificially intelligent systems to solve a variety of complex problems. Recent years has seen a surge of successes solving challenging games and smaller domain problems, including simple though non-specific robotic manipulation and grasping tasks. Rapid successes in RL have come in part due to the strong collaborative effort by the RL community to work on common, open-sourced environment simulators such as OpenAI's Gym that allow for expedited development and valid comparisons between different, state-of-art strategies. In this paper, we aim to start the bridge between the RL and the surgical robotics communities by presenting the first open-sourced reinforcement learning environments for surgical robots, called dVRL[3]{dVRL available at https://github.com/ucsdarclab/dVRL}. Through the proposed RL environments, which are functionally equivalent to Gym, we show that it is easy to prototype and implement state-of-art RL algorithms on surgical robotics problems that aim to introduce autonomous robotic precision and accuracy to assisting, collaborative, or repetitive tasks during surgery. Learned policies are furthermore successfully transferable to a real robot. Finally, combining dVRL with the over 40+ international network of da Vinci Surgical Research Kits in active use at academic institutions, we see dVRL as enabling the broad surgical robotics community to fully leverage the newest strategies in reinforcement learning, and for reinforcement learning scientists with no knowledge of surgical robotics to test and develop new algorithms that can solve the real-world, high-impact challenges in autonomous surgery.

Motivation & Objective

• Bridge RL and surgical robotics by providing open-source RL environments for the da Vinci Research Kit (dVRL).
• Demonstrate that policies learned in simulation transfer effectively to the real dVRK system.
• Showcase how modern RL techniques (DDPG, HER) can solve surgical-style reach and pick tasks.
• Enable non-domain experts to experiment with RL in a realistic surgical context.
• Highlight the potential for broad community collaboration through standardized RL interfaces.

Proposed method

• Environments inherit OpenAI Gym API and use V-REP for physics simulation in synchronous mode.
• End-effector space control with a single PSM arm and tool-agnostic actions to enable tool transferability.
• Two goal-oriented tasks: PSM Reach (move end-effector to a goal) and PSM Pick (grasp object and move to goal).
• DDPG with HER is used to learn policies, with behavioral cloning augmentation to improve exploration and sample efficiency.
• Policies learned in simulation are transferred to the real dVRK using forward kinematics and DH-parameter-informed tool integration.

Experimental results

Research questions

• RQ1 Can RL policies learned in a lightweight simulator transfer to the real dVRK with minimal adaptation?
• RQ2 Do end-effector space policies generalize across different EndoWrists and tools?
• RQ3 How do modern RL techniques (DDPG+HER, BC) perform on reach and pick tasks in a surgical robotics setting?
• RQ4 What are the practical gains in sim-to-real transfer for autonomous or assisted surgical subtasks?

Key findings

• DDPG+HER with behavioral cloning solved both PSM Reach and PSM Pick in simulation.
• Policies transferred to the real dVRK at 50 Hz demonstrated successful task completion in suction and debris removal scenarios.
• The PSM Reach policy achieved 100% success when given 1000 simulation steps (vs. 100 steps in training).
• Parallelized simulation using Docker containers enhances training efficiency (faster rollout times).
• The suction tool and LND could execute learned policies, enabling autonomous debris removal and handoff to the assistant in a recreated abdominal scene.