[Paper Review] Efficient reinforcement learning control for continuum robots based on Inexplicit Prior Knowledge

Junjia Liu, Jiaying Shou+6 more|arXiv (Cornell University)|Feb 26, 2020

Soft Robotics and Applications38 references8 citations

TL;DR

This paper proposes a model-based reinforcement learning framework that integrates inexplicit prior knowledge (IPK) to accelerate learning in continuum robots, using a Kalman filter to fuse prior actions with RL outputs and an adaptive exploitation coefficient to balance exploration and exploitation. The method achieves data-efficient, real-world deployment of visual tracking and distance maintenance in a tendon-driven robot with only 10,000–20,000 environment interactions.

ABSTRACT

Compared to rigid robots that are generally studied in reinforcement learning, the physical characteristics of some sophisticated robots such as soft or continuum robots are higher complicated. Moreover, recent reinforcement learning methods are data-inefficient and can not be directly deployed to the robot without simulation. In this paper, we propose an efficient reinforcement learning method based on inexplicit prior knowledge in response to such problems. We first corroborate the method by simulation and employed directly in the real world. By using our method, we can achieve active visual tracking and distance maintenance of a tendon-driven robot which will be critical in minimally invasive procedures. Codes are available at https://github.com/Skylark0924/TendonTrack.

Motivation & Objective

• Address the data inefficiency of reinforcement learning in real-world continuum robots with complex, non-stationary dynamics.
• Overcome the limitations of model-free RL and traditional kinematic modeling in soft and continuum robotics.
• Integrate inexplicit prior knowledge (IPK) to guide policy search without requiring explicit mathematical models.
• Ensure safe and effective exploration by fusing prior knowledge with RL outputs through a Kalman filter-based controller.
• Demonstrate real-world applicability of the method on a tendon-driven continuum robot for minimally invasive surgery tasks.

Proposed method

• Propose a model-based RL framework that incorporates inexplicit prior knowledge (IPK) to guide policy exploration toward plausible action regions.
• Use a Kalman filter to fuse action distributions from IPK and deep RL, producing a hybrid action policy that balances prior knowledge and exploration.
• Introduce an adaptive exploitation coefficient ζ that adjusts based on the Kullback-Leibler (KL) divergence between the IPK and RL action distributions.
• Implement a real-world training pipeline without sim-to-real transfer, directly learning from physical robot interactions.
• Integrate visual servoing from a first-person perspective using a pinhole camera on the robot’s end-effector for real-time state observation.
• Use motor encoders for precise action execution and safety monitoring, preventing mechanical overextension.

Experimental results

Research questions

• RQ1 Can inexplicit prior knowledge significantly reduce sample complexity in reinforcement learning for continuum robots?
• RQ2 How can prior knowledge be effectively fused with deep RL to maintain exploration while accelerating convergence?
• RQ3 Can a Kalman filter-based fusion mechanism stabilize policy learning when combining uncertain prior actions with RL outputs?
• RQ4 To what extent can the proposed method achieve real-world deployment without sim-to-real domain randomization?
• RQ5 Can the method enable both visual tracking and axial distance maintenance in a single-robot system with minimal interaction?

Key findings

• The proposed method achieved stable visual tracking of a moving target in less than 10,000 environment interactions on a real tendon-driven continuum robot.
• With the addition of height-based reward shaping, the robot learned to maintain consistent distance from the target in approximately 20,000 interactions.
• The adaptive exploitation coefficient ζ, based on KL divergence, successfully balanced prior knowledge and exploration, preventing premature convergence.
• Empirical results showed that the method required fewer interactions than state-of-the-art model-based RL methods, demonstrating superior data efficiency.
• The real-world training pipeline without sim-to-real transfer successfully captured real-world uncertainties, enabling robust performance in physical environments.
• The method enabled direct deployment on a physical robot, with successful task execution in minimally invasive surgery-relevant tasks such as tracking and distance control.