[Paper Review] Learning to Locomote with Deep Neural-Network and CPG-based Control in a Soft Snake Robot

Xuan Liu, Renato Gasoto+2 more|arXiv (Cornell University)|Jan 13, 2020

Soft Robotics and Applications36 references4 citations

TL;DR

This paper proposes a hybrid deep reinforcement learning (PPOC-CPG) and central pattern generator (CPG) control framework for soft snake robots, enabling adaptive goal tracking and diverse locomotion. By coupling a PPO-based policy with Matsuoka oscillator CPGs that independently control frequency and amplitude, the system achieves stable, efficient, and adaptive locomotion in simulation and real-world experiments, with the policy generalizing to dynamic goals after 6,400 training episodes.

ABSTRACT

In this paper, we present a new locomotion control method for soft robot snakes. Inspired by biological snakes, our control architecture is composed of two key modules: A deep reinforcement learning (RL) module for achieving adaptive goal-tracking behaviors with changing goals, and a central pattern generator (CPG) system with Matsuoka oscillators for generating stable and diverse locomotion patterns. The two modules are interconnected into a closed-loop system: The RL module, analogizing the locomotion region located in the midbrain of vertebrate animals, regulates the input to the CPG system given state feedback from the robot. The output of the CPG system is then translated into pressure inputs to pneumatic actuators of the soft snake robot. Based on the fact that the oscillation frequency and wave amplitude of the Matsuoka oscillator can be independently controlled under different time scales, we further adapt the option-critic framework to improve the learning performance measured by optimality and data efficiency. The performance of the proposed controller is experimentally validated with both simulated and real soft snake robots.

Motivation & Objective

• To develop a bio-inspired control framework that enables soft snake robots to achieve adaptive, goal-tracking locomotion in dynamic environments.
• To address the challenge of controlling soft robots with high DOF and nonlinear actuator dynamics using a hybrid learning and oscillatory control approach.
• To improve data efficiency and learning performance in reinforcement learning by exploiting the time-scale separation of Matsuoka oscillator parameters (frequency and amplitude).
• To validate the controller’s robustness and generalization in both simulated and real-world soft snake robot platforms.

Proposed method

• A deep reinforcement learning agent (PPOC-CPG) is trained to regulate two inputs to a CPG network: neural stimuli (for phase and amplitude modulation) and frequency ratio (for velocity control).
• The CPG network is composed of coupled Matsuoka oscillators, whose oscillation frequency and wave amplitude can be independently controlled, enabling diverse locomotion patterns.
• The system uses a curriculum learning strategy with a termination function and dynamic adjustment of the frequency gain $K_f$ to improve sample efficiency and convergence.
• Domain randomization is applied during simulation training to reduce sim-to-real domain gap by sampling physical parameters like friction from a distribution around measured values.
• The policy is transferred directly to a real soft snake robot using a motion capture system for real-time state feedback and low-latency wireless control.
• The controller is evaluated in goal-reaching tasks with both static and slowly moving targets, demonstrating robustness and adaptability.

Experimental results

Research questions

• RQ1 Can a hybrid deep reinforcement learning and CPG-based control system enable adaptive goal tracking in soft snake robots with complex, nonlinear dynamics?
• RQ2 How can the independent control of Matsuoka oscillator frequency and amplitude improve locomotion diversity and learning efficiency?
• RQ3 To what extent can a policy trained in simulation generalize to real-world soft robot locomotion with minimal sim-to-real transfer effort?
• RQ4 Can curriculum learning and domain randomization enhance data efficiency and convergence in training for soft robotic locomotion?

Key findings

• The PPOC-CPG controller achieved convergence to a near-optimal policy after 6,400 training episodes, completing the task in approximately 12 hours using 4 parallel robots.
• The policy successfully tracked both static and slowly moving goals in simulation and real-world experiments, demonstrating generalization beyond fixed-goal training.
• The real robot achieved an average speed of 0.092 m/s, closely matching the simulated robot’s average speed of 0.083 m/s under tuned friction parameters.
• The controller exhibited a two-phase locomotion strategy: high-speed travel at low frequency when far from the goal, switching to higher frequency for precise approach and turning.
• The system demonstrated robustness to simulation inaccuracies through domain randomization, improving real-world transfer performance.
• A delay in state evolution was observed between simulation and real robot, likely due to inaccurate contact dynamics modeling, suggesting a need for improved physical modeling or adaptive feedback mechanisms.