[Paper Review] Learning and Transfer of Modulated Locomotor Controllers

Nicolas Heess, Gregory Wayne+4 more|arXiv (Cornell University)|Oct 17, 2016

Robotic Locomotion and Control25 references105 citations

TL;DR

The paper presents a hierarchical motor-control architecture with a pre-trained low-level spinal module whose outputs are modulated by a high-level cortical controller, enabling robust transfer to diverse locomotion tasks with sparse rewards where end-to-end learning fails.

ABSTRACT

We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at https://youtu.be/sboPYvhpraQ

Motivation & Objective

• Motivate modular, hierarchical motor control that mirrors biological division between spinal primitives and cortical modulation.
• Develop a two-level controller where a fixed low-level module is modulated by a high-level controller to drive behavior.
• Enable transfer to new tasks by freezing the low-level controller and training a high-level modulator.
• Demonstrate the approach on multiple simulated bodies and transfer tasks with sparse rewards.

Proposed method

• Propose a two-level architecture: a low-level spinal controller operating at high-frequency with proprioceptive input and a high-level cortical controller operating at a slower timescale with full observations.
• High-level controller outputs a modulation signal c_t that biases the low-level controller; c_t is updated every K steps and can be stochastic to induce correlated exploration.
• Use a generalized policy gradient (actor-critic) with value function baselines and lambda-returns (R_t^λ) to train both pre-training and transfer phases.
• Adopt the re-parameterization trick to backpropagate through the stochastic high-level modulation, enabling gradients to flow into the high-level controller.
• Pre-train the low-level controller on a simple locomotion task with shaping rewards, then freeze it and train a high-level modulator for transfer tasks with sparse rewards.
• Compare the modular approach to end-to-end (FF/LSTM) baselines and variants such as reusing a pre-trained FF network or initializing with new input mappings.

Experimental results

Research questions

• RQ1 Can a fixed, pre-trained low-level locomotor primitive, when modulated by a higher-level network, solve complex tasks with sparse rewards that end-to-end learning struggles with?
• RQ2 Does hierarchical noise via the high-level controller promote more coherent and effective exploration than action-space noise alone?
• RQ3 To what extent can learned low-level primitives transfer across diverse morphologies (snake, quadruped, humanoid) and tasks (target-seeking, canyon traversal, soccer, slalom)?
• RQ4 How do time-scale separation and information hiding contribute to robust reuse of locomotor primitives across tasks?

Key findings

• A pre-trained low-level locomotor controller, when modulated by a slower high-level controller, solves several transfer tasks where end-to-end learning fails.
• Hierarchical noise induces spatially and temporally correlated exploration at the primitive level, improving exploration in sparse-reward tasks.
• Across three bodies (snake, quadruped, humanoid) and multiple tasks (target-seeking, canyon traversal, soccer, slalom), the modular approach demonstrates effective transfer while end-to-end learning struggles.
• The low-level primitives exhibit coherent, reusable locomotor behaviors that can be invoked and modulated to achieve diverse goals.