[Paper Review] Hierarchical visuomotor control of humanoids

Josh Merel, Arun Ahuja+6 more|arXiv (Cornell University)|Nov 23, 2018

Human Pose and Action Recognition40 references49 citations

TL;DR

The paper trains a hierarchical visuomotor system that combines low-level mocap-derived motor controllers with a high-level vision-and-memory controller to solve high-DoF humanoid tasks from egocentric vision. Discrete control-fragment switching yields the best performance among tested approaches.

ABSTRACT

We aim to build complex humanoid agents that integrate perception, motor control, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies. The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. For a supplementary video link, see https://youtu.be/7GISvfbykLE .

Motivation & Objective

• Motivate and demonstrate integrated perception, memory, and motor control for high-DoF humanoids.
• Develop modular agents with a high-level controller that selects among low-level motor skills derived from motion capture.
• Evaluate strategies for interfacing high- and low-level controllers under egocentric vision.
• Assess scalability of control fragments versus structured low-level controllers in visuomotor tasks.

Proposed method

• Train low-level controllers to track short motion-capture clips using an energy-based reward function that penalizes pose, velocity, orientation, end-effector errors, and gyro/velocity deviations.
• Pre-train low-level policies with supervised learning to reach target poses, followed by off-policy reinforcement learning with TD-learning and SVG(0) for further imitation and refinement.
• Interface high-level controllers (HL) with low-level controllers (LL) via either a steerable controller, a switching controller, or control fragments (cold-switching) to select among LL policies.
• High-level observations include egocentric vision (front-facing camera) and proprioception, processed through a linear encoder, a ResNet image encoder, and an LSTM inside an actor-critic framework.
• Evaluate multiple HL-LL interfaces on Go-to-target, Walls, Gaps, Forage, and Heterogeneous Forage tasks in MuJoCo, with varying fragment lengths and numbers.

Experimental results

Research questions

• RQ1 Can a hierarchical visuomotor architecture using vision and memory control a high-DoF humanoid to solve tasks from egocentric camera input?
• RQ2 Do low-level mocap-derived motor controllers, when combined with a high-level controller, enable scalable and flexible task-directed motor control?
• RQ3 Which interfacing strategy between high- and low-level controllers best supports learning and execution on visuomotor tasks?
• RQ4 How does the number and quality of control fragments affect performance and learning efficiency in complex navigation and memory tasks?

Key findings

• Control-fragment-based methods with discrete high-level selection achieved the best overall performance across core tasks.
• End-to-end learning from vision succeeded only on Go-to-target and failed on Forage tasks, highlighting limitations without structured motor priors.
• Steerable and switching controllers could solve some tasks but were less flexible or slower to converge than control fragments.
• Memory-enabled heterogeneous forage demonstrated the system’s capacity for memory-guided visuomotor control.
• High-level policies leveraged egocentric vision to select and sequence low-level skills, achieving robust locomotion and obstacle negotiation.