[论文解读] Hierarchical visuomotor control of humanoids

Josh Merel, Arun Ahuja+6 more|arXiv (Cornell University)|Nov 23, 2018

Human Pose and Action Recognition参考文献 40被引用 49

一句话总结

本文训练一个分层的 visuomotor 系统，将基于低层 mocap 的运动控制器与高层视觉与记忆控制器结合起来，以从自我视角图像解决高自由度的人形任务。在测试的方法中，离散控制片段切换实现了最佳性能。

ABSTRACT

We aim to build complex humanoid agents that integrate perception, motor control, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies. The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. For a supplementary video link, see https://youtu.be/7GISvfbykLE .

研究动机与目标

• Motivate and demonstrate integrated perception, memory, and motor control for high-DoF humanoids.
• Develop modular agents with a high-level controller that selects among low-level motor skills derived from motion capture.
• Evaluate strategies for interfacing high- and low-level controllers under egocentric vision.
• Assess scalability of control fragments versus structured low-level controllers in visuomotor tasks.

提出的方法

• Train low-level controllers to track short motion-capture clips using an energy-based reward function that penalizes pose, velocity, orientation, end-effector errors, and gyro/velocity deviations.
• Pre-train low-level policies with supervised learning to reach target poses, followed by off-policy reinforcement learning with TD-learning and SVG(0) for further imitation and refinement.
• Interface high-level controllers (HL) with low-level controllers (LL) via either a steerable controller, a switching controller, or control fragments (cold-switching) to select among LL policies.
• High-level observations include egocentric vision (front-facing camera) and proprioception, processed through a linear encoder, a ResNet image encoder, and an LSTM inside an actor-critic framework.
• Evaluate multiple HL-LL interfaces on Go-to-target, Walls, Gaps, Forage, and Heterogeneous Forage tasks in MuJoCo, with varying fragment lengths and numbers.

实验结果

研究问题

• RQ1 Can a hierarchical visuomotor architecture using vision and memory control a high-DoF humanoid to solve tasks from egocentric camera input?
• RQ2 Do low-level mocap-derived motor controllers, when combined with a high-level controller, enable scalable and flexible task-directed motor control?
• RQ3 Which interfacing strategy between high- and low-level controllers best supports learning and execution on visuomotor tasks?
• RQ4 How does the number and quality of control fragments affect performance and learning efficiency in complex navigation and memory tasks?

主要发现

• Control-fragment-based methods with discrete high-level selection achieved the best overall performance across core tasks.
• End-to-end learning from vision succeeded only on Go-to-target and failed on Forage tasks, highlighting limitations without structured motor priors.
• Steerable and switching controllers could solve some tasks but were less flexible or slower to converge than control fragments.
• Memory-enabled heterogeneous forage demonstrated the system’s capacity for memory-guided visuomotor control.
• High-level policies leveraged egocentric vision to select and sequence low-level skills, achieving robust locomotion and obstacle negotiation.