[Paper Review] Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks

Isabell Wochner, Pierre Schumacher+4 more|arXiv (Cornell University)|Jul 8, 2022

Muscle activation and electromyography studies4 citations

TL;DR

This paper demonstrates that muscle-actuated robotic systems significantly outperform torque-actuated systems in data efficiency, robustness, and hyperparameter insensitivity during learning of anthropomorphic tasks. By integrating biologically inspired Hill-type muscle models with reinforcement learning and optimal control, the study shows that nonlinear muscle dynamics—particularly force-velocity and activation dynamics—enable faster convergence and superior performance under perturbations, even when learning from scratch in complex 3D physics simulators.

ABSTRACT

Humans are able to outperform robots in terms of robustness, versatility, and learning of new tasks in a wide variety of movements. We hypothesize that highly nonlinear muscle dynamics play a large role in providing inherent stability, which is favorable to learning. While recent advances have been made in applying modern learning techniques to muscle-actuated systems both in simulation as well as in robotics, so far, no detailed analysis has been performed to show the benefits of muscles when learning from scratch. Our study closes this gap and showcases the potential of muscle actuators for core robotics challenges in terms of data-efficiency, hyperparameter sensitivity, and robustness.

Motivation & Objective

• To investigate whether muscle actuator morphology improves data efficiency and robustness in learning anthropomorphic movements.
• To compare muscle-actuated systems with idealized torque actuators under identical learning conditions.
• To isolate the contribution of individual nonlinear muscle properties—such as force-velocity, force-length, activation dynamics, and moment arm variation—to learning performance.
• To evaluate the impact of muscle morphology across diverse learning algorithms, including reinforcement learning, model predictive control, and optimal control.
• To assess robustness under unknown perturbations, such as added mass to limbs, in realistic simulation environments.

Proposed method

• Employed Hill-type muscle models to simulate nonlinear force-velocity, force-length, and activation dynamics in 3D anthropomorphic robot models.
• Used MuJoCo-based physics simulators with both muscle and torque actuator morphologies for direct comparison.
• Applied state-of-the-art learning methods: reinforcement learning (PPO), model predictive control (MPC), and optimal control (OC) with parameterized control policies.
• Conducted ablation studies by selectively disabling muscle properties (e.g., force-velocity, activation dynamics) to isolate their contributions.
• Performed hyperparameter ablation across multiple tasks and models to assess sensitivity and data efficiency.
• Added unknown mass perturbations (1–5 kg) to limbs to evaluate robustness under uncertainty.

Experimental results

Research questions

• RQ1 Does muscle actuation improve data efficiency in learning complex anthropomorphic movements compared to torque actuation?
• RQ2 Which specific nonlinear muscle properties (e.g., force-velocity, activation dynamics) most significantly contribute to improved learning performance?
• RQ3 How does muscle morphology affect robustness under unknown external perturbations such as added limb mass?
• RQ4 To what extent does muscle actuation reduce sensitivity to hyperparameter choices in reinforcement learning and optimal control?
• RQ5 Can muscle-like dynamics enhance performance even when using baseline control methods like PD control?

Key findings

• Switching off the nonlinear force-velocity relation (no Fv) in muscle models led to performance worse than even the torque-actuated baseline, indicating its critical role in learning efficiency.
• The nonlinear activation dynamics contributed meaningfully to data efficiency, though less than force-velocity, suggesting it plays a stabilizing role in learning.
• Muscle-actuated systems achieved faster convergence and lower cost in point-reaching and locomotion tasks, with up to 50% fewer generations needed to reach optimal performance compared to torque actuators.
• In robustness tests, muscle-actuated systems successfully counteracted perturbations from unknown weights up to 5 kg, while torque-actuated systems exhibited significant overshoots and instability.
• Even with a PD controller baseline, torque-actuated systems did not match the performance of muscle-actuated systems, confirming that muscle morphology provides inherent advantages beyond control strategy.
• The study provides empirical evidence that muscle-like nonlinearities reduce the information processing burden on controllers, enabling more efficient and robust learning from scratch.