[Paper Review] Lyceum: An efficient and scalable ecosystem for robot learning

Colin Summers, Kendall Lowrey+3 more|arXiv (Cornell University)|Jan 21, 2020

Reinforcement Learning in Robotics15 references4 citations

TL;DR

Lyceum is a high-performance computational ecosystem for robot learning built on Julia and MuJoCo, enabling 5–30× faster training than Gym and dm-control by combining high-level abstraction with zero-cost performance. It accelerates reinforcement learning and real-time model predictive control through efficient memory management, in-place operations, and native parallelism.

ABSTRACT

We introduce Lyceum, a high-performance computational ecosystem for robot learning. Lyceum is built on top of the Julia programming language and the MuJoCo physics simulator, combining the ease-of-use of a high-level programming language with the performance of native C. In addition, Lyceum has a straightforward API to support parallel computation across multiple cores and machines. Overall, depending on the complexity of the environment, Lyceum is 5-30x faster compared to other popular abstractions like OpenAI's Gym and DeepMind's dm-control. This substantially reduces training time for various reinforcement learning algorithms; and is also fast enough to support real-time model predictive control through MuJoCo. The code, tutorials, and demonstration videos can be found at: www.lyceum.ml.

Motivation & Objective

• To address the computational bottleneck in robot learning caused by slow Python-based frameworks like Gym and dm-control.
• To enable faster training of deep reinforcement learning policies by reducing experiment turnaround time.
• To support real-time model predictive control (MPC) in physics simulators, which existing frameworks cannot meet due to performance limits.
• To provide a high-level, extensible ecosystem that retains C-level performance through Julia’s zero-cost abstractions.
• To make advanced robotic control algorithms accessible to labs without massive cloud computing budgets.

Proposed method

• Leverages Julia’s performance and metaprogramming to wrap MuJoCo with zero-cost abstractions, achieving C-level speed while retaining high-level syntax.
• Introduces a flexible AbstractEnvironment interface that supports arbitrary state access, in-place operations, and optional evaluation metrics for controller benchmarking.
• Uses MuJoCo.jl to provide a low-level, memory-mapped interface to MuJoCo 2.0 with direct field access (e.g., d.qpos[:, :arm]).
• Builds LyceumMuJoCo.jl as a high-level environment abstraction similar to Gym and dm-control, but with performance-optimized function calls.
• Employs LyceumMuJoCoViz.jl for interactive visualization of trajectories and controllers, enabling real-time testing of robustness.
• Integrates LyceumAI.jl with Flux.jl and Zygote.jl for neural network training and automatic differentiation, enabling efficient implementation of MPPI and Natural Policy Gradient.

Experimental results

Research questions

• RQ1 Can a high-level programming language like Julia achieve C-level performance in robotics simulation without sacrificing usability?
• RQ2 To what extent can a new ecosystem reduce training time for deep reinforcement learning compared to existing frameworks like Gym and dm-control?
• RQ3 Can the ecosystem support real-time model predictive control with detailed physics simulation, given strict timing constraints?
• RQ4 How effective is the use of in-place operations and zero-cost abstractions in minimizing memory allocation and garbage collection overhead?
• RQ5 Can the ecosystem enable scalable, parallelized rollouts for stochastic control algorithms like MPPI and Natural Policy Gradient?

Key findings

• Lyceum achieves 5–30× speedup over Gym and dm-control across various environments, significantly reducing training time for reinforcement learning algorithms.
• The ecosystem supports real-time model predictive control through MuJoCo, enabling closed-loop control with feedback, which is not feasible with existing Python-based frameworks.
• In-place operations (e.g., getstate! with pre-allocated buffers) eliminate unnecessary memory allocations, making the system suitable for tight real-time control loops.
• The abstraction layer provides full access to simulator state, including non-standard state components, which is critical for model-based control and motion planning.
• The integration of Julia’s parallelism and automatic differentiation enables efficient, multi-threaded rollouts for MPPI and Natural Policy Gradient, accelerating convergence.
• The framework is extensible and supports alternative simulators beyond MuJoCo, such as RigidBodySim.jl or DART, through its clean interface abstraction.