[Paper Review] GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning

Jacky Liang, Viktor Makoviychuk+4 more|arXiv (Cornell University)|Oct 12, 2018

Reinforcement Learning in Robotics74 citations

TL;DR

The paper introduces a GPU-accelerated RL simulation framework built on NVIDIA Flex to train continuous locomotion tasks, achieving faster training on a single GPU and scalable multi-GPU performance.

ABSTRACT

Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train more challenging locomotion tasks.

Motivation & Objective

• Motivate reducing sample complexity in Deep RL by accelerating environment simulation on GPUs.
• Demonstrate that GPU-based physics can outperform CPU-based simulation for continuous control tasks.
• Show scalability of GPU-accelerated RL across multiple GPUs and nodes.

Proposed method

• Develop an in-house GPU-based physics engine using a non-smooth Newton rigid-body solver and maximal coordinates formulation.
• Provide an OpenAI Gym-like interface to run RL experiments with hundreds to thousands of agents in parallel on a GPU.
• Implement PPO with adaptive learning rates and online observation whitening, using SELU activations for stability.
• Use Horovod for distributed training with NCCL-based allreduce to synchronize gradients across GPUs.
• Benchmark single-GPU and multi-GPU configurations on locomotion tasks like Ant, Humanoid, and HFH variants.

Experimental results

Research questions

• RQ1 Can GPU-accelerated physics simulation enable faster RL training for continuous control tasks compared to CPU-based simulators?
• RQ2 How well does the GPU-based RL framework scale across multiple GPUs and nodes for complex locomotion tasks?
• RQ3 What are the performance and training time characteristics when training humanoid locomotion policies on a single GPU versus multi-GPU setups?

Key findings

• Single-machine training with 1 GPU and 1 CPU core can train Humanoid running in under 20 minutes.
• GPU simulation can reach about 60 K frames per second with around 750 concurrent humanoids; mean frame time per agent < 0.02 ms.
• Multi-GPU experiments show limited scaling for Humanoid but more pronounced speedups for HFH and HFH on Complex Terrain tasks.
• Compared to CPU-based methods, their PPO with GPU simulation achieves faster results with far fewer CPU cores.
• Table 1 compares resource-time trade-offs: CPU-only large clusters vs. 1 GPU 1 CPU core achieving 16 minutes for humanoid running.