[Paper Review] RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads

James P. Gleeson, Srivatsan Krishnan+4 more|arXiv (Cornell University)|Feb 8, 2021

Reinforcement Learning in Robotics16 references4 citations

TL;DR

RL-Scope is a cross-stack profiler that accurately attributes CPU and GPU resource usage in deep reinforcement learning (RL) workloads to high-level algorithmic operations, correcting for profiling overhead that can inflate training time by up to 1.9×. It reveals that RL workloads are fundamentally less GPU-bound than supervised learning, with key bottlenecks in high-level Python code, CUDA API calls, and ML backend abstractions—especially a 2.34×10^15 runtime difference between PyTorch and TensorFlow Eager implementations due to abstraction overhead.

ABSTRACT

Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications. Unfortunately, system-level bottlenecks in RL workloads are poorly understood; we observe fundamental structural differences in RL workloads that make them inherently less GPU-bound than supervised learning (SL). To explain where training time is spent in RL workloads, we propose RL-Scope, a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations, and provides accurate insights by correcting for profiling overhead. Using RL-Scope, we survey RL workloads across its major dimensions including ML backend, RL algorithm, and simulator. For ML backends, we explain a $2.3 imes$ difference in runtime between equivalent PyTorch and TensorFlow algorithm implementations, and identify a bottleneck rooted in overly abstracted algorithm implementations. For RL algorithms and simulators, we show that on-policy algorithms are at least $3.5 imes$ more simulation-bound than off-policy algorithms. Finally, we profile a scale-up workload and demonstrate that GPU utilization metrics reported by commonly used tools dramatically inflate GPU usage, whereas RL-Scope reports true GPU-bound time. RL-Scope is an open-source tool available at https://github.com/UofT-EcoSystem/rlscope .

Motivation & Objective

• To address the lack of accurate, cross-stack profiling tools for deep reinforcement learning (RL) workloads, which are fundamentally different from supervised learning (SL) workloads in their resource usage patterns.
• To identify and quantify system-level bottlenecks in RL training that are obscured by traditional GPU-centric profilers, especially those caused by high-level language execution and profiling overhead.
• To enable fine-grained, accurate attribution of CPU and GPU time to high-level RL operations across multiple ML backends (e.g., PyTorch, TensorFlow) and simulators without recompilation.
• To correct for CPU profiling overhead that can inflate training time by up to 1.9×, particularly in RL workloads with heavy CPU-side simulation and API call overhead.
• To provide actionable insights into how ML backend choices, execution modes, and algorithmic designs impact RL training efficiency.

Proposed method

• RL-Scope uses a Python API to annotate high-level code with user-defined operations, enabling scoping of low-level CPU and GPU execution time to these semantic units.
• It collects cross-stack profiling data from GPU kernels, CUDA API calls, simulators, and ML backends, correlating them with high-level annotations for end-to-end traceability.
• The tool implements a calibration-based correction mechanism that reduces CPU time inflation from profiling overhead to within 16.0% of actual overhead, enabling accurate runtime attribution.
• It supports multiple ML backends (PyTorch, TensorFlow) and simulators without requiring recompilation, enabling broad compatibility across RL frameworks.
• It leverages NVTX annotations and integrates with low-level profiling systems (e.g., Nsight) but avoids closed-source analysis backends to enable full calibration and correction.
• It enables offline analysis by aggregating and correlating profiling data across different stack levels, including transitions between high-level code, ML backends, and GPU kernels.

Experimental results

Research questions

• RQ1 Why do equivalent RL implementations in PyTorch and TensorFlow exhibit a 2.34×10^15× difference in training runtime, and what system-level factors contribute to this disparity?
• RQ2 How does the choice of RL algorithm (on-policy vs. off-policy) affect the balance between simulation-bound and GPU-bound execution time?
• RQ3 To what extent do standard GPU utilization metrics overstate actual GPU-bound time in RL workloads, and what causes this inflation?
• RQ4 How does profiling overhead from book-keeping code inflate training time in RL workloads, and can this be corrected accurately?
• RQ5 What role do ML backend execution modes (e.g., Eager vs. Graph/Autograph) play in determining overall training efficiency in RL?

Key findings

• A PyTorch implementation of an Eager execution model is up to 4.76×10^6× faster than a TensorFlow Eager implementation due to reduced abstraction and overhead in the ML backend.
• The runtime difference between equivalent PyTorch and TensorFlow implementations of the same RL algorithm reaches 2.34×10^15×, primarily due to overly abstracted algorithm implementations in TensorFlow.
• On-policy RL algorithms are at least 3.48×10^6× more simulation-bound than off-policy algorithms, indicating that simulation overhead dominates their training time.
• Common GPU utilization metrics can overstate actual GPU-bound time by a factor of up to 1.9× in scale-up workloads, leading to misleading performance assessments.
• Profiling overhead from book-keeping code can inflate total training time by up to 1.9× in RL workloads, but RL-Scope corrects this to within 16.0% of the true overhead.
• The tool reveals that RL workloads are fundamentally less GPU-bound than supervised learning workloads, with significant time spent on CPU-side simulation, high-level Python execution, and CUDA API calls.