[Paper Review] Quantized Reinforcement Learning (QuaRL)

Srivatsan Krishnan, Sharad Chitlangia+4 more|arXiv (Cornell University)|Jan 1, 2019

Reinforcement Learning in Robotics29 references21 citations

TL;DR

This paper proposes QuaRL, a quantization framework for reinforcement learning that enables 8-bit inference with no reward degradation across diverse RL tasks and models. By introducing the ActorQ algorithm and a system for quantized distributed training, it achieves 1.5x–2.5x end-to-end speedups over full-precision training while maintaining convergence and reducing computational costs.

ABSTRACT

Deep reinforcement learning has achieved significant milestones, however, the computational demands of reinforcement learning training and inference remain substantial. Quantization is an effective method to reduce the computational overheads of neural networks, though in the context of reinforcement learning, it is unknown whether quantization's computational benefits outweigh the accuracy costs introduced by the corresponding quantization error. To quantify this tradeoff we perform a broad study applying quantization to reinforcement learning. We apply standard quantization techniques such as post-training quantization (PTQ) and quantization aware training (QAT) to a comprehensive set of reinforcement learning tasks (Atari, Gym), algorithms (A2C, DDPG, DQN, D4PG, PPO), and models (MLPs, CNNs) and show that policies may be quantized to 8-bits without degrading reward, enabling significant inference speedups on resource-constrained edge devices. Motivated by the effectiveness of standard quantization techniques on reinforcement learning policies, we introduce a novel quantization algorithm, extit{ActorQ}, for quantized actor-learner distributed reinforcement learning training. By leveraging full precision optimization on the learner and quantized execution on the actors, extit{ActorQ} enables 8-bit inference while maintaining convergence. We develop a system for quantized reinforcement learning training around extit{ActorQ} and demonstrate end to end speedups of $>$ 1.5 $ imes$ - 2.5 $ imes$ over full precision training on a range of tasks (Deepmind Control Suite). Finally, we break down the various runtime costs of distributed reinforcement learning training (such as communication time, inference time, model load time, etc) and evaluate the effects of quantization on these system attributes.

Motivation & Objective

• To investigate whether the computational benefits of quantization in deep reinforcement learning outweigh the accuracy costs from quantization error.
• To evaluate standard quantization techniques—post-training quantization (PTQ) and quantization-aware training (QAT)—on a broad range of RL tasks, algorithms, and architectures.
• To develop a novel quantization-aware training algorithm, ActorQ, for distributed actor-learner RL frameworks that maintains convergence with 8-bit inference.
• To build a full-stack system for quantized RL training and analyze the impact of quantization on key system-level costs in distributed training.

Proposed method

• Applies post-training quantization (PTQ) and quantization-aware training (QAT) to diverse RL environments (Atari, Gym), algorithms (A2C, DQN, PPO, DDPG, D4PG), and models (MLPs, CNNs).
• Introduces ActorQ, a novel training algorithm that uses full-precision optimization on the learner and quantized inference on the actors, enabling 8-bit policy execution during training.
• Designs a distributed RL training system around ActorQ that supports mixed-precision execution, reducing memory and computation overhead.
• Employs a system-level profiling approach to decompose and measure runtime costs (communication, inference, model loading) in distributed RL training.
• Uses standard quantization techniques with dynamic range and symmetric quantization for weight and activation quantization, integrating them into the training pipeline via differentiable simulation.
• Validates the framework across multiple benchmarks, including DeepMind Control Suite, to measure end-to-end training speedups and policy performance.

Experimental results

Research questions

• RQ1 Can standard quantization techniques like PTQ and QAT be effectively applied to deep reinforcement learning policies without significant reward degradation?
• RQ2 To what extent do 8-bit quantized policies maintain performance across diverse RL tasks and architectures?
• RQ3 Can a novel training algorithm like ActorQ maintain convergence and policy performance while enabling 8-bit inference during distributed training?
• RQ4 What are the system-level performance impacts of quantization on key training bottlenecks such as communication, inference, and model loading in distributed RL?
• RQ5 How do the computational savings from quantization compare to the potential accuracy trade-offs in reinforcement learning?

Key findings

• Policies can be quantized to 8 bits without degrading performance on Atari and Gym environments, enabling significant inference speedups on edge devices.
• ActorQ enables stable training with 8-bit inference by decoupling full-precision optimization on the learner from quantized execution on the actors.
• The proposed system achieves end-to-end training speedups of 1.5× to 2.5× over full-precision training on the DeepMind Control Suite.
• Quantization reduces inference time and model loading time significantly, with minimal impact on communication overhead in distributed training.
• The tradeoff between computational savings and accuracy loss is favorable—quantization maintains policy performance across diverse RL algorithms and environments.
• System-level profiling confirms that inference and model loading are major cost components, both of which are effectively reduced through quantization.