[Paper Review] CytonRL: an Efficient Reinforcement Learning Open-source Toolkit Implemented in C++

Xiaolin Wang|arXiv (Cornell University)|Apr 14, 2018

Reinforcement Learning in Robotics7 references3 citations

TL;DR

CytonRL is an open-source, C++-based reinforcement learning toolkit that implements four advanced deep Q-learning algorithms—DQN, Double DQN, Prioritized Experience Replay, and Dueling DQN—using NVIDIA GPU-accelerated libraries and the CytonLib neural network library. It achieves competitive performance on the Atari Breakout environment, demonstrating high training efficiency and strong learning stability through optimized hyperparameters and GPU-optimized computation.

ABSTRACT

This paper presents an open-source enforcement learning toolkit named CytonRL (https://github.com/arthurxlw/cytonRL). The toolkit implements four recent advanced deep Q-learning algorithms from scratch using C++ and NVIDIA's GPU-accelerated libraries. The code is simple and elegant, owing to an open-source general-purpose neural network library named CytonLib. Benchmark shows that the toolkit achieves competitive performances on the popular Atari game of Breakout.

Motivation & Objective

• To develop a high-performance, open-source deep reinforcement learning toolkit that leverages C++ and GPU acceleration for improved training efficiency.
• To implement four state-of-the-art deep Q-learning algorithms—DQN, Double DQN, Prioritized Experience Replay, and Dueling DQN—using a clean, modular codebase.
• To enhance code simplicity and extensibility through integration with CytonLib, a general-purpose open-source neural network library.
• To achieve competitive performance on standard benchmarks like Atari Breakout with carefully tuned hyperparameters for stability and speed.

Proposed method

• The toolkit implements deep Q-learning using a neural network to approximate the optimal action-value function $ Q^*(s,a) $, trained via temporal difference learning with experience replay.
• It employs the Bellman equation to compute targets: $ Q^*(s,a) = \mathbb{E}[r + \gamma \max_{a'} Q^*(s',a')|s,a] $, with bootstrapped targets from a target network.
• The framework uses RMSprop optimization with a learning rate of 0.000625 and a discount factor $ \gamma = 0.99 $, with experience replay memory of size 1,000,000.
• Prioritized experience replay is implemented with $ \alpha = 0.6 $ and $ \beta $ annealed from 0.4 to 1.0 over training steps to reduce sampling bias.
• Dueling DQN architecture separates value and advantage streams, improving value function estimation by decoupling state value and advantage functions.
• All low-level operations are exposed via a C++-based neural network library (CytonLib), enabling full programming flexibility and GPU-accelerated computation using cuDNN and cuBLAS.

Experimental results

Research questions

• RQ1 Can a C++-based deep reinforcement learning toolkit achieve competitive performance on Atari games while maintaining high computational efficiency?
• RQ2 How do the integration of Double DQN, Prioritized Experience Replay, and Dueling DQN architectures impact learning stability and final performance in the CytonRL framework?
• RQ3 To what extent does GPU acceleration via NVIDIA libraries improve training speed and scalability compared to CPU-based or Python-based implementations?
• RQ4 How does the use of a general-purpose neural network library (CytonLib) contribute to code simplicity and maintainability in a low-level RL toolkit?

Key findings

• CytonRL achieved strong performance on the Atari Breakout environment, with dueling double DQN using prioritized replay showing the fastest learning and highest final performance.
• The model settings with dueling and prioritized replay reduced learning variance and improved sample efficiency, enabling faster convergence to high rewards.
• The toolkit demonstrated stable training over 100 million steps, with test episodes averaging over 300 rewards after 50 million steps, indicating robust policy learning.
• The use of C++ and GPU-accelerated libraries enabled efficient computation, outperforming many Python-based implementations in training speed and resource utilization.
• The modular design using CytonLib simplified code structure and allowed full visibility into low-level operations, enhancing extensibility and debugging capabilities.