[Paper Review] XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library

Wenzhang Liu, Wenzhe Cai+8 more|arXiv (Cornell University)|Dec 25, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

XuanCe is a unified, open-source deep reinforcement learning library supporting PyTorch, TensorFlow, and MindSpore, with over 40 single-agent and multi-agent DRL algorithms. It enables cross-framework compatibility, modular extensibility, and high performance across MuJoCo, Atari, and StarCraft II environments, achieving state-of-the-art results on benchmark tasks with consistent hyperparameter settings.

ABSTRACT

In this paper, we present XuanCe, a comprehensive and unified deep reinforcement learning (DRL) library designed to be compatible with PyTorch, TensorFlow, and MindSpore. XuanCe offers a wide range of functionalities, including over 40 classical DRL and multi-agent DRL algorithms, with the flexibility to easily incorporate new algorithms and environments. It is a versatile DRL library that supports CPU, GPU, and Ascend, and can be executed on various operating systems such as Ubuntu, Windows, MacOS, and EulerOS. Extensive benchmarks conducted on popular environments including MuJoCo, Atari, and StarCraftII multi-agent challenge demonstrate the library's impressive performance. XuanCe is open-source and can be accessed at https://github.com/agi-brain/xuance.git.

Motivation & Objective

• To address the lack of a unified, extensible DRL library that supports multiple deep learning frameworks (PyTorch, TensorFlow, MindSpore) and diverse RL algorithms.
• To improve algorithm reusability and extensibility by providing modular components and a unified API across different DRL and MARL algorithms.
• To enable efficient training across diverse hardware (CPU, GPU, Ascend) and operating systems (Ubuntu, Windows, macOS, EulerOS) with consistent performance.
• To support both on-policy and off-policy algorithms, including complex multi-agent settings, with standardized evaluation on standard benchmarks.
• To provide comprehensive documentation and extensibility support for users to easily add new algorithms and environments.

Proposed method

• The library uses YAML-based configuration files to manage hyperparameters, environment settings, and model architectures, enabling flexible and reproducible setup.
• It implements a modular design with decoupled components for environments, algorithms, common tools, and configurations, enhancing code reusability and extensibility.
• A unified API abstracts underlying deep learning frameworks, allowing seamless integration with PyTorch, TensorFlow, and MindSpore through a common interface layer.
• The framework supports parallel environment interaction via vectorized environments, improving sample efficiency and training speed, especially for on-policy algorithms.
• For multi-agent tasks, it employs RNN-based representations (GRU) to handle partial observability, and uses parameter sharing to reduce model complexity.
• Benchmarking is conducted using standardized training protocols and evaluation metrics (e.g., win rate, cumulative rewards) across multiple random seeds to ensure reliability.

Experimental results

Research questions

• RQ1 Can a single DRL library effectively unify diverse deep learning frameworks (PyTorch, TensorFlow, MindSpore) while maintaining high performance and extensibility?
• RQ2 How well does a modular, configuration-driven design enable rapid prototyping and integration of new DRL and MARL algorithms?
• RQ3 To what extent can XuanCe achieve performance comparable to published SOTA results on standard benchmarks like MuJoCo, Atari, and SMAC?
• RQ4 How effective is the library’s support for multi-agent training across diverse cooperative and competitive scenarios with partial observability?
• RQ5 Can the library’s unified API and modular components significantly reduce development and integration overhead for new algorithms and environments?

Key findings

• XuanCe achieved a mean human-normalized score of 4820.0 ± 2429.32 on the Atari environment, outperforming published results for DQN and PPO under identical hyperparameters and network structures.
• On the MuJoCo benchmark, XuanCe achieved a mean score of 5008.7 ± 1235, demonstrating strong performance across multiple continuous control tasks.
• In the StarCraft II multi-agent challenge (SMAC), all tested MARL algorithms—including IQL, VDN, QMIX, WQMIX, VDAC-mix, IPPO, and MAPPO—showed stable and effective learning curves across nine maps with varying complexity.
• The library achieved a win rate of over 90% on the 2m_vs_1z map using QMIX and WQMIX, indicating strong sample efficiency and generalization in cooperative multi-agent settings.
• The benchmark results confirm that XuanCe maintains high performance consistency with published SOTA results when using identical network architectures and hyperparameters.
• The library’s modular design and unified API enabled seamless integration of new algorithms and environments, as demonstrated by the successful extension of support for 40+ algorithms and multiple benchmark environments.