[Paper Review] MushroomRL: Simplifying Reinforcement Learning Research

Carlo D’Eramo, Davide Tateo+3 more|arXiv (Cornell University)|Jan 4, 2020

Evolutionary Algorithms and Applications6 references35 citations

TL;DR

MushroomRL is an open-source Python library designed to simplify implementing and running RL experiments with a modular, extensible framework and abundant ready-to-use baselines for both shallow and deep RL.

ABSTRACT

MushroomRL is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. Compared to other available libraries, MushroomRL has been created with the purpose of providing a comprehensive and flexible framework to minimize the effort in implementing and testing novel RL methodologies. Indeed, the architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. The result is a library from which RL researchers can significantly benefit in the critical phase of the empirical analysis of their works. MushroomRL stable code, tutorials and documentation can be found at https://github.com/MushroomRL/mushroom-rl.

Motivation & Objective

• Provide a flexible, easy-to-understand RL framework that unifies interaction with environments across algorithms.
• Reduce implementation effort by offering a comprehensive set of ready-to-use baselines for shallow, online, offline, on-policy, off-policy, and Deep RL.
• Enable rapid empirical analysis and experimentation through a lightweight interface and minimal code requirements.
• Facilitate extension by allowing researchers to add new algorithms in a clearly structured library.
• Demonstrate practicality with real-world use cases and tutorials that showcase complex RL experimentation.

Proposed method

• Describe MushroomRL's strongly modular architecture and its unified interface for various RL tasks.
• Showcase compatibility with standard Python libraries (NumPy, SciPy, scikit-learn, Gym, MuJoCo, PyTorch, etc.) to integrate existing tools.
• Provide a minimal user-facing API that hides low-level details while prompting users to ensure consistency (e.g., environment/state space compatibility).
• Illustrate how to implement and test new algorithms by adding them to mushroom_rl/algorithms and running experiments with simple scripts.
• Present advanced usage through an Atari DQN example script to demonstrate handling of complex RL workflows (learning, evaluation, saving/loading, and experimentation in parallel).
• Emphasize tooling support (callbacks, plotting, parallel execution via joblib) to facilitate empirical analysis.

Experimental results

Research questions

• RQ1 How can a single, unified interface support both shallow and deep RL across different task formulations (batch/online, episodic/continuous, on-policy/off-policy)?
• RQ2 Can MushroomRL reduce development effort and improve extensibility and reproducibility of RL experiments compared to existing libraries?
• RQ3 Is it practical to implement, test, and compare novel RL algorithms within MushroomRL’s framework?
• RQ4 How effectively does MushroomRL facilitate complex experiments such as Atari-style benchmarks and DQN variants?
• RQ5 What is the impact of MushroomRL on the speed and reliability of empirical RL research workflows?

Key findings

• MushroomRL provides a broad, ready-to-use set of RL and Deep RL methodologies within a unified framework.
• Experiments can be written with minimal code and without deep configuration, speeding empirical work.
• The library supports complex use cases (e.g., Atari DQN) and demonstrates how to structure experiments for learning and evaluation.
• Users can easily extend the library by adding new algorithms in a dedicated area of the codebase.
• It integrates with common libraries and tools, enabling flexible data collection, plotting, and parallel experimentation.
• Documentation, tutorials, and stable codebase are highlighted as part of supporting reliable empirical analysis.