[Paper Review] ns3-gym: Extending OpenAI Gym for Networking Research

Piotr Gawłowicz, Anatolij Zubow|arXiv (Cornell University)|Oct 9, 2018

Software-Defined Networks and 5G8 references58 citations

TL;DR

ns3-gym integrates OpenAI Gym with the ns-3 network simulator to standardize RL-based networking research and enable testing in emulation/testbeds.

ABSTRACT

OpenAI Gym is a toolkit for reinforcement learning (RL) research. It includes a large number of well-known problems that expose a common interface allowing to directly compare the performance results of different RL algorithms. Since many years, the ns-3 network simulation tool is the de-facto standard for academic and industry research into networking protocols and communications technology. Numerous scientific papers were written reporting results obtained using ns-3, and hundreds of models and modules were written and contributed to the ns-3 code base. Today as a major trend in network research we see the use of machine learning tools like RL. What is missing is the integration of a RL framework like OpenAI Gym into the network simulator ns-3. This paper presents the ns3-gym framework. First, we discuss design decisions that went into the software. Second, two illustrative examples implemented using ns3-gym are presented. Our software package is provided to the community as open source under a GPL license and hence can be easily extended.

Motivation & Objective

• Reduce barriers to prototyping RL for networking by unifying the RL framework and simulator interface.
• Provide a generic OpenAI Gym interface for any ns-3 simulation script to become a Gym environment.
• Facilitate benchmarking and fair comparison of RL approaches in networking with a common API.
• Enable emulation with real hardware to bridge simulation and real-world networks.

Proposed method

• Introduce a two-block architecture where Gym provides agents and ns-3 provides the environment.
• Design a generic OpenGymInterface that manages the ns-3 simulation lifecycle and state/action exchange.
• Implement a C++ ns3-gym interface with functions GetObservationSpace, GetActionSpace, GetObservation, GetReward, GetGameOver, GetExtraInfo, ExecuteActions.
• Support common Gym spaces (Discrete, Box, Tuple, Dict) for observations and actions.
• Allow environments to be defined entirely inside ns-3 scripts, with optional command line arguments for customization.
• Provide custom environments (e.g., TCPNs3Env, DASHNs3Env) for predefined RL mappings.

Experimental results

Research questions

• RQ1 How can OpenAI Gym be integrated with ns-3 to create a reusable RL-enabled networking research workflow?
• RQ2 Can a generic interface between Gym and ns-3 support diverse networking problems with minimal user effort?
• RQ3 Do custom environments and emulation capabilities improve practicality for RL-based networking research?
• RQ4 Can researchers compare RL approaches under a common, configurable networking environment?
• RQ5 What examples demonstrate the usefulness of the ns3-gym framework in learning-based networking tasks?

Key findings

• ns3-gym delivers a working bridge between OpenAI Gym and ns-3 using ZMQ and Protocol Buffers for communication.
• Any ns-3 script can be used as a Gym environment by implementing the ns3-gym C++ interface and OpenGymInterface.
• Custom environments for TCP and DASH illustrate practical RL mappings for networking problems.
• Emulation and real-testbed support enable training in simulation and evaluation on real hardware without code changes.
• Illustrative examples (random access control, cognitive radio) show RL can learn effective strategies in wireless networks.