[Paper Review] Experience-driven Networking: A Deep Reinforcement Learning based Approach

Zhiyuan Xu, Jian Tang+5 more|arXiv (Cornell University)|Jan 17, 2018

Software-Defined Networks and 5G17 references50 citations

TL;DR

DRL-TE is a model-free, DRL-based framework for traffic engineering that uses TE-aware exploration and prioritized experience replay to optimize end-to-end utility, delay, and throughput in dynamic networks, outperforming baselines and DDPG in ns-3 simulations.

ABSTRACT

Modern communication networks have become very complicated and highly dynamic, which makes them hard to model, predict and control. In this paper, we develop a novel experience-driven approach that can learn to well control a communication network from its own experience rather than an accurate mathematical model, just as a human learns a new skill (such as driving, swimming, etc). Specifically, we, for the first time, propose to leverage emerging Deep Reinforcement Learning (DRL) for enabling model-free control in communication networks; and present a novel and highly effective DRL-based control framework, DRL-TE, for a fundamental networking problem: Traffic Engineering (TE). The proposed framework maximizes a widely-used utility function by jointly learning network environment and its dynamics, and making decisions under the guidance of powerful Deep Neural Networks (DNNs). We propose two new techniques, TE-aware exploration and actor-critic-based prioritized experience replay, to optimize the general DRL framework particularly for TE. To validate and evaluate the proposed framework, we implemented it in ns-3, and tested it comprehensively with both representative and randomly generated network topologies. Extensive packet-level simulation results show that 1) compared to several widely-used baseline methods, DRL-TE significantly reduces end-to-end delay and consistently improves the network utility, while offering better or comparable throughput; 2) DRL-TE is robust to network changes; and 3) DRL-TE consistently outperforms a state-ofthe-art DRL method (for continuous control), Deep Deterministic Policy Gradient (DDPG), which, however, does not offer satisfying performance.

Motivation & Objective

• Motivate a model-free, experience-driven approach to traffic engineering in dynamic networks.
• Develop a DRL-based control framework that learns network dynamics and control policies without explicit models.
• Propose TE-aware exploration and actor-critic prioritized experience replay to optimize DRL for TE.
• Validate the approach via packet-level ns-3 simulations on multiple topologies and compare to baselines.

Proposed method

• Formulates TE as a continuous-control DRL problem with state s = {xk, zk} for each session.
• Defines actions as split ratios across candidate paths for all sessions.
• Uses an actor-critic DRL (DDPG-based) with TE-aware exploration guided by a base TE solution.
• Introduces prioritized experience replay for actor-critic training with a dual-priority metric (TD error and Q-gradient).
• Implements DRL-TE in ns-3, with a two-layer neural network actor/critic and target networks for stability.
• Evaluates against Shortest Path, Load Balance, NUM-TE, and DDPG across NSFNET, ARPANET, and a BRITE-generated topology.

Experimental results

Research questions

• RQ1 Can a model-free DRL approach learn effective TE policies without an accurate network model?
• RQ2 Does TE-aware exploration and prioritized experience replay improve DRL performance for continuous TE problems?
• RQ3 How does DRL-TE perform in terms of end-to-end delay, throughput, and overall network utility compared to traditional methods and DDPG?
• RQ4 Is the DRL-TE framework robust to changing network conditions and topologies?

Key findings

• DRL-TE significantly reduces end-to-end delay compared to SP, LB, NUM, and DDPG across NSFNET, ARPANET, and BRITE-topologies (example: up to 74.6% delay reduction on NSF).
• DRL-TE achieves average delay reductions of 55.4%, 47.1%, 70.5%, and 44.2% relative to SP, LB, NUM, and DDPG on NSF topology respectively.
• DRL-TE consistently outperforms a state-of-the-art DRL method (DDPG) for continuous control in TE scenarios.
• DRL-TE demonstrates robustness to network changes and provides better or comparable throughput while improving network utility.
• The framework includes novel TE-aware exploration (base TE solution guides exploration) and actor-critic prioritized experience replay (two-part priority based on TD error and Q-gradient).
• NSFNET/ARPANET results show notable gains in end-to-end utility and delay under varying traffic demands.