[Paper Review] A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization

Giorgio Stampa, Marta Arias+3 more|arXiv (Cornell University)|Sep 20, 2017

Software-Defined Networks and 5G11 references174 citations

TL;DR

A DRL-based agent learns to optimize SDN routing to minimize network delay, generalizing to unseen traffic and delivering near-optimal configurations in one step after training.

ABSTRACT

In this paper we design and evaluate a Deep-Reinforcement Learning agent that optimizes routing. Our agent adapts automatically to current traffic conditions and proposes tailored configurations that attempt to minimize the network delay. Experiments show very promising performance. Moreover, this approach provides important operational advantages with respect to traditional optimization algorithms.

Motivation & Objective

• Motivate the use of machine learning, specifically DRL, for routing optimization in SDN and Knowledge-Defined Networking (KDN).
• Develop a deep actor-critic DRL agent that maps traffic conditions to routing configurations to minimize delay.
• Demonstrate generalization to unseen traffic states and compare performance against random routing configurations.

Proposed method

• Model the problem as an off-policy, actor-critic, deterministic policy gradient DRL agent.
• Represent state with the Traffic Matrix and actions with a tuple of link-weights that determine all source-destination paths.
• Use two deep neural networks (actor and critic) to learn the policy and value function.
• Train the agent with gravity-model traffic matrices on a scale-free network and evaluate against a large set of random routing configurations.
• Employ a stochastic exploration policy during training to avoid local minima.
• Measure performance via network delay as the reward signal and use OMNeT++ for simulation.

Experimental results

Research questions

• RQ1 Can a DRL agent learn routing policies that minimize network delay under varying traffic demands?
• RQ2 Does a trained DRL agent generalize to unseen traffic matrices and outperform random routing configurations?
• RQ3 What are the operational advantages of DRL in real-time routing optimization compared to traditional optimization methods?

Key findings

• The DRL agent's performance improves with training time.
• The trained DRL agent consistently achieves routing configurations that are within the first quartile of the benchmark across all traffic intensities.
• DRL enables near-optimal, one-step routing decisions after training, offering potential real-time control advantages.
• DRL demonstrates advantages such as model-free learning and black-box optimization over traditional techniques.