[Paper Review] Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case

Paul Almasan, José Suárez‐Varela+3 more|arXiv (Cornell University)|Oct 16, 2019

Digital Transformation in Industry46 citations

TL;DR

The paper proposes a DRL agent that uses Graph Neural Networks to generalize routing optimization to unseen network topologies, outperforming state-of-the-art DRL on novel topologies.

ABSTRACT

Deep Reinforcement Learning (DRL) has shown a dramatic improvement in decision-making and automated control problems. Consequently, DRL represents a promising technique to efficiently solve many relevant optimization problems (e.g., routing) in self-driving networks. However, existing DRL-based solutions applied to networking fail to generalize, which means that they are not able to operate properly when applied to network topologies not observed during training. This lack of generalization capability significantly hinders the deployment of DRL technologies in production networks. This is because state-of-the-art DRL-based networking solutions use standard neural networks (e.g., fully connected, convolutional), which are not suited to learn from information structured as graphs. In this paper, we integrate Graph Neural Networks (GNN) into DRL agents and we design a problem specific action space to enable generalization. GNNs are Deep Learning models inherently designed to generalize over graphs of different sizes and structures. This allows the proposed GNN-based DRL agent to learn and generalize over arbitrary network topologies. We test our DRL+GNN agent in a routing optimization use case in optical networks and evaluate it on 180 and 232 unseen synthetic and real-world network topologies respectively. The results show that the DRL+GNN agent is able to outperform state-of-the-art solutions in topologies never seen during training.

Motivation & Objective

• Motivate the need for DRL in network optimization with strong generalization to topologies unseen during training.
• Introduce a GNN-enhanced DRL agent to learn routing policies that generalize over graph-structured networks.
• Demonstrate that the DRL+GNN agent outperforms SoA DRL on both synthetic and real-world topologies.
• Showcase deployability, low inference overhead, and scalability implications for production networks.

Proposed method

• Integrate Graph Neural Networks with Deep Reinforcement Learning to operate on graph-structured network topologies.
• Represent routing actions as graph-embedded choices using k=4 shortest paths per source-destination pair.
• Use a DQN-style objective where the Q-value is estimated by a GNN with a readout DNN.
• Model the environment with link-level features (capacity, betweenness) and action-induced bandwidth allocations as one-hot encoded inputs.
• Apply a message-passing scheme (MPNN) with an RNN to evolve link states over T iterations, producing Q-values.
• Train with experience replay and epsilon-greedy exploration; optimize with SGD and regularization techniques.

Experimental results

Research questions

• RQ1 Can a GNN-augmented DRL agent generalize routing decisions to network topologies unseen during training?
• RQ2 How does the DRL+GNN approach compare to state-of-the-art DRL routing solutions on both synthetic and real-world topologies?
• RQ3 What are the effects of topology size, link features, and action space design on learning generalizable routing policies?
• RQ4 What is the inference overhead and scalability of the DRL+GNN agent when deployed in production-like networks?

Key findings

• The DRL+GNN agent generalizes to unseen topologies and outperforms state-of-the-art DRL on Nsfnet and Geant2 topologies.
• In evaluations on 180 unseen synthetic and 232 unseen real-world topologies, the DRL+GNN approach achieves higher bandwidth allocation than SoA DRL.
• The agent trained on one topology (Nsfnet) can outperform SoA DRL on another topology (Geant2), demonstrating robust generalization.
• The model operates with ms-scale decisions and linear cost growth with network size, supporting deployment in production-like environments.
• Using link betweenness as a feature accelerates convergence and improves policy learning.
• The approach enables a universal model that does not require retraining for new topologies.