[Paper Review] Packet Routing with Graph Attention Multi-agent Reinforcement Learning

Xuan Mai, Quanzhi Fu+1 more|arXiv (Cornell University)|Jul 28, 2021

Software-Defined Networks and 5G9 references4 citations

TL;DR

This paper proposes Deep Graph Attention Network Routing (DGATR), a multi-agent reinforcement learning framework that leverages graph attention networks (GATs) to enable routers to make intelligent, decentralized routing decisions based on local topology and traffic information. The method outperforms baseline algorithms in reducing end-to-end packet delay and increasing network load capacity, especially under dynamic and high-load conditions.

ABSTRACT

Packet routing is a fundamental problem in communication networks that decides how the packets are directed from their source nodes to their destination nodes through some intermediate nodes. With the increasing complexity of network topology and highly dynamic traffic demand, conventional model-based and rule-based routing schemes show significant limitations, due to the simplified and unrealistic model assumptions, and lack of flexibility and adaption. Adding intelligence to the network control is becoming a trend and the key to achieving high-efficiency network operation. In this paper, we develop a model-free and data-driven routing strategy by leveraging reinforcement learning (RL), where routers interact with the network and learn from the experience to make some good routing configurations for the future. Considering the graph nature of the network topology, we design a multi-agent RL framework in combination with Graph Neural Network (GNN), tailored to the routing problem. Three deployment paradigms, centralized, federated, and cooperated learning, are explored respectively. Simulation results demonstrate that our algorithm outperforms some existing benchmark algorithms in terms of packet transmission delay and affordable load.

Motivation & Objective

• To address the limitations of traditional model-based and rule-based routing protocols in handling dynamic, complex network topologies and traffic patterns.
• To develop a model-free, data-driven routing strategy using deep reinforcement learning that adapts to real-time network conditions without relying on simplified assumptions.
• To exploit the graph structure of network topologies by integrating graph neural networks (GNNs) and attention mechanisms into a multi-agent RL framework for improved routing policy learning.
• To evaluate and compare three deployment paradigms—centralized, federated, and cooperative learning—for scalability, stability, and performance in multi-agent routing.

Proposed method

• Each router is modeled as an independent agent that observes local network state, including queue lengths and topology information, and selects next-hop actions using a value-based deep reinforcement learning algorithm.
• A Graph Attention Network (GAT) is employed at each agent to aggregate and process information from neighboring nodes, capturing relational dependencies in the network graph and extracting latent structural features.
• The GAT-based architecture enables each agent to make routing decisions based on both local observations and contextual information from the broader network topology.
• The framework is trained using off-policy experience replay with pre-training on transitions generated by a well-trained Q-routing policy to accelerate convergence.
• Three learning paradigms—centralized, federated, and cooperative—were evaluated, differing in how parameters and gradients are shared among agents to balance exploration and stability.
• The algorithm uses a value-based DRL approach inspired by Deep Q-Networks (DQN), with a deep neural network as function approximator to handle large state-action spaces.

Experimental results

Research questions

• RQ1 Can a multi-agent reinforcement learning framework that leverages graph attention mechanisms outperform traditional and deep learning-based routing algorithms in terms of end-to-end delay and load capacity?
• RQ2 How does incorporating graph structure and relational information through GAT improve routing policy learning compared to fully connected DNNs?
• RQ3 What are the performance trade-offs between centralized, federated, and cooperative learning paradigms in multi-agent routing under dynamic network loads?
• RQ4 To what extent does pre-training with expert demonstrations accelerate learning and improve initial policy performance in complex routing environments?

Key findings

• DGATR significantly outperforms Q-routing and hybrid Q-learning algorithms in end-to-end delay, especially under high network loads, due to better exploration and generalization from DNN-based function approximation.
• DGATR achieves an average end-to-end delay of 5.631 at load 1 on a 6×6 grid network after 5,000 pre-training steps, outperforming DQN-routing, which achieves 6.304 under the same conditions.
• The cooperative learning paradigm yields the lowest delay (8.486 ± 0.02) and highest stability at load 3.0, outperforming centralized and federated learning in both mean delay and variance.
• DGATR sustains higher network loads before experiencing a sharp increase in delay, demonstrating superior adaptability compared to shortest path and Q-routing, and approaching the performance of global centralized routing.
• The use of GAT-based architecture leads to faster convergence and better performance than DQN-routing, indicating that graph-aware feature extraction enhances routing policy quality.
• Pre-training significantly accelerates learning, with DGATR achieving near-optimal performance after only 3,000 pre-training steps, while DQN-routing requires over 4,000 steps to reach similar performance.