[Paper Review] A Multi-Agent, Policy-Gradient approach to Network Routing

Nigel Tao, Jonathan Baxter+1 more|ArXiv.org|Dec 2, 2025

Network Traffic and Congestion Control55 citations

TL;DR

The paper treats network routing as a multi-agent, partially observable Markov decision process and uses a multi-agent policy-gradient method (Olpomdp) to optimize average packet trip time without explicit inter-agent communication, including reward shaping to improve convergence.

ABSTRACT

Network routing is a distributed decision problem which naturally admits numerical performance measures, such as the average time for a packet to travel from source to destination. OLPOMDP, a policy-gradient reinforcement learning algorithm, was successfully applied to simulated network routing under a number of network models. Multiple distributed agents (routers) learned co-operative behavior without explicit inter-agent communication, and they avoided behavior which was individually desirable, but detrimental to the group's overall performance. Furthermore, shaping the reward signal by explicitly penalizing certain patterns of sub-optimal behavior was found to dramatically improve the convergence rate.

Motivation & Objective

• Motivate routing as a distributed, model-free optimization problem using reinforcement learning.
• Develop a multi-agent policy-gradient framework where routers are independent agents with local policies.
• Show that cooperative routing emerges without explicit inter-agent communication through a shared reward signal.
• Demonstrate that shaping the reward signal can dramatically speed up convergence.

Proposed method

• Model network routing as a multi-agent, partially observable MDP (POMDP).
• Parameterize each router's policy with Gibbs distribution over outgoing links for each destination.
• Use Olpomdp, an online policy-gradient update with eligibility traces to climb the gradient of the long-term average reward (negative total trip times).
• Update rules: theta_{t+1} = theta_t + gamma_t * r_t * z_t, with z_t updated as z_{t+1} = beta*z_t + (nabla mu / mu).
• Employ a biased gradient estimate from Olpomdp to enable online, distributed learning without full network observability.

Experimental results

Research questions

• RQ1 Can multiple distributed routers learn cooperative routing policies using only local observations and a global reward signal?
• RQ2 Do policy-gradient methods yield mixed (non-deterministic) routing strategies that outperform deterministic ones in certain network setups?
• RQ3 Does reward shaping (penalizing cycles or suboptimal patterns) improve convergence speed without sacrificing optimality?
• RQ4 How does the approach perform under different network models, including link delays, capacities, and node-flow costs (Braess-like networks)?

Key findings

• Routers learn cooperative routing policies that minimize average packet trip time without explicit inter-agent communication.
• Policy-gradient learning can produce mixed strategies that outperform any deterministic policy inهم some scenarios.
• Shaping the reward by penalizing suboptimal patterns (e.g., cycles) dramatically improves convergence rate.
• The method demonstrates optimal or near-optimal routing in multiple network models, including Braess-like networks where traditional routing may fail.