[Paper Review] Reinforcement Learning for Adaptive Routing

Leonid Peshkin, Virginia Savova|arXiv (Cornell University)|Mar 28, 2007

Network Traffic and Congestion Control4 citations

TL;DR

This paper applies policy gradient reinforcement learning to adaptive network routing, enabling decentralized, self-optimizing packet forwarding in dynamic networks without global topology knowledge. The method uses local observations and a global reward signal to train stochastic policies, achieving significantly reduced average packet delivery time compared to baseline routing strategies in simulation.

ABSTRACT

Reinforcement learning means learning a policy--a mapping of observations into actions--based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. We present an application of gradient ascent algorithm for reinforcement learning to a complex domain of packet routing in network communication and compare the performance of this algorithm to other routing methods on a benchmark problem.

Motivation & Objective

• To develop a decentralized, adaptive routing protocol for dynamic communication networks using reinforcement learning.
• To enable network nodes to learn optimal routing policies without global topology knowledge or centralized control.
• To minimize average packet delivery time through policy gradient optimization based on distributed reward signals.
• To evaluate the performance of the method against traditional routing algorithms in a benchmark network simulation.

Proposed method

• Each network node acts as an independent agent using a stochastic policy represented by a softmax function over action probabilities.
• The policy parameters are updated via gradient ascent on the expected cumulative reward using the REINFORCE algorithm.
• The reward signal is derived from end-to-end packet delivery time, with penalties for excessive hops to discourage loops.
• Nodes use local observations—current destination and link status—to select actions, without knowledge of network structure or position.
• The learning process is asynchronous and distributed, with all nodes updating their policies based on the same global reward signal at the end of each epoch.
• Policy shaping is applied by detecting and penalizing packets that exceed a hop threshold equal to the number of nodes in the network.

Experimental results

Research questions

• RQ1 Can a decentralized reinforcement learning approach effectively optimize network routing without global network knowledge?
• RQ2 How does policy gradient learning compare to traditional routing algorithms in terms of average packet delivery time?
• RQ3 Can the system adapt to dynamic network changes such as link failures and traffic variations?
• RQ4 What impact does loop detection and penalty have on learning convergence and routing efficiency?

Key findings

• The proposed reinforcement learning method achieved significantly lower average packet delivery time than baseline routing strategies in the benchmark network simulation.
• The algorithm successfully learned to avoid loops and inefficient paths through policy shaping and reward feedback.
• All nodes converged to a consistent routing policy without requiring coordination or shared topology information.
• The method demonstrated robustness in dynamic environments, adapting to link failures and changing traffic conditions.
• The performance improvement was consistent across various network topologies and traffic levels.