[Paper Review] Optimal Routing for Delay-Sensitive Traffic in Overlay Networks

Rahul Singh, Eytan Modiano|arXiv (Cornell University)|Mar 21, 2017

Advanced Wireless Network Optimization47 references4 citations

TL;DR

This paper proposes a decentralized, adaptive overlay routing policy for delay-sensitive traffic in networks where only overlay nodes make routing decisions, while underlay links use static policies like FIFO. Using multi-timescale stochastic approximation and replicator dynamics, the scheme optimally balances exploration and exploitation to minimize end-to-end delay without requiring knowledge of underlay network characteristics, outperforming existing methods in simulations.

ABSTRACT

We design dynamic routing policies for an overlay network which meet delay requirements of real-time traffic being served on top of an underlying legacy network, where the overlay nodes do not know the underlay characteristics. We pose the problem as a constrained MDP, and show that when the underlay implements static policies such as FIFO with randomized routing, then a decentralized policy, that can be computed efficiently in a distributed fashion, is optimal. Our algorithm utilizes multi-timescale stochastic approximation techniques, and its convergence relies on the fact that the recursions asymptotically track a nonlinear differential equation, namely the replicator equation. Extensive simulations show that the proposed policy indeed outperforms the existing policies.

Motivation & Objective

• To design a dynamic, decentralized routing policy for overlay networks that meets strict end-to-end delay requirements despite lack of knowledge about the underlying legacy network.
• To address the challenge of online learning in a constrained environment where overlay nodes must balance exploration and exploitation to discover low-delay paths.
• To decouple the global delay constraint into local link-level queue thresholds and use link prices to induce optimal, cooperative routing behavior.
• To develop a three-layer control architecture that integrates online learning, price adaptation, and queue threshold tuning for stable, low-delay operation.
• To ensure performance robustness across varying network loads, including light and heavy traffic regimes.

Proposed method

• Formulates the problem as a constrained Markov Decision Process (MDP) with average delay constraints, translating them into average queue length bounds via Little’s Law.
• Introduces a three-layer Price-based Overlay Controller (POC) with: (1) overlay nodes making routing decisions, (2) a link-price controller, and (3) a queue threshold manipulator.
• Employs multi-timescale stochastic approximation to simultaneously learn the optimal routing policy and adapt link prices, ensuring convergence to the optimal solution.
• Uses the replicator dynamics as a nonlinear ODE to model the evolution of link prices, enabling decentralized coordination among overlay nodes.
• Applies Q-learning with stochastic approximation to learn the optimal routing policy for a given price vector, even without knowledge of network parameters.
• Combines primal-dual structure with two-time-scale learning to solve the joint optimization problem of routing and price adaptation.

Experimental results

Research questions

• RQ1 How can overlay nodes achieve optimal routing for delay-sensitive traffic when they lack knowledge of the underlay network’s topology, routing, or link capacities?
• RQ2 Can a decentralized, adaptive routing policy be designed that maintains end-to-end delay within specified bounds without relying on centralized control?
• RQ3 What learning mechanism enables effective exploration-exploitation trade-offs in the absence of underlay state information?
• RQ4 How can link prices be dynamically adjusted to guide overlay nodes toward optimal routing while satisfying global delay constraints?
• RQ5 Does the proposed multi-timescale learning scheme converge to an optimal solution under time-varying network conditions?

Key findings

• The proposed decentralized policy achieves optimal performance in terms of end-to-end delay, even without knowledge of the underlay network characteristics.
• The use of replicator dynamics for price adaptation ensures convergence to the optimal routing policy through distributed, online learning.
• The three-layer POC architecture successfully decouples global delay constraints into local link-level queue thresholds and uses price signals to coordinate routing decisions.
• Extensive simulations demonstrate that the proposed policy outperforms existing routing policies in terms of delay and stability across various network loads.
• The algorithm is robust across different traffic conditions and maintains low average queue lengths, satisfying the delay constraints even under dynamic network changes.
• The convergence of the learning algorithm is guaranteed by the asymptotic tracking of a nonlinear differential equation (the replicator equation), ensuring long-term optimality.