[Paper Review] Learning to Route Efficiently with End-to-End Feedback: The Value of Networked Structure

Ruihao Zhu, Eytan Modiano|arXiv (Cornell University)|Oct 24, 2018

Advanced Bandit Algorithms Research31 references3 citations

TL;DR

This paper proposes adaptive routing algorithms for stochastic online shortest path problems with end-to-end feedback, leveraging network structure to achieve nearly optimal regret. By introducing the Top-Two Comparison (TTC) technique, the method efficiently balances exploration and exploitation in large-scale networks, outperforming prior approaches in both regret and computational efficiency.

ABSTRACT

We introduce efficient algorithms which achieve nearly optimal regrets for the problem of stochastic online shortest path routing with end-to-end feedback. The setting is a natural application of the combinatorial stochastic bandits problem, a special case of the linear stochastic bandits problem. We show how the difficulties posed by the large scale action set can be overcome by the networked structure of the action set. Our approach presents a novel connection between bandit learning and shortest path algorithms. Our main contribution is an adaptive exploration algorithm with nearly optimal instance-dependent regret for any directed acyclic network. We then modify it so that nearly optimal worst case regret is achieved simultaneously. Driven by the carefully designed Top-Two Comparison (TTC) technique, the algorithms are efficiently implementable. We further conduct extensive numerical experiments to show that our proposed algorithms not only achieve superior regret performances, but also reduce the runtime drastically.

Motivation & Objective

• Address the challenge of online shortest path routing in networks with unknown, stochastically evolving link delays and only end-to-end feedback.
• Overcome the computational infeasibility of traditional bandit algorithms in large-scale networks with exponentially many paths.
• Develop regret-optimal algorithms that exploit the networked structure of the action set to reduce exploration complexity.
• Achieve nearly optimal instance-dependent and worst-case regret simultaneously through adaptive exploration.
• Design a computationally efficient algorithm suitable for real-time deployment in large-scale overlay networks.

Proposed method

• Introduce the EC algorithm for identifiable networks, using a novel Top-Two Comparison (TTC) mechanism to guide adaptive exploration.
• Leverage the networked structure of the action set (paths in a DAG) to avoid brute-force path enumeration and reduce computational cost.
• Design the adaptive TTC algorithm to achieve nearly optimal instance-dependent regret by dynamically comparing top candidate paths.
• Extend the TTC framework to the MTTC algorithm to simultaneously achieve nearly optimal worst-case regret.
• Use the linear structure of path delays (sum of link delays) to model the problem as a combinatorial stochastic bandit with bandit feedback.
• Apply the Optimism-in-the-Face-of-Uncertainty (OFU) principle through confidence bounds on path delays, informed by end-to-end feedback.

Experimental results

Research questions

• RQ1 Can network structure be exploited to design efficient regret-optimal routing algorithms in large-scale stochastic online shortest path problems?
• RQ2 How can adaptive exploration be achieved with only end-to-end feedback, without individual link delay observations?
• RQ3 Can nearly optimal instance-dependent and worst-case regret be simultaneously achieved in the bandit feedback setting?
• RQ4 What is the computational cost of achieving optimal regret in networks with exponentially many paths?
• RQ5 How does the TTC technique improve regret and runtime compared to standard UCB or exhaustive search in path selection?

Key findings

• The proposed TTC-based algorithms achieve nearly optimal instance-dependent regret for any directed acyclic network, significantly improving upon prior methods.
• The MTTC algorithm achieves nearly optimal worst-case regret simultaneously with instance-dependent regret, resolving a key limitation in prior work.
• The algorithms are efficiently implementable due to the Top-Two Comparison (TTC) technique, avoiding NP-hard optimization problems.
• Numerical experiments show that the proposed algorithms reduce runtime drastically compared to existing methods while achieving superior regret performance.
• The networked structure of the action set enables scalable learning, making the approach practical for large-scale overlay networks with hidden link delays.
• The method outperforms UCB-based approaches that scale linearly with the number of paths, especially in networks with exponentially many paths.