[Paper Review] Expanding across time to deliver bandwidth efficiency and low latency

William M. Mellette, Rajdeep Das+4 more|arXiv (Cornell University)|Mar 29, 2019

Interconnection Networks and Systems37 references19 citations

TL;DR

Opera is a dynamic, circuit-switched datacenter network that uses rapidly reconfigurable, time-varying expander graphs to deliver low-latency performance for small flows while providing direct, bandwidth-efficient paths for bulk traffic over time. It achieves up to 4× higher throughput for shuffle workloads and 60% higher load capacity than cost-equivalent static topologies, with negligible impact on flow completion times.

ABSTRACT

Datacenters need networks that support both low-latency and high-bandwidth packet delivery to meet the stringent requirements of modern applications. We present Opera, a dynamic network that delivers latency-sensitive traffic quickly by relying on multi-hop forwarding in the same way as expander-graph-based approaches, but provides near-optimal bandwidth for bulk flows through direct forwarding over time-varying source-to-destination circuits. The key to Opera's design is the rapid and deterministic reconfiguration of the network, piece-by-piece, such that at any moment in time the network implements an expander graph, yet, integrated across time, the network provides bandwidth-efficient single-hop paths between all racks. We show that Opera supports low-latency traffic with flow completion times comparable to cost-equivalent static topologies, while delivering up to 4x the bandwidth for all-to-all traffic and supporting 60% higher load for published datacenter workloads.

Motivation & Objective

• To address the fundamental trade-off between low-latency and high-bandwidth requirements in datacenter networks.
• To eliminate the bandwidth tax from multi-hop forwarding for bulk traffic without sacrificing low-latency performance for small flows.
• To design a reconfigurable network that maintains low-diameter connectivity for latency-sensitive traffic while enabling direct, high-bandwidth paths over time.
• To demonstrate that dynamic circuit switching can outperform static topologies in both bandwidth efficiency and load capacity for real-world workloads.

Proposed method

• Opera constructs a sequence of time-varying expander graphs by reconfiguring a small number of top-of-rack (ToR) switch uplinks at regular intervals.
• It ensures that at any moment, the network topology forms a valid expander graph, enabling low-latency, multi-hop forwarding for latency-sensitive traffic.
• Bulk traffic is buffered and forwarded only when a direct, single-hop circuit is established between source and destination over time.
• The system pre-assigns rack-to-rack circuits in a deterministic, cyclic manner to ensure all pairs of racks are periodically connected via direct links.
• It uses a base network of dynamically reconfigured expander graphs to maintain low-latency connectivity, while integrating time-averaged direct paths for bandwidth efficiency.
• The design avoids runtime circuit selection or global traffic monitoring by relying on predictable, periodic reconfiguration schedules.

Experimental results

Research questions

• RQ1 Can a reconfigurable network maintain low-latency performance while achieving near-ideal bandwidth efficiency for bulk traffic?
• RQ2 How does the bandwidth tax of multi-hop forwarding compare to the benefits of direct, time-averaged paths in dynamic topologies?
• RQ3 Can a deterministic, periodic reconfiguration schedule support both low-latency and high-throughput performance without requiring real-time traffic monitoring?
• RQ4 What is the achievable throughput gain of such a system over cost-equivalent static topologies for real-world datacenter workloads?

Key findings

• Opera achieves up to 4× higher throughput for shuffle workloads compared to cost-equivalent static topologies, such as folded-Clos or expander graphs.
• For published, skewed datacenter workloads, Opera supports 60% higher load than static topologies while maintaining equivalent flow completion times across all flow sizes.
• Opera reduces the effective bandwidth tax to just 8.4%, significantly lower than the 200–500% tax rates common in existing reconfigurable network designs.
• The routing ruleset for Opera fits within the memory constraints of modern Tofino 65x100GE switches, with up to 85.9% utilization even in large-scale datacenters (1,200 racks).
• The system maintains low-latency performance by ensuring that the expander graph topology is always active, enabling immediate multi-hop forwarding for small flows.
• The approach is stable across a range of network scales, workloads, and cost factors, demonstrating consistent performance improvements.