[Paper Review] High throughput data center topology design

Ankit Singla, P. Brighten Godfrey+1 more|arXiv (Cornell University)|Apr 2, 2014

Interconnection Networks and Systems26 references45 citations

TL;DR

This paper establishes the first non-trivial upper bound on throughput for homogeneous data center networks under uniform traffic, showing that random topologies achieve throughput within a few percent of this bound. It leverages this insight to design heterogeneous topologies, demonstrating up to 43% higher throughput in real-world deployments like VL2 using existing equipment.

ABSTRACT

With high throughput networks acquiring a crucial role in supporting data-intensive applications, a variety of data center network topologies have been proposed to achieve high capacity at low cost. While this work explores a large number of design points, even in the limited case of a network of identical switches, no proposal has been able to claim any notion of optimality. The case of heterogeneous networks, incorporating multiple line-speeds and port-counts as data centers grow over time, introduces even greater complexity.In this paper, we present the first non-trivial upper-bound on network throughput under uniform traffic patterns for any topology with identical switches. We then show that random graphs achieve throughput surprisingly close to this bound, within a few percent at the scale of a few thousand servers. Apart from demonstrating that homogeneous topology design may be reaching its limits, this result also motivates our use of random graphs as building blocks for design of heterogeneous networks. Given a heterogeneous pool of network switches, we explore through experiments and analysis, how the distribution of servers across switches and the interconnection of switches affect network throughput. We apply these insights to a real-world heterogeneous data center topology, VL2, demonstrating as much as 43% higher throughput with the same equipment.

Motivation & Objective

• To establish a theoretical upper bound on network throughput for data center topologies using identical switches under uniform traffic.
• To investigate whether existing topologies, particularly random graphs, approach this theoretical limit in practice.
• To extend insights from homogeneous networks to heterogeneous data centers with mixed switch line-speeds and port counts.
• To optimize server distribution and switch interconnection in heterogeneous networks to maximize throughput.
• To validate the proposed design in a real-world data center topology (VL2), demonstrating significant throughput gains with existing hardware.

Proposed method

• Derives a non-trivial theoretical upper bound on network throughput for any topology composed of identical switches under uniform traffic patterns.
• Employs random graph models as candidate topologies to evaluate how close they come to the theoretical throughput bound.
• Analyzes the impact of switch heterogeneity—specifically varying line-speeds and port counts—on end-to-end network throughput.
• Uses simulation and analytical modeling to study the effects of server distribution across switches and switch interconnection patterns.
• Applies the derived design principles to reconfigure the real-world VL2 data center topology, focusing on switch-level resource allocation and interconnect structure.
• Validates performance improvements through comparative experiments between original and optimized topologies using the same physical equipment.

Experimental results

Research questions

• RQ1 What is the theoretical maximum throughput achievable by any topology composed of identical switches under uniform traffic?
• RQ2 How close do random graph-based topologies come to this theoretical throughput upper bound in realistic-scale data centers?
• RQ3 How does switch heterogeneity—specifically differences in line-speed and port count—affect end-to-end network throughput?
• RQ4 What is the optimal distribution of servers across heterogeneous switches and interconnection pattern to maximize throughput?
• RQ5 Can the proposed design principles achieve significant throughput gains in a real-world data center like VL2 without additional hardware?

Key findings

• The paper establishes the first non-trivial theoretical upper bound on throughput for data center networks using identical switches under uniform traffic.
• Random graph topologies achieve throughput within a few percent of this theoretical upper bound at scale, suggesting homogeneous designs may be approaching their performance limits.
• In heterogeneous networks, the distribution of servers across switches and the interconnection structure significantly impact throughput, with optimal configurations yielding substantial gains.
• When applied to the real-world VL2 topology, the proposed design increases network throughput by up to 43% using only existing hardware and without additional equipment.
• The results demonstrate that careful switch-level resource allocation and interconnect planning can yield major performance improvements even without upgrading physical infrastructure.
• The theoretical bound serves as a benchmark, showing that current designs are approaching theoretical optimality in homogeneous settings, but significant gains remain in heterogeneous configurations.