[Paper Review] Automated Design of Two-Layer Fat-Tree Networks

Konstantin S. Solnushkin|arXiv (Cornell University)|Jan 25, 2013

Interconnection Networks and Systems9 references3 citations

TL;DR

This paper presents an automated algorithm for designing two-layer fat-tree networks in data centers and supercomputers, optimizing for cost and hardware utilization by leveraging real equipment prices and technical specs. It uses per-port metrics and heuristics to minimize unused ports and cable complexity, enabling efficient, scalable, and economically optimal network configurations with minimal manual intervention.

ABSTRACT

This paper presents an algorithm to automatically design two-level fat-tree networks, such as ones widely used in large-scale data centres and cluster supercomputers. The two levels may each use a different type of switches from design database to achieve an optimal network structure. Links between layers can run in bundles to simplify cabling. Several sample network designs are examined and their technical and economic characteristics are discussed. The characteristic feature of our approach is that real life equipment prices and values of technical characteristics are used. This allows to select an optimal combination of hardware to build the network (including semi-populated configurations of modular switches) and accurately estimate the cost of this network. We also show how technical characteristics of the network can be derived from its per-port metrics and suggest heuristics for equipment placement. The algorithm is useful as a part of a bigger design procedure that selects optimal hardware of cluster supercomputer as a whole. Therefore the article is focused on the use of fat-trees for high-performance computing, although the results are valid for any type of data centres.

Motivation & Objective

• To develop an automated algorithm for designing two-layer fat-tree networks that minimizes cost and hardware waste in large-scale data centers and supercomputers.
• To enable optimal selection of switch types, including semi-populated modular configurations, based on real-world pricing and technical specifications.
• To reduce cabling complexity by enabling link bundling and proposing heuristics for efficient equipment placement in racks.
• To support network design as part of a broader CAD system for cluster supercomputers by enabling rapid iteration over hardware combinations.
• To provide accurate cost and performance estimates using per-port metrics, with support for future expandability.

Proposed method

• The algorithm uses per-port metrics—such as cost, bandwidth, and port count—to derive aggregate network characteristics like total cost, blocking factor, and reliability.
• It performs automated search over switch types and configurations, including modular switches with variable numbers of installed leaf modules, to minimize unused ports and total cost.
• The method supports link bundling (e.g., 12x InfiniBand cables) to reduce cable count and physical bulk, improving manageability.
• Heuristics guide equipment placement in racks: blocks are placed densely, edge switches are positioned near the top to shorten downlinks, and a serpentine pattern ensures proximity across rows.
• An average cable price ($80) is used based on estimated cable lengths derived from rack layout and placement patterns.
• The algorithm is integrated into a CAD system for cluster supercomputers, allowing iterative evaluation of hardware combinations with network design as a self-contained subtask.

Experimental results

Research questions

• RQ1 How can two-layer fat-tree networks be automatically designed to minimize total cost while efficiently utilizing switch ports?
• RQ2 What is the impact of using real equipment prices and technical specs on the accuracy of network cost estimation and hardware selection?
• RQ3 How can link bundling and equipment placement heuristics reduce cabling complexity and improve physical layout efficiency?
• RQ4 To what extent can per-port metrics be used to derive accurate aggregate network characteristics such as cost, bandwidth, and reliability?
• RQ5 How can the design algorithm support future network expansion and scalability in large-scale HPC and data center environments?

Key findings

• The algorithm successfully designs non-blocking fat-tree networks with up to 200,000 nodes using high-radix modular switches, significantly exceeding typical supercomputer requirements.
• By optimizing switch configuration and minimizing unused ports, the algorithm reduces hardware waste and total network cost, especially in intermediate-sized deployments.
• Link bundling (e.g., 12x cables) reduces cable count and physical bulk, with estimated savings in cable length and routing complexity.
• The proposed rack placement heuristics allow dense packing of equipment, reducing rack count by one in a 396-node example (11 racks instead of 12), improving space and cable efficiency.
• Using per-port metrics enables accurate estimation of total network cost, bandwidth, and reliability, with a lower bound approximation achievable without full topology enumeration.
• The algorithm supports semi-populated modular switch configurations, allowing precise cost and port utilization optimization based on real equipment data.