[Paper Review] CLEX: Yet Another Supercomputer Architecture?

Christoph Lenzen, Roger Wattenhofer|arXiv (Cornell University)|Jul 1, 2016

Interconnection Networks and Systems15 references3 citations

TL;DR

CLEX proposes a novel supercomputer interconnection topology and routing scheme that achieves near-optimal bandwidth utilization and low communication delays by leveraging a hierarchical, clique-based structure with $ n^\varepsilon $ node degrees. It outperforms traditional 3D torus networks by improving point-to-point bandwidth utilization by at least 10x and reducing average path length by 5x in million-node systems, while ensuring deadlock-free, fault-tolerant, and scalable communication.

ABSTRACT

We propose the CLEX supercomputer topology and routing scheme. We prove that CLEX can utilize a constant fraction of the total bandwidth for point-to-point communication, at delays proportional to the sum of the number of intermediate hops and the maximum physical distance between any two nodes. Moreover, % applying an asymmetric bandwidth assignment to the links, all-to-all communication can be realized $(1+o(1))$-optimally both with regard to bandwidth and delays. This is achieved at node degrees of $n^{\varepsilon}$, for an arbitrary small constant $\varepsilon\in (0,1]$. In contrast, these results are impossible in any network featuring constant or polylogarithmic node degrees. Through simulation, we assess the benefits of an implementation of the proposed communication strategy. Our results indicate that, for a million processors, CLEX can increase bandwidth utilization and reduce average routing path length by at least factors $10$ respectively $5$ in comparison to a torus network. Furthermore, the CLEX communication scheme features several other properties, such as deadlock-freedom, inherent fault-tolerance, and canonical partition into smaller subsystems.

Motivation & Objective

• Address the fundamental limitations of current supercomputer interconnects, particularly the 3D torus, which suffers from poor bisection bandwidth and high communication delays at scale.
• Overcome the trade-off between low node degree and short routing paths by designing a topology that enables high connectivity and short paths without relying on constant or polylogarithmic degrees.
• Develop a practical, distributed, and fault-tolerant routing scheme that exploits network redundancy to ensure robustness under link or node failures.
• Integrate physical constraints—such as signal propagation delay and link length—into the theoretical analysis of network performance, moving beyond hop-count-only models.
• Demonstrate that hierarchical, expander-based topologies can achieve near-optimal all-to-all and point-to-point communication performance in large-scale systems.

Proposed method

• Propose CLEX (CLique-EXpander), a hierarchical interconnection topology composed of multiple levels of cliques and expanders, enabling high connectivity and short paths.
• Design a distributed, adaptive routing algorithm that assigns bandwidth dynamically across levels based on average message hop counts, prioritizing lower levels for higher throughput.
• Model communication delays as a function of both hop count and physical distance, incorporating signal propagation delays into performance analysis.
• Implement a fault-tolerant routing mechanism that automatically reroutes messages around failed nodes or links using the inherent redundancy of the CLEX topology.
• Use simulation to evaluate performance on large-scale configurations (e.g., 32^4 and 64^3 nodes), comparing CLEX against 3D torus under identical traffic patterns.
• Apply asymmetric bandwidth allocation across levels to optimize for common communication patterns, especially all-to-all and point-to-point traffic.

Experimental results

Research questions

• RQ1 Can a supercomputer topology achieve constant-factor optimal point-to-point bandwidth utilization with node degrees growing as $ n^\varepsilon $ for arbitrarily small $ \varepsilon > 0 $?
• RQ2 To what extent can a topology with high connectivity and small diameter reduce communication delays when physical signal propagation delays are considered?
• RQ3 How does the performance of a CLEX-based interconnect compare to a 3D torus in terms of effective bandwidth and average path length under realistic, non-ideal routing conditions?
• RQ4 Can a distributed, deadlock-free, and fault-tolerant routing scheme be designed for a high-connectivity topology like CLEX without centralized control?
• RQ5 Does hierarchical bandwidth allocation across levels significantly improve all-to-all communication efficiency in large-scale systems?

Key findings

• CLEX achieves $ (1+o(1)) $-optimal all-to-all communication in both bandwidth and delay, outperforming traditional topologies that cannot match this efficiency with low node degrees.
• For a million-node system, CLEX improves effective point-to-point bandwidth utilization by at least a factor of 10 compared to a 3D torus network.
• The average routing path length in CLEX is reduced by at least a factor of 5 compared to a 3D torus, even under realistic, non-ideal routing conditions.
• In light-traffic scenarios, average message delays in CLEX are reduced to approximately 1.8–2.3 times the physical minimum delay, compared to significantly higher values in torus networks.
• The CLEX routing scheme is inherently deadlock-free and supports seamless fault tolerance through redundant routing paths, even with node or link failures.
• Simulations confirm that CLEX maintains high performance across diverse configurations, including 32^4 and 64^3 node systems, with significant gains in both bandwidth and latency metrics.