[Paper Review] TAPSpMV: Topology-Aware Parallel Sparse Matrix Vector Multiplication.

Amanda Bienz, William Gropp+1 more|arXiv (Cornell University)|Dec 23, 2016

Interconnection Networks and Systems3 citations

TL;DR

TAPSpMV proposes a topology-aware parallel sparse matrix-vector multiplication (SpMV) method that reorganizes messages on each node to shift inter-node communication to cheaper intra-node communication. By minimizing the number and size of network messages, it reduces communication overhead and improves performance on distributed-memory systems.

ABSTRACT

This paper introduces a method to reduce communication that is injected into the network during a sparse matrix-vector multiply by reorganizing messages on each node. This results in a reduction of the inter-node communication, replaced by less-costly intra-node communication, which reduces both the number and size of messages that are injected into the network.

Motivation & Objective

• To reduce inter-node communication costs in parallel sparse matrix-vector multiplication (SpMV) on distributed-memory systems.
• To address the performance bottleneck caused by high-volume, high-latency network communication during SpMV computations.
• To improve scalability and efficiency by reorganizing messages to favor lower-cost intra-node communication.
• To leverage node-level topology awareness to optimize message injection into the network.

Proposed method

• Reorganizing messages on each compute node to minimize the number and size of messages sent across the network.
• Using topology-aware data layout to group communication patterns that can be handled via faster intra-node communication.
• Restructuring the SpMV computation to co-locate data and operations that reduce remote memory access and network traffic.
• Applying a message aggregation strategy that combines multiple small messages into fewer, larger messages to reduce network injection overhead.
• Designing a communication pattern that respects the underlying network topology to avoid contention and improve bandwidth utilization.
• Integrating the reorganization logic into the SpMV kernel to maintain load balance and scalability.

Experimental results

Research questions

• RQ1 How can inter-node communication in parallel SpMV be reduced to improve performance?
• RQ2 What role does node-level topology awareness play in minimizing network message injection?
• RQ3 Can intra-node communication replace costly inter-node communication without degrading load balance?
• RQ4 How does message reorganization affect the overall communication volume and latency in SpMV?
• RQ5 What is the performance impact of topology-aware message scheduling on real-world sparse matrices?

Key findings

• TAPSpMV reduces the number of messages injected into the network by reorganizing communication patterns on each node.
• The method decreases the total communication volume by shifting traffic from inter-node to intra-node communication.
• By minimizing network injection, TAPSpMV achieves lower latency and improved scalability on distributed systems.
• The topology-aware reorganization leads to more efficient use of available bandwidth and reduced contention.
• The approach maintains load balance while significantly reducing communication overhead in SpMV workloads.
• The performance improvement is most pronounced in matrices with irregular sparsity patterns that cause high communication volume.