[Paper Review] Chebyshev Filter Diagonalization on Modern Manycore Processors and GPGPUs

Moritz Kreutzer, Georg Hager+4 more|arXiv (Cornell University)|Mar 6, 2018

Neural Networks and Applications4 citations

TL;DR

This paper proposes optimized Chebyshev filter diagonalization (ChebFD) on Intel Xeon Phi Knights Landing and NVIDIA P100 GPUs using subspace blocking and pipelined communication to hide latency. It achieves over 100 TFlop/s on 512 nodes for computing 100 eigenvalues of 10^9-dimensional sparse matrices, enabling efficient use of hybrid memory hierarchies without performance penalties for large problems.

ABSTRACT

Chebyshev filter diagonalization is well established in quantum chemistry and quantum physics to compute bulks of eigenvalues of large sparse matrices. Choosing a block vector implementation, we investigate optimization opportunities on the new class of high-performance compute devices featuring both high-bandwidth and low-bandwidth memory. We focus on the transparent access to the full address space supported by both architectures under consideration: Intel Xeon Phi "Knights Landing" and Nvidia "Pascal." We propose two optimizations: (1) Subspace blocking is applied for improved performance and data access efficiency. We also show that it allows transparently handling problems much larger than the high-bandwidth memory without significant performance penalties. (2) Pipelining of communication and computation phases of successive subspaces is implemented to hide communication costs without extra memory traffic. As an application scenario we use filter diagonalization studies on topological insulator materials. Performance numbers on up to 512 nodes of the OakForest-PACS and Piz Daint supercomputers are presented, achieving beyond 100 Tflop/s for computing 100 inner eigenvalues of sparse matrices of dimension one billion.

Motivation & Objective

• Address the performance bottleneck in large-scale eigenvalue computations for quantum physics and materials science on modern manycore and GPGPU architectures.
• Overcome memory bandwidth limitations in high-performance computing by efficiently utilizing both high-bandwidth (HBM2/MCDRAM) and low-bandwidth (DDR4) memory on modern nodes.
• Enable transparent handling of problem sizes far exceeding high-bandwidth memory capacity through optimized memory access patterns.
• Reduce communication overhead in distributed-memory implementations by overlapping communication and computation via pipelining.
• Demonstrate strong scaling and high performance on production supercomputers like Oakforest-PACS and Piz Daint for real-world topological insulator applications.

Proposed method

• Implement a block vector formulation of Chebyshev filter diagonalization to increase computational intensity and improve data locality.
• Apply subspace blocking to process only a subset of vectors at a time, enabling efficient use of high-bandwidth memory and transparent access to low-bandwidth memory.
• Introduce a pipelined communication strategy where computation on one subspace overlaps with communication for the next, reducing idle time.
• Leverage non-blocking MPI and GPUdirect on P100 to hide communication latency without increasing memory traffic.
• Optimize sparse matrix-multiple-vector multiplication (SpMMV) through kernel fusion and regular memory access patterns.
• Use algorithmic reformulation to evaluate high-degree Chebyshev polynomials incrementally across subblocks, maintaining computational intensity.

Experimental results

Research questions

• RQ1 How can Chebyshev filter diagonalization be optimized for modern manycore and GPGPU architectures with hybrid memory hierarchies?
• RQ2 To what extent can subspace blocking enable efficient computation on problems larger than the high-bandwidth memory capacity?
• RQ3 Can pipelined communication and computation hide communication costs effectively in distributed-memory ChebFD implementations?
• RQ4 What performance gains are achievable on modern supercomputers like Oakforest-PACS and Piz Daint using these optimizations?
• RQ5 How does the proposed approach scale in strong scaling regimes, particularly when communication becomes a bottleneck?

Key findings

• The block vector implementation achieves approximately 10% of theoretical peak performance on both Xeon Phi and P100 architectures, indicating it is no longer memory-bound.
• Subspace blocking enables transparent use of the full node memory space (including low-bandwidth DDR4) without performance penalties, even when the working set exceeds high-bandwidth memory capacity.
• Pipelined communication and computation reduce communication overhead, achieving up to 50% speed-up over vector mode on Oakforest-PACS for strong scaling with 512 nodes.
• On Piz Daint, pipelining did not yield benefits due to lack of overlap between GPU computation and non-blocking MPI communication, highlighting architecture-specific limitations.
• Performance of 100+ TFlop/s was achieved on Oakforest-PACS and Piz Daint for computing 10^2 eigenvalues of 10^9-dimensional sparse matrices.
• The method maintains high computational intensity and avoids redundant memory traffic, making it suitable for large-scale iterative solvers and spectral analysis in quantum materials.