[Paper Review] On Parallel Solution of Sparse Triangular Linear Systems in CUDA

Ruipeng Li|arXiv (Cornell University)|Oct 13, 2017

Matrix Theory and Algorithms34 references6 citations

TL;DR

This paper proposes self-scheduling algorithms for solving sparse triangular linear systems on GPUs using CUDA, overcoming the sequential bottlenecks of traditional level-scheduling methods. The proposed approach achieves up to 2.6× speedup over cuSPARSE's CSRSV2 solver by eliminating global synchronizations and enabling immediate task execution upon data availability, with GPU-accelerated setup phases further reducing total solve time for repeated problems.

ABSTRACT

The acceleration of sparse matrix computations on modern many-core processors, such as the graphics processing units (GPUs), has been recognized and studied over a decade. Significant performance enhancements have been achieved for many sparse matrix computational kernels such as sparse matrix-vector products and sparse matrix-matrix products. Solving linear systems with sparse triangular structured matrices is another important sparse kernel as demanded by a variety of scientific and engineering applications such as sparse linear solvers. However, the development of efficient parallel algorithms in CUDA for solving sparse triangular linear systems remains a challenging task due to the inherently sequential nature of the computation. In this paper, we will revisit this problem by reviewing the existing level-scheduling methods and proposing algorithms with self-scheduling techniques. Numerical results have indicated that the CUDA implementations of the proposed algorithms can outperform the state-of-the-art solvers in cuSPARSE by a factor of up to $2.6$ for structured model problems and general sparse matrices.

Motivation & Objective

• To address the performance bottleneck in solving sparse triangular systems on GPUs due to their inherently sequential nature.
• To improve parallel efficiency by replacing global synchronization in level-scheduling with dynamic, dependency-aware self-scheduling.
• To reduce total solution time by accelerating the setup phase on GPUs for matrices with repeated solves.
• To evaluate and compare the performance of self-scheduling algorithms against state-of-the-art solvers like cuSPARSE CSRSV2.
• To justify the cost of setup phases in practice by determining the minimum number of solves needed for performance gains.

Proposed method

• Introduces self-scheduling algorithms where each unknown is solved as soon as its dependencies are resolved, eliminating global synchronization barriers.
• Adapts dependency graph analysis to identify task readiness dynamically, enabling fine-grained parallelism.
• Implements GPU-accelerated setup phases to discover parallelism from sparse triangular matrix patterns.
• Uses CUDA kernels to execute solve phases with minimal latency between task readiness and execution.
• Employs a hybrid approach combining CPU and GPU for setup, with GPU-based analysis showing speedups on 3D Laplacian and general sparse matrices.
• Evaluates performance across structured and unstructured matrices using metrics like solve time and total time (setup + solve).

Experimental results

Research questions

• RQ1 Can self-scheduling algorithms outperform level-scheduling methods in solving sparse triangular systems on GPUs?
• RQ2 To what extent can GPU-accelerated setup phases reduce the total time for multiple solves?
• RQ3 What is the minimum number of solves required for self-scheduling solvers to outperform cuSPARSE CSRSV2 in total time?
• RQ4 How does the performance of self-scheduling compare to state-of-the-art GPU solvers across diverse sparse matrix types?
• RQ5 Can the overhead of setup phases be justified in real-world applications with repeated solves?

Key findings

• The proposed self-scheduling solver (SLFC) achieves up to 2.6× speedup over cuSPARSE CSRSV2 for structured and general sparse matrices.
• For 11 out of 12 test matrices, GPU-based setup phases reduced setup time compared to CPU-based setup, with speedups observed on 3D Laplacian matrices.
• The smallest number of solves required for SLFC to outperform cuSPARSE CSRSV2 was often low—zero for several matrices, indicating immediate performance advantage.
• GSF solver, despite having a very cheap setup phase, required more solves (e.g., 9 for webbase1M) to match SLFC’s total time, highlighting SLFC’s solve-phase efficiency.
• For matrices like Queen_4147, SLFC required only 11 solves to outperform cuSPARSE CSRSV2 in total time, demonstrating strong practical viability.
• The study confirms that the setup phase cost is justified in applications with multiple right-hand sides, such as iterative solvers and iterative refinement in direct methods.