[Paper Review] On the Performance Prediction of BLAS-based Tensor Contractions

Elmar Peise, Diego Fabregat‐Traver+1 more|arXiv (Cornell University)|Sep 30, 2014

Parallel Computing and Optimization Techniques8 references4 citations

TL;DR

This paper presents a micro-benchmark-driven performance prediction framework for BLAS-based tensor contractions, enabling accurate identification of the fastest algorithm among hundreds of possible decompositions without executing them. By emulating cache conditions and timing low-level BLAS kernels in isolation, the method predicts performance with high accuracy and achieves speedups of up to 10^6× over direct algorithm execution, particularly benefiting algorithms with fine-grained BLAS calls.

ABSTRACT

Tensor operations are surging as the computational building blocks for a variety of scientific simulations and the development of high-performance kernels for such operations is known to be a challenging task. While for operations on one- and two-dimensional tensors there exist standardized interfaces and highly-optimized libraries (BLAS), for higher dimensional tensors neither standards nor highly-tuned implementations exist yet. In this paper, we consider contractions between two tensors of arbitrary dimensionality and take on the challenge of generating high-performance implementations by resorting to sequences of BLAS kernels. The approach consists in breaking the contraction down into operations that only involve matrices or vectors. Since in general there are many alternative ways of decomposing a contraction, we are able to methodically derive a large family of algorithms. The main contribution of this paper is a systematic methodology to accurately identify the fastest algorithms in the bunch, without executing them. The goal is instead accomplished with the help of a set of cache-aware micro-benchmarks for the underlying BLAS kernels. The predictions we construct from such benchmarks allow us to reliably single out the best-performing algorithms in a tiny fraction of the time taken by the direct execution of the algorithms.

Motivation & Objective

• To address the challenge of selecting the fastest algorithm among hundreds of possible BLAS-based decompositions for tensor contractions.
• To enable accurate performance prediction for tensor contractions without executing full algorithms.
• To overcome the performance variability caused by differing cache behaviors and dimensionality skew in higher-order tensors.
• To provide a scalable, automated method for selecting optimal algorithms in high-performance computing workloads.
• To support efficient execution of tensor contractions in scientific computing, especially where BLAS-1 and BLAS-2 kernels dominate.

Proposed method

• The method generates multiple algorithmic variants by decomposing tensor contractions into sequences of BLAS kernels (gemm, gemv, ger, axpy, dot).
• It uses micro-benchmarks that isolate and time individual BLAS operations under controlled memory and cache conditions.
• Cache state and prefetcher behavior are modeled and replicated in micro-benchmarks to mirror real execution contexts.
• Performance predictions are extrapolated from micro-benchmark timings to estimate total execution time across all algorithm variants.
• The framework avoids full algorithm execution by relying solely on low-level kernel profiling under representative workloads.
• The approach is validated across diverse tensor contractions, including those with small dimensions and multi-threaded execution.

Experimental results

Research questions

• RQ1 What is the performance impact of different algorithmic decompositions of the same tensor contraction when using BLAS kernels?
• RQ2 Can accurate performance predictions be made for BLAS-based tensor contractions without executing the full algorithms?
• RQ3 How does the choice of BLAS kernel (e.g., gemm vs. axpy) affect performance in tensor contractions with skewed dimensionality?
• RQ4 To what extent can micro-benchmarking of low-level kernels predict the performance of complex, high-level tensor operations?
• RQ5 How does the prediction overhead compare to the time required to execute and time the actual algorithms?

Key findings

• The micro-benchmark-based prediction framework correctly identifies the fastest algorithms among dozens of alternatives, even when performance differences exceed 3×.
• For the contraction Cabc = AijaBjbic, the i′c-gemm and i′b-gemm algorithms achieve 60 flops/cycle, while the slowest (jb′-gemm) reaches only 20 flops/cycle, highlighting significant performance variation.
• The prediction method achieves speedups of up to 10^6× over direct algorithm execution, especially for BLAS-1-based algorithms with fine-grained operations.
• The framework maintains high accuracy across diverse scenarios, including contractions with small dimensions and multi-threaded execution using 10 cores.
• At a = b = c = 1,000, the prediction is still 10^3× faster than executing gemm-based algorithms, and up to 10^6× faster for BLAS-1-based ones.
• The method reliably separates performance groups and identifies optimal algorithms in a tiny fraction of the time required for full execution.