[Paper Review] cuDNN: Efficient Primitives for Deep Learning

Sharan Chetlur, Cliff Woolley+5 more|arXiv (Cornell University)|Oct 3, 2014

Parallel Computing and Optimization Techniques16 references1,026 citations

TL;DR

cuDNN is a GPU-optimized library providing highly efficient, low-level primitives for deep learning workloads, including convolutions, pooling, and activation functions. By leveraging highly tuned kernels and minimal auxiliary memory, it accelerates training and inference—achieving a 36% speedup in Caffe and reducing memory usage, with seamless drop-in integration into existing deep learning frameworks like Caffe and PaddlePaddle.

ABSTRACT

We present a library of efficient implementations of deep learning primitives. Deep learning workloads are computationally intensive, and optimizing their kernels is difficult and time-consuming. As parallel architectures evolve, kernels must be reoptimized, which makes maintaining codebases difficult over time. Similar issues have long been addressed in the HPC community by libraries such as the Basic Linear Algebra Subroutines (BLAS). However, there is no analogous library for deep learning. Without such a library, researchers implementing deep learning workloads on parallel processors must create and optimize their own implementations of the main computational kernels, and this work must be repeated as new parallel processors emerge. To address this problem, we have created a library similar in intent to BLAS, with optimized routines for deep learning workloads. Our implementation contains routines for GPUs, although similarly to the BLAS library, these routines could be implemented for other platforms. The library is easy to integrate into existing frameworks, and provides optimized performance and memory usage. For example, integrating cuDNN into Caffe, a popular framework for convolutional networks, improves performance by 36% on a standard model while also reducing memory consumption.

Motivation & Objective

• To address the lack of a standardized, highly optimized library for deep learning primitives analogous to BLAS in HPC.
• To reduce the burden on deep learning frameworks by providing portable, high-performance implementations of core operations like convolution and pooling.
• To enable performance portability across evolving GPU architectures without requiring framework developers to re-optimize kernels manually.
• To minimize memory overhead while maximizing throughput for diverse workloads, including small mini-batch sizes.
• To support flexible data layouts and seamless integration into existing deep learning frameworks without changing model schemas or data structures.

Proposed method

• The library provides a C-language API with opaque descriptors for tensors and filters, supporting arbitrary strides and data layouts to enable integration with diverse frameworks.
• It exposes optimized routines for forward and backward passes of convolution, pooling, and activation functions in both single and double precision.
• The implementation uses highly optimized GEMM (general matrix multiplication) kernels as a foundation for convolution operations, enabling high performance on GPU architectures.
• Memory usage is minimized by avoiding auxiliary buffers, with data managed directly through user-controlled GPU memory buffers.
• The library supports flexible tensor transformations and sub-tensor indexing, enabling efficient handling of non-square inputs and asymmetric padding.
• Integration is achieved via drop-in replacements for standard framework layers, with automatic fallback to native implementations when needed.

Experimental results

Research questions

• RQ1 How can a library of optimized deep learning primitives improve performance and memory efficiency in deep learning frameworks?
• RQ2 Can a GPU-optimized library achieve significant speedups across diverse deep learning workloads without requiring changes to existing model architectures or data layouts?
• RQ3 To what extent can such a library enable performance portability across evolving GPU hardware platforms?
• RQ4 How does the library’s minimal memory usage impact the ability to train larger models and larger mini-batches?
• RQ5 Can the library be effectively integrated into existing frameworks like Caffe and PaddlePaddle with minimal engineering overhead?

Key findings

• Integrating cuDNN into Caffe improved overall training time by 36% on the bvlc_reference_caffenet model using an NVIDIA Tesla K40 GPU.
• The forward pass speedup was 1.45×, the backward pass 1.30×, and testing improved by 1.49×, demonstrating consistent gains across training phases.
• Memory consumption was reduced compared to alternative implementations based on matrix multiplication, enabling larger models and mini-batch sizes.
• The library achieved a 30% average performance improvement over a matrix-multiplication-based convolution implementation in Baidu’s PaddlePaddle framework on a Tesla K10 GPU.
• cuDNN enabled seamless integration into existing frameworks without changing model schemas or data structure layouts, due to its flexible descriptor-based interface.
• The library supports a wide range of input sizes and configurations, including non-square inputs and asymmetric padding, which is particularly useful for speech and language processing.