[Paper Review] Communication-Optimal Convolutional Neural Nets

James Demmel, Grace Dinh|arXiv (Cornell University)|Feb 19, 2018

Stochastic Gradient Optimization Techniques13 references18 citations

TL;DR

This paper presents communication-optimal algorithms for convolutional neural networks (CNNs) by deriving tight lower bounds on data movement and designing loop reorganizations and tilings that achieve these bounds. It demonstrates significantly higher data reuse—up to 2.75× better than matrix multiplication—by exploiting problem-specific structure in CNN computations, leading to reduced communication costs on modern architectures.

ABSTRACT

Efficiently executing convolutional neural nets (CNNs) is important in many machine-learning tasks. Since the cost of moving a word of data, either between levels of a memory hierarchy or between processors over a network, is much higher than the cost of an arithmetic operation, minimizing data movement is critical to performance optimization. In this paper, we present both new lower bounds on data movement needed for CNNs, and optimal sequential algorithms that attain these lower bounds. In most common cases, our optimal algorithms can attain significantly more data reuse than matrix multiplication.

Motivation & Objective

• Address the performance bottleneck in CNNs caused by high communication costs relative to arithmetic operations.
• Identify fundamental lower bounds on data movement for convolutional and pooling layers in CNNs, independent of implementation details.
• Design optimal loop reordering and tiling strategies that achieve these lower bounds across all common CNN hyperparameter configurations.
• Extend the analysis to parallel settings and real-world models like AlexNet, showing practical communication cost reductions.
• Lay the foundation for automatic communication-optimizing code generation in compilers via formalized optimization techniques.

Proposed method

• Derive new communication lower bounds using advanced mathematical tools from functional analysis, group theory, and lattice theory.
• Formulate the optimal tiling problem as a linear program (LP) over loop tile sizes, constrained by fast memory capacity (cache size M).
• Prove that the lower bounds are always attainable by showing feasible solutions exist in the LP formulation for all parameter combinations.
• Use logarithmic transformation to convert the non-linear tiling optimization into a linear program for tractable solution.
• Generalize the model to include stride parameters and multiple memory levels, enabling analysis of real-world CNN workloads.
• Extend results to pooling layers and discuss implications for distributed, multi-processor systems with shared memory.

Experimental results

Research questions

• RQ1 What is the theoretical minimum amount of data movement required to compute a convolutional layer in a CNN, given arbitrary dimensions and cache size?
• RQ2 Can this lower bound be achieved in practice through algorithmic reorganization of loops and data access patterns?
• RQ3 How does the data reuse in optimal CNN algorithms compare to that of standard matrix multiplication, a canonical dense linear algebra operation?
• RQ4 What are the implications of these bounds and algorithms for real-world deep learning models such as AlexNet?
• RQ5 Can the proposed method be generalized to automatically optimize arbitrary nested loop nests for communication efficiency in compilers?

Key findings

• The paper establishes a new communication lower bound for CNNs as the maximum of five expressions involving input dimensions, filter size, strides, and cache size M.
• In most common cases—especially with small filters and limited cache—the fifth term in the bound, $ BCKWH(RSrac{ ho}{M})^{1/2} $, dominates and represents the minimal communication cost.
• The proposed optimal algorithms achieve up to 2.75× more data reuse than matrix multiplication, significantly reducing communication volume.
• For real-world models like AlexNet, the method enables an integer-factor reduction in communication cost over matrix-multiply-based approaches when M is small (e.g., L1/L2 cache sizes).
• The linear programming formulation for tiling is always feasible and guarantees attainment of the theoretical lower bound across all parameter regimes.
• The approach generalizes to parallel settings with shared memory, where communication cost scales optimally with the number of processors and problem partitioning.