[Paper Review] How deep the machine learning can be

János Végh|arXiv (Cornell University)|May 2, 2020

Parallel Computing and Optimization Techniques4 references4 citations

TL;DR

This paper investigates the fundamental performance limits of deep learning systems under extreme parallelization, revealing that communication overhead and architectural inefficiencies—especially data movement between processors and memory—significantly reduce computational efficacy despite increasing hardware scale. It demonstrates that even with advanced hardware like GPUs, efficiency drops due to non-parallelizable components such as data copying, challenging the assumption of linear scalability in high-performance AI systems.

ABSTRACT

Today we live in the age of artificial intelligence and machine learning; from small startups to HW or SW giants, everyone wants to build machine intelligence chips, applications. The task, however, is hard: not only because of the size of the problem: the technology one can utilize (and the paradigm it is based upon) strongly degrades the chances to succeed efficiently. Today the single-processor performance practically reached the limits the laws of nature enable. The only feasible way to achieve the needed high computing performance seems to be parallelizing many sequentially working units. The laws of the (massively) parallelized computing, however, are different from those experienced in connection with assembling and utilizing systems comprising just-a-few single processors. As machine learning is mostly based on the conventional computing (processors), we scrutinize the (known, but somewhat faded) laws of the parallel computing, concerning AI. This paper attempts to review some of the caveats, especially concerning scaling the computing performance of the AI solutions.

Motivation & Objective

• To analyze the scalability limitations of deep learning systems under extreme parallelization.
• To identify the root causes of performance degradation in large-scale AI hardware, particularly communication and synchronization overhead.
• To evaluate the efficacy of modern high-performance computing architectures, such as GPUs and accelerators, in real-world AI workloads.
• To challenge the assumption of linear performance scaling in AI systems despite increasing hardware resources.
• To highlight the gap between theoretical performance and actual payload efficiency in supercomputing and AI hardware.

Proposed method

• Analyzes the theoretical and practical limits of parallel computing using Amdahl's Law and Gustafson's Law to model performance scaling.
• Models distributed parallel processing as a system with computation, communication, and synchronization components, quantifying their impact on overall efficiency.
• Uses real-world supercomputer data (e.g., Piz Daint) to compare predicted vs. actual performance and efficiency across hardware generations.
• Evaluates the impact of reduced precision (e.g., half-precision) on communication-to-computation ratio and overall system efficacy.
• Applies the efficiency surface model to visualize how architectural changes (e.g., GPU integration) affect performance and efficiency.
• Compares non-accelerated and GPU-accelerated configurations to isolate the effects of data movement and memory hierarchy on performance.

Experimental results

Research questions

• RQ1 What are the fundamental performance limits of deep learning systems when scaled to thousands of processors?
• RQ2 Why does performance not scale linearly with increasing numbers of processors in AI workloads?
• RQ3 How does data movement between memory and processors affect the effective performance of AI systems?
• RQ4 To what extent do hardware accelerators like GPUs improve real-world AI performance, and what are the hidden costs?
• RQ5 How does reducing floating-point precision impact communication-to-computation ratio and system efficiency?

Key findings

• The payload performance of supercomputer Piz Daint increased by roughly threefold after GPU integration, despite only a slight nominal performance gain, indicating a significant efficiency jump.
• The introduction of GPUs increased the effective efficiency, but also raised the non-parallelizable component (1−αeff), due to data copying between host and device memory.
• When the GPU’s memory space was increased, the performance gain was offset by longer data transfer times, causing a drop in both (1−αeff) and overall efficiency.
• Using half-precision arithmetic reduced computation time by four times but left communication time unchanged, increasing the communication-to-computation ratio and reducing relative efficiency.
• The performance of large-scale systems is constrained not only by single-processor limits but also by inherent bounds in parallelization, as predicted by Amdahl’s Law.
• The paper concludes that the classic computing paradigm—based on sequential, clock-driven processors—fails to efficiently emulate biological systems like the brain, especially at extreme scales.