[Paper Review] Measuring the Algorithmic Efficiency of Neural Networks

Danny Hernandez, T. B. Brown|arXiv (Cornell University)|May 8, 2020

Advanced Neural Network Applications39 references87 citations

TL;DR

The paper argues that algorithmic progress can be measured by reductions in compute needed to reach past capabilities, showing a 44x decrease in training FLOPs to AlexNet-level ImageNet from 2012 to 2019, corresponding to algorithmic efficiency doubling every 16 months.

ABSTRACT

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.

Motivation & Objective

• Motivate a measurable facet of algorithmic progress alongside data and compute.
• Quantify how much compute is reduced to achieve AlexNet-level performance on ImageNet over time.
• Compare algorithmic efficiency growth to Moore's Law to understand their combined effect on AI progress.

Proposed method

• Define a metric: compute (FLOPs) required to train a classifier to AlexNet-level performance on ImageNet.
• Track historical FLOP requirements from 2012 to 2019 to quantify efficiency gains.
• Compute the implied rate of algorithmic progress and compare it to Moore's Law for context.

Experimental results

Research questions

• RQ1 What is the historical trend in compute required to reach AlexNet-level performance on ImageNet?
• RQ2 How does algorithmic efficiency growth compare to classic hardware-driven growth (Moore's Law)?
• RQ3 Do hardware and algorithmic efficiency gains compound over meaningful horizons to influence AI progress?

Key findings

• The FLOPs needed to train to AlexNet-level performance on ImageNet decreased by 44x between 2012 and 2019.
• This corresponds to algorithmic efficiency doubling roughly every 16 months over seven years.
• Hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons.
• A model of AI progress should integrate measures from both hardware and algorithmic efficiency.