[Paper Review] No Multiplication? No Floating Point? No Problem! Training Networks for Efficient Inference

Shumeet Baluja, David Marwood+2 more|arXiv (Cornell University)|Sep 24, 2018

Topic Modeling40 references4 citations

TL;DR

This paper proposes a method to train deep neural networks with no floating-point operations, no multiplications, and no non-linear function computations at inference by quantizing both weights and activations into a small set of discrete values. Using periodic adaptive clustering during training, the approach achieves near-ideal performance on classification and regression tasks while reducing model size to less than one-third of standard floating-point networks.

ABSTRACT

For successful deployment of deep neural networks on highly--resource-constrained devices (hearing aids, earbuds, wearables), we must simplify the types of operations and the memory/power resources used during inference. Completely avoiding inference-time floating-point operations is one of the simplest ways to design networks for these highly-constrained environments. By discretizing both our in-network non-linearities and our network weights, we can move to simple, compact networks without floating point operations, without multiplications, and avoid all non-linear function computations. Our approach allows us to explore the spectrum of possible networks, ranging from fully continuous versions down to networks with bi-level weights and activations. Our results show that discretization can be done without loss of performance and that we can train a network that will successfully operate without floating-point, without multiplication, and with less RAM on both regression tasks (auto encoding) and multi-class classification tasks (ImageNet). The memory needed to deploy our discretized networks is less than one third of the equivalent architecture that does use floating-point operations.

Motivation & Objective

• Enable deployment of deep neural networks on highly resource-constrained devices such as hearing aids and earbuds.
• Eliminate floating-point operations and multiplications during inference to reduce power and memory usage.
• Maintain high model performance despite extreme quantization of weights and activations.
• Explore the trade-off between network capacity and efficiency by clustering weights into a small number of unique values.
• Demonstrate that quantized networks can match or exceed performance of full-precision models using the same architecture and training process.

Proposed method

• Quantize network activations into a fixed set of predefined levels (e.g., 32 levels) using uniform quantization in the output space.
• Apply periodic adaptive clustering during training to group all network weights into a small number of unique values (e.g., 100–1000), reducing memory footprint.
• Use a differentiable approximation of the gradient for quantized non-linearities (e.g., tanhD) by backpropagating through the unquantized function.
• Replace multiplication operations with lookup tables (LUTs) that store precomputed products of quantized activations and clustered weights.
• Ensure all inputs to multiplication and summation steps are bounded and known in advance, enabling fixed-point arithmetic and efficient hardware mapping.
• Train the network using standard backpropagation with floating-point precision, but deploy the final model using only integer arithmetic and discrete values.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained and deployed without floating-point operations, multiplications, or non-linear function evaluations at inference time?
• RQ2 How effective is weight clustering with periodic adaptive clustering in maintaining model accuracy while drastically reducing memory usage?
• RQ3 To what extent can quantized activations and weights preserve performance on both regression and classification tasks?
• RQ4 What are the effects of quantization on generalization, and can it act as a regularizer in the absence of dropout?
• RQ5 How does the choice of clustering strategy (global vs. per-layer) affect model performance and memory efficiency?

Key findings

• The proposed method reduces model memory size to less than one-third of the equivalent floating-point network by clustering weights into a small number of unique values.
• Quantized networks achieve performance comparable to or better than full-precision networks on both ImageNet classification and autoencoding regression tasks.
• The use of periodic adaptive clustering enables stable training even with extreme quantization, maintaining high accuracy across diverse architectures.
• The method eliminates the need for floating-point operations and multiplications at inference by replacing them with fixed-point arithmetic and lookup tables.
• The approach shows regularization-like effects, with improved performance observed when combined with or without additional regularizers like dropout.
• The final model can be deployed on ultra-low-power devices such as wearables and earbuds, enabling on-device inference without remote processing.