[Paper Review] Up or Down? Adaptive Rounding for Post-Training Quantization

Markus Nagel, Rana Ali Amjad+3 more|arXiv (Cornell University)|Apr 22, 2020

Advanced Neural Network ApplicationsComputer Science37 references55 citations

TL;DR

AdaRound is a data-adaptive weight rounding method for post-training quantization that improves over nearest-rounding without fine-tuning, using a per-layer QUBO formulation with a continuous relaxation and small unlabeled data. It sets new state-of-the-art performance for several networks and tasks.

ABSTRACT

When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not the best we can do. In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss. AdaRound is fast, does not require fine-tuning of the network, and only uses a small amount of unlabelled data. We start by theoretically analyzing the rounding problem for a pre-trained neural network. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. We simplify this to a layer-wise local loss and propose to optimize this loss with a soft relaxation. AdaRound not only outperforms rounding-to-nearest by a significant margin but also establishes a new state-of-the-art for post-training quantization on several networks and tasks. Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4 bits while staying within an accuracy loss of 1%.

Motivation & Objective

• Motivate and analyze why rounding-to-nearest may be suboptimal for post-training quantization in neural networks.
• Develop a theoretically grounded, efficient per-layer rounding method (AdaRound) that adapts to data and task loss without fine-tuning.
• Demonstrate AdaRound’s effectiveness across multiple networks and tasks, achieving high accuracy with 4-bit weights.
• Show that AdaRound outperforms existing post-training quantization methods and requires only small amounts of unlabeled data.

Proposed method

• Formulate weight rounding as a per-layer Quadratic Unconstrained Binary Optimization (QUBO) problem derived from a second-order Taylor expansion of the task loss.
• Introduce a diagonal-approximation of the Hessian to enable layer-wise optimization and reduce complexity.
• Relax the NP-hard QUBO with a continuous relaxation using soft quantization variables and a differentiable regularizer to encourage binarization.
• Use an asymmetric reconstruction loss and activation-aware formulation to better capture post-quantization effects.
• Optimize layer-by-layer with data-efficient optimization (small unlabeled dataset) and an AdaRound objective that can be solved via Hopfield-style continuous relaxation.

Experimental results

Research questions

• RQ1 Can weight rounding during post-training quantization be formulated as a per-layer optimization problem that accounts for data and task loss interactions?
• RQ2 Does AdaRound, using a continuous relaxation and layer-wise optimization, outperform traditional rounding-to-nearest across multiple architectures and bit-widths?
• RQ3 What is the impact of different design choices (Hessian approximation, local MSE loss, asymmetric reconstruction, activation awareness) on post-training quantization performance?
• RQ4 How much unlabeled data is needed for AdaRound to achieve competitive accuracy, and does data domain affect performance?

Key findings

• AdaRound substantially improves over rounding-to-nearest for post-training quantization across several networks and tasks.
• Using a diagonal Hessian approximation and a local MSE objective yields competitive performance and enables feasible layer-wise optimization.
• Continuous relaxation with Hopfield-inspired optimization and an explicit regularizer yields strong performance and often outperforms STE-based methods.
• Asymmetric reconstruction and activation-aware loss formulations provide additional gains over the base AdaRound objective.
• AdaRound can quantize networks (e.g., ResNet-18/50, MobileNetV2, InceptionV3, DeepLabV3) to 4-bit weights with little to no accuracy loss (within ~1% for some cases), and requires only a small set of unlabeled data (as few as 256 images) to approach FP32 performance.