[Paper Review] High-Accuracy Low-Precision Training

Christopher De, Megan Leszczynski+5 more|arXiv (Cornell University)|Mar 9, 2018

Medical Imaging and Analysis20 references74 citations

TL;DR

HALP combines SVRG with bit centering to achieve high-accuracy training using fixed low-precision, matching full-precision SVRG in convergence while running faster on CPUs. It demonstrates 3–4x speedups over full-precision SVRG and favorable validation performance on deep learning tasks.

ABSTRACT

Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it. Still, it has been used primarily for inference - not training. Previous low-precision training algorithms suffered from a fundamental tradeoff: as the number of bits of precision is lowered, quantization noise is added to the model, which limits statistical accuracy. To address this issue, we describe a simple low-precision stochastic gradient descent variant called HALP. HALP converges at the same theoretical rate as full-precision algorithms despite the noise introduced by using low precision throughout execution. The key idea is to use SVRG to reduce gradient variance, and to combine this with a novel technique called bit centering to reduce quantization error. We show that on the CPU, HALP can run up to $4 imes$ faster than full-precision SVRG and can match its convergence trajectory. We implemented HALP in TensorQuant, and show that it exceeds the validation performance of plain low-precision SGD on two deep learning tasks.

Motivation & Objective

• Motivate low-precision training for reducing time and energy costs during model training.
• Develop algorithms that maintain or approach full-precision accuracy with fixed bit-widths.
• Analyze how to mitigate quantization noise and gradient variance in low-precision training.

Proposed method

• Propose LP-SVRG, a low-precision SVRG variant without bit centering, with convergence limited by quantization.
• Introduce HALP, which applies bit centering to dynamically re-center and rescale low-precision representations to reduce quantization noise as optimization proceeds.
• Prove that HALP preserves linear convergence like SVRG but to arbitrarily high accuracy using fixed-bit representations.
• Provide a practical implementation for linear models showing how to compute gradients and updates in low-precision.
• Implement and evaluate in TensorQuant, comparing against LP-SVRG and LP-SGD on deep learning and logistic regression tasks.

Experimental results

Research questions

• RQ1 Can low-precision training algorithms converge at the same rate as full-precision SVRG on strongly convex problems?
• RQ2 Does bit centering enable HALP to achieve arbitrarily high accuracy with fixed-bit low-precision arithmetic?
• RQ3 What are the practical throughput and accuracy tradeoffs of HALP versus standard low-precision SGD and SVRG on real tasks?
• RQ4 How do LP-SVRG and HALP perform on deep learning models and logistic regression in terms of training loss and validation accuracy?

Key findings

• LP-SVRG converges linearly to a precision-limited neighborhood determined by quantization delta, matching SVRG until hitting an accuracy floor.
• HALP achieves linear convergence down to arbitrarily high accuracy by using bit centering to shrink quantization noise as optimization proceeds.
• On CPU, HALP runs up to 3× faster than plain SVRG on MNIST and up to 4× faster on a synthetic 10k-feature dataset, while matching or exceeding SVRG validation performance on deep models.
• In deep learning experiments, 8-bit HALP closely matches full-precision SVRG training loss for CNNs and LSTMs, and often matches or improves validation metrics relative to LP-SVRG/LP-SGD.
• HALP outperforms LP-SVRG and LP-SGD on multi-class logistic regression tasks in accuracy while achieving up to 4× faster iterations; HALP remains within 25% per-epoch of LP-SGD.