[Paper Review] Alternating Direction Method of Multipliers for Quantization

Tianjian Huang, Prajwal Singhania+3 more|arXiv (Cornell University)|Sep 8, 2020

Sparse and Compressive Sensing Techniques53 references4 citations

TL;DR

This paper proposes ADMM-Q, a novel application of the Alternating Direction Method of Multipliers for quantized machine learning models, specifically targeting nonconvex discrete optimization problems like binarized neural networks. It establishes convergence of ADMM-Q iterates to stationary points and proves monotonic convergence of the Lagrangian function, providing the first theoretical analysis of ADMM for discrete constraints. The method is extended with soft projection and randomized updates, showing improved empirical performance over baseline algorithms.

ABSTRACT

Quantization of the parameters of machine learning models, such as deep neural networks, requires solving constrained optimization problems, where the constraint set is formed by the Cartesian product of many simple discrete sets. For such optimization problems, we study the performance of the Alternating Direction Method of Multipliers for Quantization ($ exttt{ADMM-Q}$) algorithm, which is a variant of the widely-used ADMM method applied to our discrete optimization problem. We establish the convergence of the iterates of $ exttt{ADMM-Q}$ to certain $ extit{stationary points}$. To the best of our knowledge, this is the first analysis of an ADMM-type method for problems with discrete variables/constraints. Based on our theoretical insights, we develop a few variants of $ exttt{ADMM-Q}$ that can handle inexact update rules, and have improved performance via the use of "soft projection" and "injecting randomness to the algorithm". We empirically evaluate the efficacy of our proposed approaches.

Motivation & Objective

• To address the lack of theoretical understanding of ADMM when applied to nonconvex discrete optimization problems in machine learning.
• To analyze the convergence behavior of ADMM-Q, a variant of ADMM tailored for quantization constraints.
• To investigate whether ADMM-Q improves the objective function monotonically and converges to meaningful stationary points in discrete settings.
• To develop and evaluate practical variants of ADMM-Q with inexact, randomized, or soft projection updates for improved robustness and performance.
• To isolate the core algorithm from heuristic enhancements (e.g., Straight-Through Estimator) to better understand ADMM-Q’s intrinsic behavior.

Proposed method

• ADMM-Q is applied to quantization problems where model parameters are constrained to discrete sets (e.g., ±1 for binarization), using alternating minimization over primal and dual variables.
• The algorithm enforces quantization constraints via augmented Lagrangian minimization, with dual ascent updates for constraint violation reduction.
• A soft projection mechanism is introduced to allow smoother transitions during updates, improving convergence and stability.
• Randomized updates are incorporated to break symmetry and escape poor local minima, enhancing robustness.
• The method is extended to handle inexact or stochastic updates, modeling real-world training scenarios with noise or approximation.
• Theoretical analysis proves that the Lagrangian function converges monotonically and that all limit points of the iterates satisfy a stationarity condition.

Experimental results

Research questions

• RQ1 Does ADMM-Q guarantee improvement of the objective function over iterations in discrete, nonconvex optimization problems?
• RQ2 What can be said about the limit points of the ADMM-Q iterates in the context of discrete constraints?
• RQ3 Can ADMM-Q tolerate inexact, randomized, or stochastic updates without losing convergence or performance?
• RQ4 How does ADMM-Q compare to simpler methods like projected gradient descent in solving discrete quantization problems?
• RQ5 Can theoretical guarantees be established for ADMM when applied to discrete, nonconvex problems—previously unexplored in the literature?

Key findings

• ADMM-Q converges monotonically in the Lagrangian function value, providing a theoretical foundation for its empirical success.
• All limit points of the ADMM-Q iterates satisfy a stationarity condition, indicating convergence to meaningful solutions.
• The proposed variants—ADMM-R (randomized) and ADMM-S (soft projection)—show improved empirical performance over standard ADMM-Q.
• ADMM-Q outperforms competing algorithms such as projected gradient descent and standard ADMM in both quantized quadratic optimization and neural network training.
• The method achieves strong performance without relying on heuristic techniques like Straight-Through Estimator or architectural modifications, demonstrating the efficacy of the core algorithm.
• Empirical results on MNIST and CIFAR-10 show that ADMM-Q with pre-training and soft projection achieves high test accuracy while maintaining low-precision weights.