[Paper Review] Efficient and Robust Quantization-aware Training via Adaptive Coreset Selection

Xijie Huang, Zechun Liu+2 more|arXiv (Cornell University)|Jun 12, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Adaptive Coreset Selection (ACS), a novel quantization-aware training method that dynamically selects informative training samples using two metrics—error vector score (EVS) and disagreement score (DS)—to improve training efficiency and accuracy. By adaptively balancing these metrics across training epochs, ACS achieves state-of-the-art performance with only 10% of ImageNet-1K data, attaining 68.39% top-1 accuracy on 4-bit quantized ResNet-18, a 4.24% gain over random selection.

ABSTRACT

Quantization-aware training (QAT) is a representative model compression method to reduce redundancy in weights and activations. However, most existing QAT methods require end-to-end training on the entire dataset, which suffers from long training time and high energy costs. In addition, the potential label noise in the training data undermines the robustness of QAT. We propose two metrics based on analysis of loss and gradient of quantized weights: error vector score and disagreement score, to quantify the importance of each sample during training. Guided by these two metrics, we proposed a quantization-aware Adaptive Coreset Selection (ACS) method to select the data for the current training epoch. We evaluate our method on various networks (ResNet-18, MobileNetV2, RetinaNet), datasets(CIFAR-10, CIFAR-100, ImageNet-1K, COCO), and under different quantization settings. Specifically, our method can achieve an accuracy of 68.39\% of 4-bit quantized ResNet-18 on the ImageNet-1K dataset with only a 10\% subset, which has an absolute gain of 4.24\% compared to the baseline. Our method can also improve the robustness of QAT by removing noisy samples in the training set.

Motivation & Objective

• To address the high computational cost and long training time of quantization-aware training (QAT) on large-scale datasets.
• To investigate whether data redundancy in training sets can be leveraged to improve QAT efficiency without sacrificing accuracy.
• To develop a coreset selection strategy tailored specifically for QAT, considering the unique dynamics of quantized model training.
• To design metrics that accurately reflect sample importance during QAT, accounting for both loss gradient and knowledge distillation dynamics.
• To achieve significant data efficiency gains in QAT while maintaining or improving model accuracy across diverse networks and datasets.

Proposed method

• Proposes the error vector score (EVS) as a theoretical approximation of sample importance based on loss gradient analysis during QAT.
• Introduces the disagreement score (DS) to measure the prediction gap between the full-precision and quantized models, reflecting knowledge distillation dynamics.
• Designs an adaptive coreset selection (ACS) framework that combines EVS and DS using a learnable annealing strategy to balance their contributions across training epochs.
• Applies a dynamic weighting scheme, $\beta(t)$, to shift focus from EVS (early training) to DS (late training), improving data diversity and convergence.
• Employs a two-stage selection process: compute EVS and DS for all samples, then select top-$k$ samples per epoch using a weighted combination.
• Uses cosine annealing for $\beta(t)$ to smoothly transition from early to late training priorities, outperforming fixed or linear schedules.

Experimental results

Research questions

• RQ1 How does sample importance vary during quantization-aware training, and can it be quantified effectively using gradient-based metrics?
• RQ2 Can combining error vector score (EVS) and disagreement score (DS) improve coreset selection for QAT compared to using either metric alone?
• RQ3 How should the relative weighting of EVS and DS be adjusted across training epochs to maximize QAT efficiency and accuracy?
• RQ4 To what extent can coreset selection reduce data usage in QAT while maintaining or improving model performance on ImageNet-1K and CIFAR-100?
• RQ5 What is the optimal annealing strategy for balancing EVS and DS during coreset selection in QAT?

Key findings

• ACS achieves 68.39% top-1 accuracy on 4-bit quantized ResNet-18 for ImageNet-1K using only 10% of the training data, a 4.24% absolute improvement over random selection.
• For 2-bit weights-only quantization on CIFAR-100, ACS achieves 67.19% mean accuracy with only 50% of the training data, outperforming baseline coreset methods.
• The cosine annealing strategy for $\beta(t)$ yields the best performance, outperforming fixed, linear, square root, and quadratic schedules.
• The disagreement score (DS) distribution shifts toward zero over time, confirming its role in tracking model convergence and knowledge distillation progress.
• Visualizations show that QAT with ACS produces a smoother and more centralized loss landscape compared to random or baseline coreset selection.
• Ablation studies confirm that both EVS and DS are complementary: EVS excels with small data fractions, while DS improves performance with larger fractions.