[Paper Review] Fast, Better Training Trick -- Random Gradient

Jiakai Wei|arXiv (Cornell University)|Aug 13, 2018

Advanced Image and Video Retrieval Techniques19 references3 citations

TL;DR

This paper introduces Random Gradient (RG), a simple yet effective training trick that accelerates convergence and improves model performance by multiplying the loss with a random number between 0 and 1 before backpropagation. The method requires no additional computation, works across diverse tasks like image classification, semantic segmentation, and GANs, and consistently speeds up training while reducing optimization oscillations, especially with higher learning rates and momentum values of 0.5–0.95.

ABSTRACT

In this paper, we will show an unprecedented method to accelerate training and improve performance, which called random gradient (RG). This method can be easier to the training of any model without extra calculation cost, we use Image classification, Semantic segmentation, and GANs to confirm this method can improve speed which is training model in computer vision. The central idea is using the loss multiplied by a random number to random reduce the back-propagation gradient. We can use this method to produce a better result in Pascal VOC, Cifar, Cityscapes datasets.

Motivation & Objective

• To develop a lightweight, computation-free method to accelerate deep learning training and improve convergence.
• To investigate whether random weighting of the loss function can stabilize and speed up optimization across diverse computer vision tasks.
• To explore the interplay between random gradient, learning rate, momentum, and model architecture in determining training performance.
• To demonstrate the method's effectiveness on standard benchmarks without relying on complex optimization algorithms.

Proposed method

• The core method multiplies the loss value by a uniformly random number between 0 and 1 before computing gradients, creating a 'random gradient' that reduces the magnitude of the backpropagated signal.
• The random scaling is applied independently per batch, introducing controlled noise into the gradient update process to encourage faster convergence and reduced oscillation.
• The approach is model-agnostic and requires no architectural changes or additional parameters, making it applicable to any deep learning model.
• Experiments use standard training protocols with data augmentation, normalization, and learning rate scheduling, while varying learning rates and momentum values to assess generalization.
• The method is evaluated on ResNet, DenseNet, MobileNetV2, PSPNet, and pix2pix, using datasets including Pascal VOC, CIFAR-10/100, and Cityscapes.
• The Adam optimizer is used in GAN experiments, confirming that the method is compatible with adaptive optimizers and does not depend on stochastic gradient descent.

Experimental results

Research questions

• RQ1 Can random scaling of the loss function accelerate convergence and improve generalization in deep learning models?
• RQ2 How does the random gradient method interact with hyperparameters such as learning rate and momentum across different architectures?
• RQ3 Does the method maintain or improve performance on image classification, semantic segmentation, and image generation tasks?
• RQ4 Is the method effective when combined with adaptive optimizers like Adam in GAN training?
• RQ5 Can the random gradient trick be generalized across different model architectures and datasets without architectural modifications?

Key findings

• The random gradient method significantly accelerates convergence in ResNet34 for image classification, particularly when the initial learning rate exceeds 0.1.
• On semantic segmentation with PSPNet, the method reduces optimization oscillations and improves mIOU performance, especially with momentum values of 0.5–0.95.
• For GANs using pix2pix, the random gradient method produces sharper and more realistic image generation results compared to standard backpropagation.
• The method achieves faster convergence and better stability across all evaluated tasks, even when using the Adam optimizer, indicating broad compatibility.
• The optimal momentum value is not fixed across models, but a momentum of 0.5 consistently yields strong results across image classification and segmentation tasks.
• The method shows no measurable computational cost, as it only involves a scalar multiplication per loss value, making it highly efficient for deployment.