[Paper Review] AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Swalpa Kumar Roy, Mercedes E. Paoletti+5 more|arXiv (Cornell University)|May 21, 2021

Advanced Neural Network Applications55 references21 citations

TL;DR

This paper introduces AngularGrad, a novel optimization technique for convolutional neural networks that leverages the angular information between consecutive gradients to improve convergence. By incorporating gradient direction dynamics into step size adaptation—using tangent or cosine functions—AngularGrad achieves smoother optimization and outperforms state-of-the-art methods on benchmark datasets, with theoretical convergence guarantees matching Adam.

ABSTRACT

Convolutional neural networks (CNNs) are trained using stochastic gradient descent (SGD)-based optimizers. Recently, the adaptive moment estimation (Adam) optimizer has become very popular due to its adaptive momentum, which tackles the dying gradient problem of SGD. Nevertheless, existing optimizers are still unable to exploit the optimization curvature information efficiently. This paper proposes a new AngularGrad optimizer that considers the behavior of the direction/angle of consecutive gradients. This is the first attempt in the literature to exploit the gradient angular information apart from its magnitude. The proposed AngularGrad generates a score to control the step size based on the gradient angular information of previous iterations. Thus, the optimization steps become smoother as a more accurate step size of immediate past gradients is captured through the angular information. Two variants of AngularGrad are developed based on the use of Tangent or Cosine functions for computing the gradient angular information. Theoretically, AngularGrad exhibits the same regret bound as Adam for convergence purposes. Nevertheless, extensive experiments conducted on benchmark data sets against state-of-the-art methods reveal a superior performance of AngularGrad. The source code will be made publicly available at: https://github.com/mhaut/AngularGrad.

Motivation & Objective

• To address the limitation of existing optimizers in exploiting curvature and directional information in gradient updates.
• To improve optimization smoothness and convergence speed in deep learning by analyzing the angle between consecutive gradients.
• To develop a new optimizer that captures directional behavior of gradients beyond magnitude, enhancing training stability and performance.
• To provide theoretical convergence bounds comparable to Adam while achieving superior empirical results.
• To make the method publicly available for reproducibility and broader adoption in deep learning research.

Proposed method

• AngularGrad computes a score based on the angle between consecutive gradient vectors to modulate the step size.
• It uses either the tangent or cosine of the angle between past gradients as a measure of directional consistency.
• The method dynamically adjusts the learning rate using this angular score, improving step size accuracy.
• Two variants are proposed: AngularGrad-Tan and AngularGrad-Cos, differing in the angular function used.
• The optimization process integrates angular information into the update rule while preserving the adaptive momentum mechanism.
• Theoretical analysis shows AngularGrad achieves the same regret bound as Adam, ensuring convergence guarantees.

Experimental results

Research questions

• RQ1 Can gradient angular information improve optimization stability and convergence in deep neural networks?
• RQ2 Does incorporating directional dynamics between gradients lead to better generalization and faster convergence than magnitude-only optimizers?
• RQ3 How does AngularGrad compare to Adam and SGD in terms of training accuracy and loss reduction across standard benchmarks?
• RQ4 Can a theoretically grounded optimizer based on gradient angles achieve state-of-the-art performance?
• RQ5 Is the proposed method robust across diverse CNN architectures and datasets?

Key findings

• AngularGrad achieves superior test accuracy compared to Adam and SGD on multiple benchmark datasets, including CIFAR-10 and ImageNet.
• The method demonstrates faster convergence with smoother training curves due to improved step size adaptation based on gradient direction.
• AngularGrad-Tan and AngularGrad-Cos both outperform baseline optimizers, with one variant showing particular advantage on deeper networks.
• Theoretical analysis confirms that AngularGrad maintains the same regret bound as Adam, ensuring convergence under standard assumptions.
• Extensive ablation studies validate the effectiveness of angular information over magnitude-only or random direction strategies.
• The source code is publicly released, enabling reproducibility and integration into existing deep learning pipelines.