[Paper Review] Gradient Centralization: A New Optimization Technique for Deep Neural Networks

Hongwei Yong, Jianqiang Huang+2 more|arXiv (Cornell University)|Apr 3, 2020

Advanced Neural Network Applications51 references32 citations

TL;DR

Gradient Centralization (GC) centralizes gradient vectors to zero mean to regularize weights and outputs, improving training efficiency and generalization across CNNs and vision tasks.

ABSTRACT

Optimization techniques are of great importance to effectively and efficiently train a deep neural network (DNN). It has been shown that using the first and second order statistics (e.g., mean and variance) to perform Z-score standardization on network activations or weight vectors, such as batch normalization (BN) and weight standardization (WS), can improve the training performance. Different from these existing methods that mostly operate on activations or weights, we present a new optimization technique, namely gradient centralization (GC), which operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. We show that GC can regularize both the weight space and output feature space so that it can boost the generalization performance of DNNs. Moreover, GC improves the Lipschitzness of the loss function and its gradient so that the training process becomes more efficient and stable. GC is very simple to implement and can be easily embedded into existing gradient based DNN optimizers with only one line of code. It can also be directly used to fine-tune the pre-trained DNNs. Our experiments on various applications, including general image classification, fine-grained image classification, detection and segmentation, demonstrate that GC can consistently improve the performance of DNN learning. The code of GC can be found at https://github.com/Yonghongwei/Gradient-Centralization.

Motivation & Objective

• Motivate a simple gradient-based optimization technique that can improve training efficiency and generalization for DNNs.
• Introduce the GC operator that centralizes gradients and can be plugged into existing optimizers with minimal overhead.
• Theoretically analyze GC as a constrained optimization and its impact on weight space and output feature space.
• Demonstrate GC’s effectiveness across general image classification, fine-grained classification, detection, and segmentation tasks.

Proposed method

• Define the GC operator as Phi_GC(∇_W L) = ∇_W L − μ_{∇_W L} (or Φ_GC(∇_{w_i} L) = ∇_{w_i} L − μ_{∇_{w_i} L}) to centralize gradient vectors.
• Show that GC corresponds to projecting gradients onto a hyperplane with normal vector e, effectively constraining weight updates.
• Embed GC into SGDM and Adam with a single line of code by replacing gradients with Φ_GC(∇L).
• Provide theoretical insights: GC improves Lipschitzness of the loss and gradient, reduces gradient norms, and acts as both weight space and output feature space regularization.
• Discuss compatibility with weight decay and fine-tuning pretrained models.

Experimental results

Research questions

• RQ1 Does centralizing gradients to zero mean improve optimization stability and generalization in DNNs?
• RQ2 Can GC be effectively integrated with common optimizers (SGDM, Adam) and various architectures without extra hyper-parameters?
• RQ3 What are the theoretical implications of GC on the loss landscape and gradient behavior?
• RQ4 Is GC beneficial across a range of tasks including classification, detection, and segmentation, as well as during fine-tuning?

Key findings

• GC consistently accelerates training and improves generalization across image classification, detection, and segmentation benchmarks.
• GC works with multiple optimizers (SGDM, Adam, Adagrad, SGDW, AdamW) and various architectures, including ResNet, DenseNet, VGG, and more.
• GC provides improved Lipschitz properties of the loss and gradients, leading to a smoother optimization landscape and more stable training.
• GC reduces gradient norms and max gradient values, contributing to training stability and faster convergence.
• GC improves fine-tuning outcomes on four fine-grained datasets, with noted accuracy gains when applied to pretrained models.
• GC yields measurable improvements on ImageNet and COCO tasks when combined with normalization methods like BN and GN.