[Paper Review] $\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space

Qi Meng, Shuxin Zheng+4 more|arXiv (Cornell University)|Feb 11, 2018

Stochastic Gradient Optimization Techniques34 references4 citations

TL;DR

This paper proposes G-SGD, a novel optimization algorithm that trains ReLU neural networks in a positively scale-invariant space called G-space, which is constructed from basis paths whose values remain invariant under positive weight scaling. By optimizing in G-space using efficient gradient computation via inverse-chain rule and weight allocation, G-SGD achieves significantly better generalization than standard SGD on CIFAR-10 and CIFAR-100, improving test accuracy to 94.67% and 75.20% respectively with regularization.

ABSTRACT

It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant. Conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights, which is, however, not positively scale-invariant. This mismatch may lead to problems during the optimization process. Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }? In this paper, we provide our positive answer to this question. First, we conduct a formal study on the positive scaling operators which forms a transformation group, denoted as $\mathcal{G}$. We show that the value of a path (i.e. the product of the weights along the path) in the neural network is invariant to positive scaling and prove that the value vector of all the paths is sufficient to represent the neural networks under mild conditions. Second, we show that one can identify some basis paths out of all the paths and prove that the linear span of their value vectors (denoted as $\mathcal{G}$-space) is an invariant space with lower dimension under the positive scaling group. Finally, we design stochastic gradient descent algorithm in $\mathcal{G}$-space (abbreviated as $\mathcal{G}$-SGD) to optimize the value vector of the basis paths of neural networks with little extra cost by leveraging back-propagation. Our experiments show that $\mathcal{G}$-SGD significantly outperforms the conventional SGD algorithm in optimizing ReLU networks on benchmark datasets.

Motivation & Objective

• To address the mismatch between the positively scale-invariant nature of ReLU networks and the non-invariant optimization in standard weight space.
• To formally define a positively scale-invariant vector space (G-space) that can fully represent ReLU neural networks.
• To design an efficient stochastic gradient descent algorithm (G-SGD) that optimizes in G-space with negligible computational cost.
• To empirically validate that optimization in G-space improves generalization performance, especially under high positive scale invariance.

Proposed method

• Define a transformation group G of positive scaling operators that preserve the output of ReLU networks.
• Prove that path values—products of weights along paths—are invariant under G, and that all path values fully represent a ReLU network under mild conditions.
• Identify a minimal set of linearly independent paths (basis paths) whose value vectors span a lower-dimensional, invariant G-space.
• Construct a gradient computation method using inverse-chain rule and weight allocation to back-propagate gradients with respect to basis path values efficiently.
• Implement G-SGD by optimizing the basis path value vector instead of the full weight vector, leveraging standard back-propagation with minimal modification.
• Integrate regularization in G-space (e.g., basis path norm) to further improve generalization.

Experimental results

Research questions

• RQ1 Can a positively scale-invariant space be constructed that fully represents ReLU neural networks?
• RQ2 Does optimizing in this invariant space (G-space) lead to better optimization dynamics and generalization than standard SGD in weight space?
• RQ3 How does the dimensionality of G-space compare to the original weight space, and what is its impact on optimization efficiency?
• RQ4 What is the effect of positive scale invariance on the performance gain of G-SGD relative to SGD?
• RQ5 Can regularization in G-space further improve generalization, and how does it compare to weight norm regularization in weight space?

Key findings

• G-SGD achieves a test accuracy of 94.67% on CIFAR-10, outperforming SGD with weight decay (91.25%) and surpassing the reported SOTA by 3.42 percentage points.
• On CIFAR-100, G-SGD with basis path regularization reaches 75.20% test accuracy, improving upon SGD with weight decay (74.39%) by 0.81 percentage points.
• The performance gain of G-SGD increases with higher positive scale invariance, confirming that optimization in G-space is more effective when the network structure is more invariant.
• G-SGD achieves these gains with minimal computational overhead, as the gradient computation in G-space is efficiently enabled via inverse-chain rule and weight allocation.
• The dimension of G-space is H smaller than the weight space, where H is the total number of hidden units, indicating a significant reduction in effective degrees of freedom.
• Regularization in G-space (e.g., basis path norm) is more effective than weight decay in weight space, demonstrating the advantage of optimizing in an invariant, lower-dimensional space.