[Paper Review] Gradually Updated Neural Networks for Large-Scale Image Recognition

Siyuan Qiao, Zhishuai Zhang+3 more|arXiv (Cornell University)|Nov 25, 2017

Advanced Neural Network Applications39 references3 citations

TL;DR

This paper proposes Gradually Updated Neural Networks (GUNN), a novel method to increase network depth by introducing channel-wise computation orderings in convolutional layers, enabling deeper feature learning without additional parameters. GUNN eliminates overlap singularities in loss landscapes, improving training convergence and performance, achieving state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet with minimal architectural changes.

ABSTRACT

Depth is one of the keys that make neural networks succeed in the task of large-scale image recognition. The state-of-the-art network architectures usually increase the depths by cascading convolutional layers or building blocks. In this paper, we present an alternative method to increase the depth. Our method is by introducing computation orderings to the channels within convolutional layers or blocks, based on which we gradually compute the outputs in a channel-wise manner. The added orderings not only increase the depths and the learning capacities of the networks without any additional computation costs, but also eliminate the overlap singularities so that the networks are able to converge faster and perform better. Experiments show that the networks based on our method achieve the state-of-the-art performances on CIFAR and ImageNet datasets.

Motivation & Objective

• To address the challenge of training very deep networks by introducing a new mechanism to increase effective depth without increasing parameters or computational cost.
• To eliminate overlap singularities in the loss landscape that hinder training and degrade performance in standard deep networks.
• To improve generalization and convergence speed in deep convolutional networks through structural reordering of feature computation.
• To achieve state-of-the-art performance on standard image recognition benchmarks using a minimal, plug-in modification to existing architectures.

Proposed method

• GUNN introduces a sequential computation ordering across channels in convolutional layers, where each output channel is computed after the previous ones, effectively increasing depth.
• The method maintains the same number of parameters and FLOPs as the original network, making it computationally efficient.
• By breaking symmetry between neurons through sequential processing, GUNN prevents internal neuron collapse and eliminates overlap singularities in the loss landscape.
• The approach is compatible with residual learning and can be applied to any residual network architecture, such as ResNet or DenseNet.
• The computation order is applied at the channel level within each block, allowing gradual feature refinement across the network.
• The method is applied to standard architectures like ResNet and DenseNet, converting them into GUNN variants by reordering channel computations.

Experimental results

Research questions

• RQ1 Can a network achieve greater effective depth and improved performance without increasing parameters or FLOPs?
• RQ2 Does introducing sequential computation orderings across channels eliminate overlap singularities in the loss landscape of deep networks?
• RQ3 Can GUNN improve training convergence and generalization in standard image recognition benchmarks like CIFAR and ImageNet?
• RQ4 How does GUNN compare to state-of-the-art architectures such as ResNet, DenseNet, and ResNeXt in terms of accuracy and training dynamics?

Key findings

• GUNN-18 achieved a top-1 error of 21.65% on ImageNet, outperforming ResNet-18 (24.0%) and ResNeXt-18 (22.2%) with similar parameter counts.
• On CIFAR-10, GUNN-15 reduced error to 4.15% with residual learning, compared to 5.64% for the baseline SUNN-15.
• The ablation study confirmed that GUNN with residual learning reduced error by 1.49% on CIFAR-10 and 7.06% on CIFAR-100 compared to SUNN.
• GUNN-24 achieved a top-1 error of 3.21% on CIFAR-10, significantly outperforming SUNN-24 (3.88%) and demonstrating faster convergence.
• The training dynamics showed that GUNN networks converge faster and avoid performance degradation caused by collapsed neurons.
• GUNN achieved state-of-the-art performance on ImageNet with 28.9M parameters, matching or surpassing more complex models like DenseNet-264 and ResNeXt-101.