[Paper Review] Curriculum By Smoothing

Samarth Sinha, Animesh Garg+1 more|arXiv (Cornell University)|Mar 3, 2020

Generative Adversarial Networks and Image Synthesis62 references4 citations

TL;DR

This paper proposes Curriculum by Smoothing (CBS), a training regime that enhances CNN performance by progressively increasing high-frequency information in feature maps through annealing Gaussian filter variance. By smoothing early training features to reduce noise from untrained weights and gradually allowing richer details, CBS improves accuracy and generalization across image classification, transfer learning, object detection, semantic segmentation, and VAEs without adding parameters.

ABSTRACT

Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation. Moreover, recent work in Generative Adversarial Networks (GANs) has highlighted the importance of learning by progressively increasing the difficulty of a learning task [26]. When learning a network from scratch, the information propagated within the network during the earlier stages of training can contain distortion artifacts due to noise which can be detrimental to training. In this paper, we propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters. We propose to augment the train-ing of CNNs by controlling the amount of high frequency information propagated within the CNNs as training progresses, by convolving the output of a CNN feature map of each layer with a Gaussian kernel. By decreasing the variance of the Gaussian kernel, we gradually increase the amount of high-frequency information available within the network for inference. As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data. Our proposed augmented training scheme significantly improves the performance of CNNs on various vision tasks without either adding additional trainable parameters or an auxiliary regularization objective. The generality of our method is demonstrated through empirical performance gains in CNN architectures across four different tasks: transfer learning, cross-task transfer learning, and generative models.

Motivation & Objective

• To address the challenge of noisy, distorted feature representations in early CNN training stages due to untrained parameters.
• To improve deep network representation learning by systematically controlling the flow of high-frequency information during training.
• To develop a generic, non-parametric curriculum that enhances performance across diverse vision tasks without architectural changes.
• To demonstrate the generality and robustness of the method in supervised, self-supervised, and generative learning settings.

Proposed method

• Apply a Gaussian low-pass filter to the output feature maps of each CNN layer during training to suppress high-frequency noise from untrained weights.
• Anneal the standard deviation (σ) of the Gaussian kernel over time to gradually increase the amount of high-frequency information propagated through the network.
• Use anti-aliasing via Gaussian filtering to reduce aliasing artifacts in feature spaces during early training stages.
• Integrate the smoothing operation directly into the forward pass of each layer, making it compatible with any CNN architecture.
• Maintain the same training objective and loss function, ensuring no additional trainable parameters or regularization terms are introduced.
• Apply the method end-to-end across all layers, with the filter applied after each convolutional layer and before activation.

Experimental results

Research questions

• RQ1 Can progressive smoothing of feature maps improve representation learning in CNNs during early training stages?
• RQ2 Does gradually increasing high-frequency information through annealed Gaussian filtering enhance downstream performance across diverse vision tasks?
• RQ3 Can this method improve performance in transfer learning, object detection, semantic segmentation, and generative models without adding parameters?
• RQ4 How does the method compare to existing curriculum learning approaches, especially in terms of robustness and generalization?

Key findings

• CBS improved ResNet-18 accuracy on CIFAR-10 by 10.2 percentage points (from 80.0% to 90.2%) compared to baseline.
• On CIFAR-100, CBS achieved 65.4% accuracy, a 19.7 percentage point improvement over the baseline.
• When used as a feature extractor on unseen datasets, CBS-trained ImageNet models outperformed standard training in zero-shot and few-shot transfer learning settings.
• CBS significantly improved performance on object detection and semantic segmentation tasks when using ImageNet-pretrained weights.
• In VAEs, CBS improved both reconstruction quality (lower NLL) and representation quality (higher MI and active units) on both CIFAR-10 and CelebA.
• Ablation studies confirmed that smoothing only the features (not the input) and progressive annealing of σ are essential—constant σ or smoothing inputs degraded performance.