[Paper Review] Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Wenling Shang, Kihyuk Sohn+2 more|arXiv (Cornell University)|Mar 16, 2016

Advanced Neural Network Applications30 references237 citations

TL;DR

The paper introduces Concatenated ReLU (CReLU), an activation that preserves both positive and negative phase information, analyzes its reconstruction properties, and shows performance gains and parameter efficiency when integrated into various CNN architectures on CIFAR-10/100 and ImageNet.

ABSTRACT

Recently, convolutional neural networks (CNNs) have been used as a powerful tool to solve many problems of machine learning and computer vision. In this paper, we aim to provide insight on the property of convolutional neural networks, as well as a generic method to improve the performance of many CNN architectures. Specifically, we first examine existing CNN models and observe an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). Inspired by our observation, we propose a novel, simple yet effective activation scheme called concatenated ReLU (CRelu) and theoretically analyze its reconstruction property in CNNs. We integrate CRelu into several state-of-the-art CNN architectures and demonstrate improvement in their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer trainable parameters. Our results suggest that better understanding of the properties of CNNs can lead to significant performance improvement with a simple modification.

Motivation & Objective

• Identify intrinsic properties of CNNs in lower convolutional layers that suggest redundancy.
• Propose a simple activation (CReLU) to preserve both positive and negative information.
• Theoretically analyze reconstruction properties of CNNs with CReLU.
• Demonstrate performance gains and parameter efficiency across standard architectures on CIFAR-10/100 and ImageNet.

Proposed method

• Observe and analyze filter pairings in early CNN layers, noting opposite-phase pairs.
• Define Concatenated ReLU (CReLU) as [ReLU(x), ReLU(-x)].
• Theoretically characterize reconstruction capability of a convolution layer followed by CReLU (with and without pooling).
• Integrate CReLU into existing architectures (ReLU replacements) and compare to AVR across CIFAR-10/100 and ImageNet.
• Evaluate regularization and invariance properties via empirical metrics and simple reconstructions.

Experimental results

Research questions

• RQ1 Do lower-layer CNN filters form opposite-phase pairs, indicating redundancy under ReLU?
• RQ2 Can an activation that preserves both positive and negative phase information (CReLU) improve performance and reduce parameters?
• RQ3 What is the reconstruction capability of CNNs with CReLU, and how does it affect feature expressiveness?
• RQ4 How does CReLU impact regularization and invariant feature learning in practice?
• RQ5 What is the impact of applying CReLU to different network depths on large-scale datasets like ImageNet?

Key findings

• CReLU integration yields improved recognition performance on CIFAR-10/100 and ImageNet compared to ReLU, with reduced or comparable parameter counts.
• Lower-layer filters in ReLU nets show negative-phase pairing; CReLU eliminates the need for pairing by preserving both phases, and on ImageNet, pairing is not observed when using CReLU.
• CReLU can achieve similar or better accuracy with about half the parameters in certain configurations, notably when applied to early layers in deeper networks.
• Theoretical results show that a single CReLU layer preserves information sufficient for reconstructing the input component in its range, supporting its expressiveness.
• CReLU models exhibit higher invariance scores across datasets, suggesting more robust representations.
• Reconstruction experiments indicate that CReLU enables meaningful feature inversion, aligning with the proposed reconstruction property.