[Paper Review] ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations

Jamshaid Ul Rahman, Faiza Makhdoom+1 more|arXiv (Cornell University)|May 28, 2023

Neural Networks and Applications4 citations

TL;DR

This paper proposes ASU-CNN, a deep convolutional neural network that employs the non-monotonic, oscillatory activation function ASU across its layers to enhance feature representation and classification performance. Trained with Adam optimizer and learning rate fine-tuning, the model achieves strong results on CIFAR-10, demonstrating computational efficiency and effective feature visualization capabilities.

ABSTRACT

Activation functions play a decisive role in determining the capacity of Deep Neural Networks as they enable neural networks to capture inherent nonlinearities present in data fed to them. The prior research on activation functions primarily focused on the utility of monotonic or non-oscillatory functions, until Growing Cosine Unit broke the taboo for a number of applications. In this paper, a Convolutional Neural Network model named as ASU-CNN is proposed which utilizes recently designed activation function ASU across its layers. The effect of this non-monotonic and oscillatory function is inspected through feature map visualizations from different convolutional layers. The optimization of proposed network is offered by Adam with a fine-tuned adjustment of learning rate. The network achieved promising results on both training and testing data for the classification of CIFAR-10. The experimental results affirm the computational feasibility and efficacy of the proposed model for performing tasks related to the field of computer vision.

Motivation & Objective

• To explore the impact of non-monotonic, oscillatory activation functions like ASU on deep neural network performance in computer vision.
• To design and evaluate a new CNN architecture, ASU-CNN, that integrates the ASU activation function across its convolutional layers.
• To analyze feature maps from different layers to visualize how ASU influences feature learning and representation.
• To demonstrate the computational feasibility and classification efficacy of ASU-CNN on standard benchmarks like CIFAR-10.

Proposed method

• The ASU-CNN model is constructed using the ASU (Adaptive Sine Unit) activation function, which is non-monotonic and oscillatory, allowing for richer representational capacity.
• The network is trained using the Adam optimizer with a carefully tuned initial learning rate to improve convergence and generalization.
• Feature maps from multiple convolutional layers are visualized to analyze the learned spatial and channel-wise representations.
• The model is evaluated on the CIFAR-10 dataset using standard training and testing protocols to assess accuracy and efficiency.
• The architecture is designed to be computationally efficient, balancing depth, width, and activation complexity for real-world deployment.
• The ASU activation function is applied uniformly across all convolutional layers to maintain consistent nonlinear transformation behavior.

Experimental results

Research questions

• RQ1 How does the non-monotonic and oscillatory nature of the ASU activation function affect feature learning in convolutional networks?
• RQ2 Can ASU-CNN achieve competitive classification accuracy on CIFAR-10 compared to standard ReLU-based models?
• RQ3 What insights do feature map visualizations from ASU-CNN provide about the learned representations in different layers?
• RQ4 How does the Adam optimizer with fine-tuned learning rate influence the training dynamics and performance of ASU-CNN?
• RQ5 Is the ASU-CNN architecture computationally efficient and feasible for practical computer vision applications?

Key findings

• ASU-CNN achieves strong classification performance on the CIFAR-10 dataset, demonstrating the effectiveness of the ASU activation function in deep networks.
• Feature map visualizations reveal that ASU-CNN learns diverse and complex spatial patterns across multiple layers, indicating enhanced representational capacity.
• The use of ASU activation leads to more discriminative feature maps compared to traditional monotonic activations, especially in deeper layers.
• The model shows stable training dynamics and convergence when optimized with Adam and a fine-tuned learning rate.
• The experimental results confirm that non-monotonic activation functions like ASU can be computationally feasible and effective in real-world vision tasks.
• The architecture demonstrates robustness and efficiency, supporting its potential for deployment in resource-constrained environments.