[Paper Review] Deep Learning using Rectified Linear Units (ReLU)

Abien Fred Agarap|arXiv (Cornell University)|Mar 22, 2018

Neural Networks and Applications13 references2,484 citations

TL;DR

This study tests using ReLU as the classification function in deep neural networks, comparing DL-ReLU against DL-Softmax across MNIST, Fashion-MNIST, and WDBC datasets using FFNN and CNN architectures.

ABSTRACT

We introduce the use of rectified linear units (ReLU) as the classification function in a deep neural network (DNN). Conventionally, ReLU is used as an activation function in DNNs, with Softmax function as their classification function. However, there have been several studies on using a classification function other than Softmax, and this study is an addition to those. We accomplish this by taking the activation of the penultimate layer $h_{n - 1}$ in a neural network, then multiply it by weight parameters $θ$ to get the raw scores $o_{i}$. Afterwards, we threshold the raw scores $o_{i}$ by $0$, i.e. $f(o) = \max(0, o_{i})$, where $f(o)$ is the ReLU function. We provide class predictions $\hat{y}$ through argmax function, i.e. argmax $f(x)$.

Motivation & Objective

• Motivate replacing Softmax as the final classifier with ReLU in deep networks.
• Evaluate DL-ReLU performance against DL-Softmax on standard benchmarks.
• Analyze training convergence, accuracy, and class-wise metrics across architectures and datasets.
• Identify potential drawbacks of ReLU classification and suggest future improvements.

Proposed method

• Use two network types (FFNN and CNN) with Softmax and ReLU last-layer classifiers.
• Train with Adam optimizer using identical hyper-parameters to enable fair comparison.
• Preprocess data with normalization and PCA for MNIST/Fashion-MNIST to reduce dimensionality.
• Replace the Softmax cross-entropy loss with a ReLU-based cross-entropy formulation and backpropagate gradients as usual.
• Evaluate using 10-fold cross-validation, test accuracy, precision, recall, F1-score, and confusion matrices.

Experimental results

Research questions

• RQ1 Does replacing Softmax with ReLU at the classification layer yield comparable or superior accuracy to Softmax-based models on MNIST, Fashion-MNIST, and WDBC?
• RQ2 How does ReLU-based classification affect training convergence and learning dynamics across FFNN and CNN architectures?
• RQ3 What are the class-wise performance patterns (precision/recall/F1) when using ReLU as the final classifier?
• RQ4 What limitations (e.g., dying ReLU) affect DL-ReLU performance and how might they be mitigated?

Key findings

• DL-ReLU often achieves comparable performance to DL-Softmax across datasets and architectures.
• On MNIST, FFNN-ReLU nearly matches FFNN-Softmax in test accuracy (97.77% vs 97.98%).
• CNN-ReLU on MNIST converges slower and trails CNN-Softmax in cross-validation accuracy (73.53% vs 97.23%), but reaches 91.74% test accuracy.
• On Fashion-MNIST, FFNN-ReLU is close to FFNN-Softmax (89.06% vs 89.35% test accuracy).
• CNN-ReLU on Fashion-MNIST has lower cross-validation accuracy than CNN-Softmax but similar test accuracy (85.84% vs 86.08%).
• On WDBC, FFNN-ReLU underperforms FFNN-Softmax in both cross-validation and test (90.64% vs 92.40% test accuracy; F1 0.90 vs 0.92).