[Paper Review] Do We Need Fully Connected Output Layers in Convolutional Networks?

Zhongchao Qian, Tyler L. Hayes+2 more|arXiv (Cornell University)|Apr 28, 2020

Advanced Neural Network Applications32 references4 citations

TL;DR

This paper proposes eliminating the fully connected output layer in convolutional neural networks by directly computing classification scores from the final convolutional feature map using global average pooling. The method achieves comparable accuracy to standard models on ImageNet-1K, CIFAR-100, Stanford Cars-196, and Oxford Flowers-102 while reducing parameters by up to 75%, demonstrating that learned fully connected layers are inefficient and unnecessary for classification.

ABSTRACT

Traditionally, deep convolutional neural networks consist of a series of convolutional and pooling layers followed by one or more fully connected (FC) layers to perform the final classification. While this design has been successful, for datasets with a large number of categories, the fully connected layers often account for a large percentage of the network's parameters. For applications with memory constraints, such as mobile devices and embedded platforms, this is not ideal. Recently, a family of architectures that involve replacing the learned fully connected output layer with a fixed layer has been proposed as a way to achieve better efficiency. In this paper we examine this idea further and demonstrate that fixed classifiers offer no additional benefit compared to simply removing the output layer along with its parameters. We further demonstrate that the typical approach of having a fully connected final output layer is inefficient in terms of parameter count. We are able to achieve comparable performance to a traditionally learned fully connected classification output layer on the ImageNet-1K, CIFAR-100, Stanford Cars-196, and Oxford Flowers-102 datasets, while not having a fully connected output layer at all.

Motivation & Objective

• To investigate whether fully connected output layers are necessary in modern CNNs for image classification.
• To reduce model parameters and memory usage, especially for mobile and embedded applications.
• To evaluate the effectiveness of fixed classifier layers versus eliminating the output layer entirely.
• To determine if global average pooling from the final convolutional layer can replace learned fully connected layers without performance loss.
• To propose a simpler, more parameter-efficient alternative to fixed classifier methods for large-scale classification.

Proposed method

• Replace the final fully connected layer with global average pooling applied to the final convolutional feature map.
• Train the network end-to-end using standard backpropagation, directly optimizing the final convolutional layer for classification.
• Use a fixed identity matrix as the classifier, effectively removing the need for learnable weights in the output layer.
• Apply standard training protocols with cross-entropy loss on the global average pooled features.
• Compare performance and parameter count against models with learned fully connected layers and fixed classifier baselines.
• Evaluate on multiple datasets (ImageNet-1K, CIFAR-100, Stanford Cars-196, Oxford Flowers-102) and architectures (MobileNet-v2, ShuffleNet-v2, etc.).

Experimental results

Research questions

• RQ1 Does removing the fully connected output layer degrade classification performance on large-scale image datasets?
• RQ2 Can global average pooling from the final convolutional layer achieve performance comparable to learned fully connected layers?
• RQ3 How does parameter efficiency compare between models with learned FC layers, fixed classifier layers, and no output layer at all?
• RQ4 Why do fixed classifier methods fail to outperform the proposed method of eliminating the output layer?
• RQ5 To what extent does the final classifier layer contribute to overall model accuracy?

Key findings

• The proposed method achieves comparable top-1 accuracy to standard models on ImageNet-1K, CIFAR-100, Stanford Cars-196, and Oxford Flowers-102 without using a fully connected layer.
• Parameter reduction reaches up to 75% in ShuffleNet-v2 x0.5 (from 1.3M to 0.3M parameters) with only a small drop in ImageNet-1K accuracy.
• On ImageNet-100, the method shows no performance degradation, indicating the accuracy drop on ImageNet-1K is due to model size, not the absence of a classifier layer.
• Fixed classifier methods, including those using Hadamard or polytope-based matrices, are outperformed by simply removing the output layer.
• The final fully connected layer contributes little to classification accuracy despite consuming a large fraction of parameters—especially in efficient models like MobileNet-v2 (37% of parameters).
• The method is limited to datasets with fewer classes than the number of channels in the final convolutional layer, but this constraint is shared with fixed classifier baselines.