[Paper Review] Committees of deep feedforward networks trained with few data

Bogdan Miclut, Thomas Kaester+2 more|arXiv (Cornell University)|Jun 23, 2014

Advanced Neural Network Applications8 references4 citations

TL;DR

This paper proposes a committee of shallow deep feedforward networks trained in a fully unsupervised, layer-wise manner without backpropagation to improve image classification on small datasets. By combining multiple networks trained with data augmentation (mirroring, rotation, scaling) and using a voting scheme, the method achieves 68.0% accuracy on the STL-10 dataset—surpassing prior state-of-the-art results—using only two-layer networks and linear SVMs for classification.

ABSTRACT

Deep convolutional neural networks are known to give good results on image classification tasks. In this paper we present a method to improve the classification result by combining multiple such networks in a committee. We adopt the STL-10 dataset which has very few training examples and show that our method can achieve results that are better than the state of the art. The networks are trained layer-wise and no backpropagation is used. We also explore the effects of dataset augmentation by mirroring, rotation, and scaling.

Motivation & Objective

• To improve image classification performance on small labeled datasets by combining multiple deep feedforward networks in a committee.
• To explore whether unsupervised, layer-wise training without backpropagation can yield competitive results on image recognition tasks with limited training data.
• To evaluate the impact of data augmentation (mirroring, rotation, scaling) on the performance of individual and committee networks.
• To determine whether ensemble averaging of diverse network outputs improves generalization beyond individual network performance.
• To demonstrate that simple, shallow architectures with unsupervised feature learning can achieve state-of-the-art results on benchmark datasets like STL-10.

Proposed method

• Feature extraction is performed via layer-wise unsupervised training: filters are learned using k-means clustering on extracted patches from input feature maps.
• Each network layer applies patch-wise preprocessing (normalization, ZCA whitening), followed by convolution with learned filters, absolute value rectification, and local contrast normalization.
• Pooling is applied via a generalized norm: $ y = \left(\sum_i x_i^\alpha\none\right)^{1/\alpha} $, with $ \alpha = 1 $ (average pooling) showing optimal performance.
• Connections between layers use random grouping of feature maps (e.g., groups of 4) to reduce dimensionality and improve convergence during k-means clustering.
• Data augmentation is applied during training by mirroring, rotating, and scaling input images to generate diverse training samples for each network in the committee.
• Final classification is performed by averaging normalized scores from all committee members and selecting the class with the highest summed score.

Experimental results

Research questions

• RQ1 Can a committee of unsupervised, shallow deep feedforward networks outperform individual networks on small-data image classification tasks?
• RQ2 Does data augmentation (mirroring, rotation, scaling) significantly improve performance when combined with unsupervised feature learning?
• RQ3 To what extent can a voting scheme across multiple networks mitigate the limitations of individual shallow networks trained without backpropagation?
• RQ4 How does the choice of pooling strategy and rectification method affect the final classification accuracy in the absence of backpropagation?
• RQ5 Can a simple, non-backpropagation-based deep network architecture achieve state-of-the-art performance on the STL-10 dataset with minimal labeled data?

Key findings

• The committee of networks achieved 68.0% top-1 accuracy on the STL-10 dataset, surpassing the previous state of the art of 67.4% reported in [6].
• Using only mirroring and rotation as data augmentation, the committee achieved 67.39% accuracy, demonstrating the effectiveness of augmentation in unsupervised settings.
• The best-performing individual network (N2) achieved 64.59% accuracy with mirroring and rotation, while the committee improved this to 68.0%, showing consistent gains from ensemble averaging.
• Average pooling ($ \alpha = 1 $) outperformed max pooling and other $ \alpha $ values in the generalized pooling operation, indicating better feature representation stability.
• ON-OFF rectification (separating positive and negative responses) yielded a 64.73% accuracy with mirroring, showing that nonlinearity choice affects performance but is less impactful than data augmentation.
• The committee’s performance was consistently better than any single network, and performance saturated or slightly decreased only when poor-quality networks were added, indicating that diversity and quality of members are critical.