[Paper Review] ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks

Francesco Visin, Kyle Kastner+4 more|arXiv (Cornell University)|May 3, 2015

Advanced Neural Network ApplicationsComputer Science58 references234 citations

TL;DR

ReNet proposes a recurrent neural network architecture that replaces convolutional and pooling layers in deep networks with four unidirectional RNNs—two horizontal and two vertical—processing feature maps in both directions. Evaluated on MNIST, CIFAR-10, and SVHN, ReNet achieves competitive accuracy (0.45% test error on MNIST, 12.35% on CIFAR-10, 2.38% on SVHN), demonstrating that RNNs can serve as a viable alternative to CNNs for image recognition tasks.

ABSTRACT

In this paper, we propose a deep neural network architecture for object recognition based on recurrent neural networks. The proposed network, called ReNet, replaces the ubiquitous convolution+pooling layer of the deep convolutional neural network with four recurrent neural networks that sweep horizontally and vertically in both directions across the image. We evaluate the proposed ReNet on three widely-used benchmark datasets; MNIST, CIFAR-10 and SVHN. The result suggests that ReNet is a viable alternative to the deep convolutional neural network, and that further investigation is needed.

Motivation & Objective

• To explore whether recurrent neural networks (RNNs) can serve as a viable alternative to convolutional neural networks (CNNs) for image recognition tasks.
• To address the limitation of local receptive fields in CNNs by enabling global context modeling through sequential processing of image features.
• To design a computationally efficient RNN-based architecture that avoids the exponential complexity of multidimensional RNNs while maintaining global context awareness.
• To evaluate the performance of the proposed ReNet architecture on standard benchmark datasets for object recognition.
• To demonstrate that RNNs, when properly structured, can achieve performance comparable to state-of-the-art CNNs on image classification tasks.

Proposed method

• The ReNet architecture replaces each convolutional + pooling layer in a CNN with four unidirectional RNNs: bottom-to-top, top-to-bottom, left-to-right, and right-to-left, each processing feature maps along their respective directions.
• Each RNN processes non-overlapping patches of the input image, with hidden states updated sequentially along the sweep direction, enabling each output activation to depend on the entire image context.
• The model uses two separate RNNs per direction (forward and reverse) to capture long-range dependencies, with the final output formed by concatenating or combining features from all four RNNs.
• The architecture is modular and hierarchical, with each layer operating on the feature maps from the previous layer, allowing for deep hierarchical representation learning.
• Gated recurrent units (GRUs or LSTMs) are used instead of simple RNNs to better capture long-term dependencies across the image.
• The model is trained using standard backpropagation through time, with parallelization enabled by processing forward and reverse RNNs independently.

Experimental results

Research questions

• RQ1 Can a recurrent neural network architecture achieve performance comparable to convolutional neural networks on standard image recognition benchmarks?
• RQ2 Does the use of bidirectional RNNs sweeping across image features enable better modeling of global spatial context than local convolutional operations?
• RQ3 How does the proposed ReNet architecture compare in accuracy and efficiency to state-of-the-art CNNs on MNIST, CIFAR-10, and SVHN?
• RQ4 What is the impact of using gated recurrent units (GRUs/LSTMs) versus standard RNNs in the ReNet architecture for image recognition?
• RQ5 Can a purely RNN-based architecture effectively learn hierarchical representations of image data without inductive biases from convolutional operations?

Key findings

• ReNet achieved a test error of 0.45% on the MNIST dataset, placing it among the top-performing models and comparable to state-of-the-art CNNs.
• On CIFAR-10, ReNet achieved a test error of 12.35%, which is competitive with other deep learning models but does not surpass the best-performing CNNs.
• On the SVHN dataset, ReNet achieved a test error of 2.38%, demonstrating strong performance on a more complex, real-world image classification task.
• The results indicate that ReNet is a viable alternative to CNNs for image recognition, particularly given its ability to model long-range spatial dependencies through sequential processing.
• The use of gated recurrent units (GRUs or LSTMs) significantly improved performance over standard RNNs, suggesting effective learning of long-term spatial dependencies.
• Despite its competitive accuracy, ReNet did not outperform state-of-the-art CNNs on any of the three benchmark datasets, indicating room for further architectural innovation and optimization.