[Paper Review] Task-Driven Convolutional Recurrent Models of the Visual System

Aran Nayebi, Daniel M. Bear+6 more|arXiv (Cornell University)|Jun 20, 2018

Neural dynamics and brain functionNeuroscience32 references121 citations

TL;DR

This paper integrates local recurrence and long-range feedback into CNNs to form ConvRNNs, discovers effective local recurrence cells via hyperparameter search, and shows these models match primate visual dynamics as well as deeper feedforward nets with fewer parameters.

ABSTRACT

Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback connections useful for object recognition. Moreover, these task-optimized ConvRNNs matched the dynamics of neural activity in the primate visual system better than feedforward networks, suggesting a role for the brain's recurrent connections in performing difficult visual behaviors.

Motivation & Objective

• Motivate and quantify the role of recurrence and feedback in visual object recognition beyond feedforward CNNs.
• Develop ConvRNNs that can operate at ImageNet scale with realistic cortical-like timing.
• Identify local recurrent cell motifs and long-range feedback patterns that improve performance.
• Assess whether task-optimized ConvRNNs predict neural dynamics in the primate ventral visual stream.

Proposed method

• Augment standard CNN backbones with local recurrent cells and long-range feedback using a time-stepped update rule aligned to ~10 ms cortical conduction delays.
• Compare ConvRNNs to parameter-matched wider/deeper feedforward baselines and minimally unrolled controls to isolate recurrence effects.
• Introduce and evaluate the Reciprocal Gated Cell that combines bypassing and gating within a local recurrent unit.
• Automate architecture search (Tree-structured Parzen Estimator) over hundreds of ConvRNN variants with varying local and global recurrence.
• Train models on ImageNet (and 128 px subset for faster search) and compare Top1 accuracy to ResNet-18/34 baselines.
• Fit ConvRNN features to macaque V4, pIT, and cIT/aIT neural dynamics using linear mappings and time-resolved predictions.

Experimental results

Research questions

• RQ1 Does adding standard recurrent cells (vanilla RNNs, LSTMs) to CNNs improve ImageNet performance beyond parameter-matched baselines?
• RQ2 Can novel local recurrent cells that include gating and bypassing outperform standard cells in deep CNNs on ImageNet?
• RQ3 What architecture motifs (local recurrence and long-range feedback) emerge as optimal for object recognition, and how do they scale with network depth?
• RQ4 Do task-optimized ConvRNNs capture temporal neural dynamics in the primate ventral visual stream better than feedforward models?

Key findings

• Standard RNNs and LSTMs do not substantially improve ImageNet performance beyond parameter-matched baselines.
• A Reciprocal Gated Cell with gating and bypassing substantially improves accuracy while using fewer parameters than LSTMs.
• Hyperparameter search reveals depth-separable convolutions for updating hidden states and selective long-range feedbacks are beneficial.
• A median ConvRNN after training matches a ResNet-34 with about 75% as many parameters (15.5M vs 21.8M) and achieves near-top-1 accuracy compared to the deeper model (73.1% vs 73.1% reported for ResNet-34 in text).
• ConvRNNs trained on ImageNet predict primate ventral-stream neural dynamics (V4 to IT) as well as or better than feedforward features, with persistent temporal trajectories improving late-phase predictions; time-constant only models do not match ConvRNN performance.
• Fully trained ConvRNNs can outperform feedforward baselines in predicting single-image neural dynamics across time, indicating recurrence captures rich temporal structure beyond initial feedforward responses.