[Paper Review] Deep Networks with Internal Selective Attention through Feedback Connections

Marijn Stollenga, Jonathan Masci+2 more|arXiv (Cornell University)|Jul 11, 2014

Advanced Memory and Neural Computing45 references129 citations

TL;DR

This paper proposes Deep Attention Selective Networks (dasNet), a deep learning architecture that introduces feedback connections to enable internal selective attention in convolutional neural networks. By using reinforcement learning via Separable Natural Evolution Strategies (SNES), dasNet dynamically modulates filter sensitivities across multiple inference passes, improving classification accuracy on CIFAR-10 and CIFAR-100 by correcting initial misclassifications through iterative attention, achieving a new state-of-the-art result with a 6% relative improvement over vanilla CNNs.

ABSTRACT

Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.

Motivation & Objective

• To develop a non-stationary deep neural network that can adapt its behavior post-training to improve classification on difficult examples.
• To enable internal selective attention in deep CNNs by introducing feedback connections from higher to lower layers.
• To learn a control policy for filter modulation using scalable reinforcement learning in a high-dimensional parameter space.
• To outperform standard feedforward CNNs on challenging image classification benchmarks like CIFAR-10 and CIFAR-100.
• To demonstrate that feedback-driven attention can correct misclassifications made by a pre-trained CNN through iterative refinement.

Proposed method

• The architecture uses Maxout networks with dropout as the base model, enabling robust feature learning through max-pooling over filter outputs.
• Feedback connections are introduced from higher to lower layers, allowing top-down modulation of convolutional filter activities during inference.
• A control policy is learned via Separable Natural Evolution Strategies (SNES) to determine which filters to emphasize or suppress across multiple inference passes.
• The policy operates in a non-differentiable, high-dimensional space (>1 million parameters), making traditional backpropagation infeasible, hence the use of SNES for direct policy search.
• The network performs multiple inference passes on the same input, with filter weights dynamically adjusted based on the policy, enabling iterative refinement of feature representations.
• Classification is performed after each pass, and the final prediction is based on the sequence of activations, with the policy favoring features that improve confidence in correct classes.

Experimental results

Research questions

• RQ1 Can feedback connections in a deep CNN enable internal selective attention that improves classification accuracy on difficult examples?
• RQ2 Can a high-dimensional control policy for filter modulation be effectively learned using evolution strategies rather than gradient-based methods?
• RQ3 Does iterative refinement through multiple inference passes allow a network to correct its own misclassifications after an initial feedforward pass?
• RQ4 To what extent do the learned attention mechanisms alter filter activations in a non-uniform, task-relevant manner across layers?
• RQ5 Can the final gate values from the policy contain sufficient information for classification, indicating effective use of attention?

Key findings

• dasNet achieved a 6% relative improvement in classification accuracy over the baseline vanilla CNN on the CIFAR-10 dataset.
• On CIFAR-100, dasNet outperformed the previous state-of-the-art model, establishing a new SOTA result.
• The network successfully corrected an initial misclassification of a cat as a dog by emphasizing relevant filters in higher layers and suppressing others, leading to a correct final prediction.
• Analysis showed that filter modulations were complex and non-uniform, involving both emphasis and suppression of specific feature patterns, not simple global scaling.
• The final gate values from the policy contained significant predictive information, achieving 40.70% accuracy with 15-nearest neighbor and 45.74% with logistic regression, confirming their functional relevance.
• The dynamics of the network remained stable even when evaluated beyond the training horizon (up to 10 steps), indicating robustness of the learned policy.