[论文解读] Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

Qianli Liao, Tomaso Poggio|arXiv (Cornell University)|Apr 13, 2016

Neural dynamics and brain function参考文献 37被引用 194

一句话总结

该论文表明残差网络在形式上等价于带权共享的浅层循环网络，可推广到模仿腹侧视觉通路处理的多状态循环模型，并在 CIFAR-10 与 ImageNet 上对时间特定批量归一化进行评估。

ABSTRACT

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.

研究动机与目标

• Motivate and study the relationship between Residual Networks (ResNet), Recurrent Neural Networks (RNNs), and the primate visual cortex.
• Demonstrate that a shallow RNN with shared weights can match the performance of very deep ResNets.
• Generalize to a class of biologically plausible multi-state recurrent models for the ventral visual stream and evaluate on CIFAR-10 and ImageNet.
• Introduce time-specific batch normalization (TSBN) and show improved training with ReLUs in RNNs.
• Discuss implications for biological plausibility and future directions in deep learning and neuroscience.

提出的方法

• Establish formal equivalence between ResNet with weight sharing and a specific RNN implementing h_t+1 = K(h_t) + h_t.
• Generalize to a multi-state fully recurrent network (FRNN) on a directed graph modeling ventral stream stages (e.g., LGN, V1, V2, V4, IT).
• Use transition matrices to define inter-state computations and allow time-varying transitions; employ pre-net and post-net components for end-to-end training.
• Incorporate weight sharing schemes and explore readout time t as the unrolling depth, linking biological timing to network depth.
• Propose time-specific batch normalization (TSBN) to stabilize training of RNNs with ReLUs and recurrent connections.

实验结果

研究问题

• RQ1 Can ResNets be formally interpreted as RNNs with shared weights and do they retain performance when unrolled as recurrent systems?
• RQ2 Do multi-state FRNN architectures provide a biologically plausible and effective model of the ventral visual stream for CIFAR-10 and ImageNet?
• RQ3 Does time-specific batch normalization improve training stability and performance for RNNs with recurrent transitions and ReLUs?
• RQ4 How does readout time (unrolling depth) affect accuracy and generalization in ResNet-like and FRNN models?
• RQ5 What are the trade-offs between shared versus non-shared weights in multi-state recurrent architectures across datasets?

主要发现

• A ResNet with shared weights across time is formally equivalent to a shallow RNN unrolled over depth.
• A weight-sharing RNN can retain most of ResNet performance with substantially fewer parameters.
• 3-state and 4-state FRNNs achieve competitive CIFAR-10 results compared to prior best models, with readout time influencing performance.
• On ImageNet, shared-weight 4-state FRNNs can approach results of deeper architectures under certain settings.
• Time-specific Batch Normalization enables stable training of recurrent networks with ReLUs, addressing previous training difficulties.
• The models show better alignment with biological plausibility, offering insights into how cortex-like recurrent processing could support rapid visual recognition.