[Paper Review] Learning Implicitly Recurrent CNNs Through Parameter Sharing

Pedro Savarese, Michael Maire|arXiv (Cornell University)|Feb 1, 2019

Advanced Neural Network Applications19 citations

TL;DR

This paper proposes a parameter sharing scheme that enables convolutional neural networks (CNNs) to implicitly learn recurrent structures through a learned linear combination of shared parameter templates. By restricting the number of templates, the method achieves substantial parameter efficiency while maintaining accuracy and enabling fast, generalizable training on algorithmic tasks—offering a hybrid CNN-recurrent architecture with performance competitive to neural architecture search.

ABSTRACT

We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Compared to traditional CNNs, we demonstrate substantial parameter savings on standard image classification tasks, while maintaining accuracy. Our simple parameter sharing scheme, though defined via soft weights, in practice often yields trained networks with near strict recurrent structure; with negligible side effects, they convert into networks with actual loops. Training these networks thus implicitly involves discovery of suitable recurrent architectures. Though considering only the design aspect of recurrent links, our trained networks achieve accuracy competitive with those built using state-of-the-art neural architecture search (NAS) procedures. Our hybridization of recurrent and convolutional networks may also represent a beneficial architectural bias. Specifically, on synthetic tasks which are algorithmic in nature, our hybrid networks both train faster and extrapolate better to test examples outside the span of the training set.

Motivation & Objective

• To design a parameter sharing mechanism that allows CNNs to implicitly learn recurrent architectures without explicit recurrent units.
• To reduce model parameters while maintaining or improving classification accuracy on standard image benchmarks.
• To explore whether implicit recurrence through parameter sharing can yield better generalization and faster training on algorithmic, out-of-distribution tasks.
• To evaluate whether this hybrid architecture can achieve performance comparable to state-of-the-art neural architecture search (NAS) methods.

Proposed method

• The method defines different layers of a CNN as a learned linear combination of parameter tensors drawn from a global bank of shared templates.
• The number of templates is restricted to enforce parameter sharing, creating a flexible hybrid between traditional CNNs and recurrent networks.
• The parameter sharing is implemented via soft weights, allowing gradients to update the combination coefficients during training.
• Despite being defined with soft weights, trained networks often converge to near-strict recurrent structures with actual loops, enabling efficient inference.
• The approach implicitly discovers suitable recurrent architectures through end-to-end training, without explicit architectural search.

Experimental results

Research questions

• RQ1 Can parameter sharing across layers induce implicit recurrence in CNNs, leading to effective recurrent behavior?
• RQ2 Does this implicit recurrence lead to better parameter efficiency and generalization on algorithmic tasks?
• RQ3 Can the resulting hybrid architecture achieve accuracy competitive with state-of-the-art NAS methods?
• RQ4 How does the number of shared templates affect model performance and parameter efficiency?

Key findings

• The proposed method achieves substantial parameter savings on standard image classification tasks while maintaining competitive accuracy.
• Trained networks often converge to near-strict recurrent structures, which can be converted into actual recurrent networks with negligible performance drop.
• On synthetic algorithmic tasks, the hybrid networks train faster and generalize better to out-of-distribution test examples than standard CNNs.
• The method achieves performance comparable to state-of-the-art neural architecture search (NAS) procedures, despite not using explicit architecture search.