[Paper Review] Locally Smoothed Neural Networks

Liang Pang, Yanyan Lan+3 more|arXiv (Cornell University)|Nov 22, 2017

Face and Expression Recognition17 references3 citations

TL;DR

This paper proposes Locally Smoothed Neural Networks (LSNN), a novel architecture that enhances local receptive field modeling by factorizing the weight matrix of locally connected layers into a kernel and a content- and location-aware smoother. Using a multi-variate Gaussian function to generate the smoother, LSNN captures spatial relationships and content-dependent importance of receptive fields, outperforming CNNs and locally connected layers on MNIST variants, especially in tasks requiring selective attention to multiple relevant regions.

ABSTRACT

Convolutional Neural Networks (CNN) and the locally connected layer are limited in capturing the importance and relations of different local receptive fields, which are often crucial for tasks such as face verification, visual question answering, and word sequence prediction. To tackle the issue, we propose a novel locally smoothed neural network (LSNN) in this paper. The main idea is to represent the weight matrix of the locally connected layer as the product of the kernel and the smoother, where the kernel is shared over different local receptive fields, and the smoother is for determining the importance and relations of different local receptive fields. Specifically, a multi-variate Gaussian function is utilized to generate the smoother, for modeling the location relations among different local receptive fields. Furthermore, the content information can also be leveraged by setting the mean and precision of the Gaussian function according to the content. Experiments on some variant of MNIST clearly show our advantages over CNN and locally connected layer.

Motivation & Objective

• To address the limitations of CNNs and locally connected layers in modeling the importance and spatial relationships of local receptive fields.
• To enable neural networks to dynamically focus on relevant image regions based on content and spatial context.
• To design a differentiable, learnable mechanism that balances invariance (like CNNs) and flexibility (like locally connected layers).
• To improve performance on tasks requiring selective attention, such as visual question answering and multi-digit recognition in cluttered scenes.

Proposed method

• Factorize the weight matrix of a locally connected layer into a product of a shared kernel and a smoother matrix.
• Use a multi-variate Gaussian function to generate the smoother, encoding spatial proximity and correlation among local receptive fields.
• Adapt the mean and precision of the Gaussian based on input content using a regression head, enabling content-dependent attention.
• Train the model end-to-end with backpropagation, allowing the smoother to be learned from data.
• Extend LSNN to multiple smoothers for multi-object attention by using separate Gaussian groups per target region.
• Integrate LSNN into a CNN framework, allowing it to attend to relevant image patches before classification.

Experimental results

Research questions

• RQ1 Can a learnable smoother improve the modeling of local receptive field importance beyond fixed weight sharing in CNNs?
• RQ2 How well can a content-based smoother automatically identify relevant image regions based on contextual cues?
• RQ3 Can LSNN effectively attend to multiple related regions in a single image, such as multiple digits in a cluttered scene?
• RQ4 How does LSNN compare to CNN and locally connected layers in terms of accuracy and generalization on structured local feature tasks?
• RQ5 Does the spatial structure of the smoother (via multi-variate Gaussian) lead to better localization and robustness in attention?

Key findings

• LSNN achieved a 6.18% error rate on the cluttered MNIST sequence task, significantly outperforming the locally connected layer (13.5%) and convolutional layer (23.97%).
• The content-based smoother enabled LSNN to automatically focus on the digit pointed to by an arrow, even when the arrow's position varied.
• Visualization confirmed that LSNN's smoother concentrated on the correct digit region, with the attention mechanism operating independently from the classification network.
• In multi-digit scenarios, LSNN successfully identified and attended to three distinct, spatially related digits using multiple Gaussian-based smoothers.
• The model demonstrated that spatial smoothness and content-awareness jointly improved performance, especially in complex, cluttered environments.
• LSNN was shown to be a generalization of both CNN (identical smoother) and locally connected layer (free smoother), providing a continuous trade-off between invariance and flexibility.