[Paper Review] SimNets: A Generalization of Convolutional Networks

Nadav Cohen, Amnon Shashua|arXiv (Cornell University)|Oct 3, 2014

Advanced Neural Network Applications28 references18 citations

TL;DR

SimNets introduce a generalization of convolutional neural networks using a similarity operator that extends convolution and a novel MEX (soft max-min-mean) activation that generalizes ReLU and pooling. The architecture enables state-of-the-art accuracy with networks 9× smaller than comparable ConvNets, supports natural unsupervised initialization, and offers a higher abstraction level than kernel machines.

ABSTRACT

We present a deep layered architecture that generalizes classical convolutional neural networks (ConvNets). The architecture, called SimNets, is driven by two operators, one being a similarity function whose family contains the convolution operator used in ConvNets, and the other is a new soft max-min-mean operator called MEX that realizes classical operators like ReLU and max pooling, but has additional capabilities that make SimNets a powerful generalization of ConvNets. Three interesting properties emerge from the architecture: (i) the basic input to hidden layer to output machinery contains as special cases kernel machines with the Exponential and Generalized Gaussian kernels, the output units being "neurons in feature space" (ii) in its general form, the basic machinery has a higher abstraction level than kernel machines, and (iii) initializing networks using unsupervised learning is natural. Experiments demonstrate the capability of achieving state of the art accuracy with networks that are an order of magnitude smaller than comparable ConvNets.

Motivation & Objective

• To develop a deep layered architecture that generalizes classical ConvNets by lifting their core operations into a more abstract, kernel-like framework.
• To address the limitations of ConvNets in model compactness and the lack of effective unsupervised initialization by introducing a natural initialization scheme based on unlabeled data.
• To bridge the gap between kernel machines and deep learning by embedding kernel methods (e.g., SVM with Exponential and Generalized Gaussian kernels) within a deep network structure.
• To enable automatic determination of network depth and channel count via variance analysis of unsupervised feature patterns.
• To demonstrate that the proposed architecture can achieve superior or comparable performance with significantly fewer parameters than state-of-the-art ConvNets.

Proposed method

• Introduce a similarity operator that generalizes the convolutional inner product using $ l_p $-norms with learnable weights, enabling a broader hypothesis space than standard kernel machines.
• Propose the MEX operator—a soft max-min-mean function—that generalizes ReLU and pooling layers, with learnable offsets that allow for locality-based biases in feature representation.
• Formulate the network as a multilayer kernel network where each layer applies the similarity and MEX operators in sequence, enabling higher-level abstraction than traditional ConvNet layers.
• Enable unsupervised initialization via statistical estimation of Generalized Gaussian mixture distributions on unlabeled data, which naturally determines initial parameters and channel counts.
• Use weighted $ l_p $ similarity to extend beyond the hypothesis space of kernel-SVMs, introducing a new building block with stronger representational capacity.
• Implement a basic SimNet architecture and benchmark it against equivalent-depth ConvNets and a single-layer network from prior work on CIFAR-10.

Experimental results

Research questions

• RQ1 Can a deep network architecture be designed that generalizes ConvNets while enabling higher abstraction and compactness?
• RQ2 Does the inclusion of learnable weights in the $ l_p $ similarity operator provide a representational advantage beyond standard kernel machines?
• RQ3 Can unsupervised learning be effectively leveraged to initialize SimNet parameters, and does it outperform random initialization?
• RQ4 Can the network structure (e.g., number of channels per layer) be automatically determined from unlabeled data using variance analysis?
• RQ5 Can SimNets achieve state-of-the-art performance with significantly fewer parameters than comparable ConvNets?

Key findings

• SimNets achieved comparable or better accuracy than state-of-the-art ConvNets on CIFAR-10 with approximately 1/9 the number of learned parameters.
• The inclusion of weights in the $ l_p $ similarity operator led to a super-linear increase in performance with respect to parameter count, indicating a representational advantage beyond kernel-SVMs.
• Unsupervised initialization based on Generalized Gaussian mixture estimation outperformed random initialization, especially in small network settings.
• The MEX layer with learnable offsets enabled locality-based biases not expressible in standard ConvNets, suggesting enhanced modeling flexibility.
• The architecture naturally supports automatic channel determination via variance analysis of unsupervised feature patterns, enabling data-driven network structure design.
• The weighted $ l_p $ similarity operator cannot be realized in the kernel-SVM framework, proving its capacity to access a larger hypothesis space than traditional kernel machines.