[Paper Review] Non-Parametric Transformation Networks

Dipan K. Pal, Marios Savvides|arXiv (Cornell University)|Jan 14, 2018

Graph Theory and Algorithms34 references3 citations

TL;DR

This paper introduces Non-Parametric Transformation Networks (NPTNs), a new class of deep convolutional architectures that learn general invariances and symmetries directly from data without assuming a parametric form for transformations. NPTNs generalize ConvNets by using transformation pooling over non-parametrically learned filters, enabling superior performance on MNIST, CIFAR10, and ETH-80 with the same number of parameters, and significantly improving Capsule Networks by replacing their convolutional layers with NPTN modules.

ABSTRACT

ConvNets, through their architecture, only enforce invariance to translation. In this paper, we introduce a new class of deep convolutional architectures called Non-Parametric Transformation Networks (NPTNs) which can learn extit{general} invariances and symmetries directly from data. NPTNs are a natural generalization of ConvNets and can be optimized directly using gradient descent. Unlike almost all previous works in deep architectures, they make no assumption regarding the structure of the invariances present in the data and in that aspect are flexible and powerful. We also model ConvNets and NPTNs under a unified framework called Transformation Networks (TN), which yields a better understanding of the connection between the two. We demonstrate the efficacy of NPTNs on data such as MNIST with extreme transformations and CIFAR10 where they outperform baselines, and further outperform several recent algorithms on ETH-80. They do so while having the same number of parameters. We also show that they are more effective than ConvNets in modelling symmetries and invariances from data, without the explicit knowledge of the added arbitrary nuisance transformations. Finally, we replace ConvNets with NPTNs within Capsule Networks and show that this enables Capsule Nets to perform even better.

Motivation & Objective

• To develop a deep learning architecture that can learn general invariances and symmetries from data without assuming a parametric form for the transformations.
• To generalize ConvNets by extending their invariance mechanism beyond translation to arbitrary, data-observed transformations.
• To enable end-to-end training with standard optimization methods like SGD while maintaining computational efficiency and memory efficiency.
• To demonstrate that modeling invariances directly through architecture, rather than data augmentation or auxiliary objectives, leads to improved performance on vision benchmarks.
• To integrate NPTNs into emerging architectures like Capsule Networks to enhance their invariance learning and overall performance.

Proposed method

• NPTNs are built on a unified framework called Transformation Networks (TN), which generalizes ConvNets by replacing fixed weight sharing with learnable, non-parametric transformations.
• Each NPTN node applies a convolution operation using a set of $|G|$ transformed filters (templates) derived from a base filter $w$, where $g$ denotes the transformation applied to $w$.
• The output is computed as the dot product between the input patch and each transformed filter, producing $|G|$ scalar responses.
• Transformation pooling is applied across the $|G|$ channels (representing different transformations) via max-pooling, resulting in an output invariant to the transformations encoded in $G$.
• The network is trained using standard backpropagation and stochastic gradient descent, with no need to explicitly transform inputs, activations, or filters during training or inference.
• The method supports integration into existing architectures such as Capsule Networks by replacing standard convolutional layers with NPTN layers while preserving parameter count.

Experimental results

Research questions

• RQ1 Can a deep neural network architecture learn general invariances to arbitrary, non-parametric transformations directly from data without prior structural assumptions?
• RQ2 How does the performance of NPTNs compare to standard ConvNets when both are constrained to the same number of parameters on benchmarks with complex transformations?
• RQ3 To what extent can NPTNs improve the performance of state-of-the-art architectures like Capsule Networks by replacing their convolutional components?
• RQ4 Does learning invariances through architectural design (via transformation pooling) outperform data augmentation or auxiliary loss-based invariance enforcement?
• RQ5 Can NPTNs effectively model symmetries and invariances even when the underlying transformation group is unknown or complex?

Key findings

• NPTNs outperform vanilla ConvNets on MNIST with extreme transformations, achieving significantly lower test error despite having fewer channels and the same number of parameters.
• On CIFAR10, NPTNs achieve better generalization than baseline ConvNets under the same parameter budget, demonstrating improved invariance learning on natural image data.
• On ETH-80, NPTNs outperform several recent state-of-the-art algorithms, including those using data augmentation and auxiliary losses, while maintaining identical parameter counts.
• When replacing convolutional layers in Capsule Networks with NPTN layers, test error decreased from 1.90% to 0.78% with only one-third the number of channels, showing substantial performance gains.
• The performance gain in Capsule Networks using NPTNs indicates that modeling invariance within each capsule improves the overall representation learning and object recognition capability.
• NPTNs effectively learn invariances without prior knowledge of the transformation group, demonstrating their ability to capture complex, non-parametric symmetries directly from data.