[Paper Review] SORT: Second-Order Response Transform for Visual Recognition

Yan Wang, Lingxi Xie+4 more|arXiv (Cornell University)|Mar 20, 2017

Domain Adaptation and Few-Shot Learning34 references4 citations

TL;DR

This paper proposes Second-Order Response Transform (SORT), a lightweight module that enhances deep neural networks by appending an element-wise product operation to the linear sum of two-branch features. By introducing second-order interactions, SORT improves cross-branch information flow and nonlinearity, leading to consistent accuracy gains across CIFAR and ImageNet benchmarks with under 5% inference overhead.

ABSTRACT

In this paper, we reveal the importance and benefits of introducing second-order operations into deep neural networks. We propose a novel approach named Second-Order Response Transform (SORT), which appends element-wise product transform to the linear sum of a two-branch network module. A direct advantage of SORT is to facilitate cross-branch response propagation, so that each branch can update its weights based on the current status of the other branch. Moreover, SORT augments the family of transform operations and increases the nonlinearity of the network, making it possible to learn flexible functions to fit the complicated distribution of feature space. SORT can be applied to a wide range of network architectures, including a branched variant of a chain-styled network and a residual network, with very light-weighted modifications. We observe consistent accuracy gain on both small (CIFAR10, CIFAR100 and SVHN) and big (ILSVRC2012) datasets. In addition, SORT is very efficient, as the extra computation overhead is less than 5%.

Motivation & Objective

• Address the limited representational capacity of linear fusion in deep networks, which restricts their ability to model complex feature distributions.
• Overcome the limitations of standard residual and multi-branch networks that rely solely on linear summation for feature fusion.
• Develop a simple yet effective module that enhances nonlinearity and cross-branch information flow without significant computational cost.
• Enable consistent performance gains across diverse architectures, including chain-styled and residual networks, on both small-scale and large-scale visual recognition tasks.
• Demonstrate the transferability of SORT-boosted features to downstream tasks like transfer learning on Caltech256.

Proposed method

• Propose a two-branch network module where outputs are fused via a second-order operation: $\mathbf{F}_1(\mathbf{x}) + \mathbf{F}_2(\mathbf{x}) + \mathbf{F}_1(\mathbf{x}) \odot \mathbf{F}_2(\mathbf{x})$, combining linear sum and element-wise product.
• Apply SORT to residual blocks by modifying the identity shortcut connection to $\mathbf{x} + \mathbf{F}(\mathbf{x}) + \sqrt{\mathbf{x} \odot \mathbf{F}(\mathbf{x})}$, introducing nonlinearity while preserving residual learning.
• Introduce cross-branch response propagation: each branch updates its weights based on the current state of the other via the product term during backpropagation.
• Ensure computational efficiency by adding only element-wise operations, resulting in less than 5% extra computation and no additional memory usage.
• Integrate SORT into various architectures, including branched variants of AlexNet and ResNet, with minimal architectural changes.
• Use standard training protocols and evaluation metrics on CIFAR10, CIFAR100, SVHN, and ILSVRC2012 to validate performance across datasets and network depths.

Experimental results

Research questions

• RQ1 Can introducing second-order operations like element-wise product improve the representational capacity of deep neural networks beyond linear fusion?
• RQ2 Does the addition of a second-order transform enhance cross-branch information flow during both forward and backward propagation?
• RQ3 Can SORT be efficiently applied to diverse architectures such as chain-styled networks and residual networks without significant computational cost?
• RQ4 Does the improved nonlinearity from SORT lead to consistent accuracy gains across small-scale (CIFAR) and large-scale (ImageNet) datasets?
• RQ5 To what extent do features learned with SORT generalize to transfer learning scenarios on external datasets like Caltech256?

Key findings

• On CIFAR10, SORT reduced top-1 error from 36.71% to 35.99% when applied to a branched AlexNet variant, representing a 1.96% relative improvement.
• On CIFAR100, SORT reduced top-1 error from 14.77% to 14.46%, achieving a 2.10% relative reduction compared to the baseline.
• On ILSVRC2012, SORT reduced top-1 error on ResNet-18 from 34.50% to 32.37%, a 6.17% relative improvement, and on ResNetT-50 from 23.82% to 23.10%.
• The method incurred less than 5% extra computation time on a 4-GPU machine, with no additional memory consumption, confirming its efficiency.
• Transfer learning experiments on Caltech256 showed that features from SORT-enhanced models achieved 74.88% accuracy on the pool-5 layer, outperforming the baseline AlexNet* (74.20%) and AlexNet (69.19%).
• Learning curves indicate that while SORT may slow convergence slightly in some cases, it consistently improves final generalization performance across architectures.