[Paper Review] Factorized Bilinear Models for Image Recognition

Yanghao Li, Naiyan Wang+2 more|arXiv (Cornell University)|Nov 17, 2016

Advanced Neural Network Applications35 references4 citations

TL;DR

This paper proposes Factorized Bilinear (FB) layers to model pairwise feature interactions in deep neural networks using a low-rank parameterization, enabling quadratic non-linearity with linear computational and parameter complexity. The method achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet with minimal computational overhead and improved generalization via the DropFactor regularization technique.

ABSTRACT

Although Deep Convolutional Neural Networks (CNNs) have liberated their power in various computer vision tasks, the most important components of CNN, convolutional layers and fully connected layers, are still limited to linear transformations. In this paper, we propose a novel Factorized Bilinear (FB) layer to model the pairwise feature interactions by considering the quadratic terms in the transformations. Compared with existing methods that tried to incorporate complex non-linearity structures into CNNs, the factorized parameterization makes our FB layer only require a linear increase of parameters and affordable computational cost. To further reduce the risk of overfitting of the FB layer, a specific remedy called DropFactor is devised during the training process. We also analyze the connection between FB layer and some existing models, and show FB layer is a generalization to them. Finally, we validate the effectiveness of FB layer on several widely adopted datasets including CIFAR-10, CIFAR-100 and ImageNet, and demonstrate superior results compared with various state-of-the-art deep models.

Motivation & Objective

• To enhance the representation capacity of convolutional and fully connected layers in CNNs beyond linear transformations.
• To model pairwise feature interactions efficiently without incurring prohibitive parameter or computational costs.
• To address overfitting in bilinear models through a novel regularization method, DropFactor.
• To generalize bilinear pooling to all layers in a deep network while maintaining scalability.

Proposed method

• Proposes a Factorized Bilinear (FB) layer that models quadratic feature interactions via low-rank factorization of the bilinear transformation matrix.
• Uses a rank-constrained parameterization to reduce the number of parameters and computation cost to linear complexity relative to the input dimension.
• Introduces DropFactor, a regularization method that randomly sets elements of the bilinear factors to zero during training, analogous to Dropout.
• Applies the FB layer to both convolutional and fully connected layers, enabling end-to-end training with standard backpropagation.
• Employs standard data augmentation and initialization strategies when training on ImageNet for fair comparison.

Experimental results

Research questions

• RQ1 Can pairwise feature interactions be modeled efficiently in deep networks without exploding parameter counts?
• RQ2 Does a low-rank factorization of bilinear layers preserve representational power while reducing computational cost?
• RQ3 Can a dedicated regularization method like DropFactor effectively prevent overfitting in bilinear layers?
• RQ4 Does the proposed FB layer improve performance across diverse benchmarks, including small-scale (CIFAR) and large-scale (ImageNet) datasets?

Key findings

• On CIFAR-100, the proposed ResNet-164-FBN model achieves a top-1 error of 23.64%, outperforming the baseline ResNet-164 (5.30% error) and other state-of-the-art models.
• On ImageNet, ResNet-50-FBN reduces the top-1 error to 24.0%, a 0.7% improvement over the baseline ResNet-50 (24.7% error).
• The Inception-BN-FBN variant achieves a top-1 error of 26.4% on ImageNet, a 1.1% improvement over the original Inception-BN (27.5% error).
• On fine-grained datasets, FBN achieves 17.09% error on CUB-200-2011 and 32.20% on DTD, outperforming standard VGG-16 and matching or surpassing bilinear pooling methods with lower computational cost.
• The FBN model maintains low memory and computation costs, with only a linear increase in parameters and inference time compared to standard layers.