[Paper Review] IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks

Ke Sun, Mingjie Li+2 more|arXiv (Cornell University)|May 31, 2018

Advanced Neural Network Applications100 citations

TL;DR

IGCV3 combines low-rank and sparse kernels via interleaved low-rank group convolutions to build efficient dense kernels, improving accuracy over IGCV2 and MobileNetV2 on CIFAR/ImageNet and COCO while maintaining similar parameter counts.

ABSTRACT

In this paper, we are interested in building lightweight and efficient convolutional neural networks. Inspired by the success of two design patterns, composition of structured sparse kernels, e.g., interleaved group convolutions (IGC), and composition of low-rank kernels, e.g., bottle-neck modules, we study the combination of such two design patterns, using the composition of structured sparse low-rank kernels, to form a convolutional kernel. Rather than introducing a complementary condition over channels, we introduce a loose complementary condition, which is formulated by imposing the complementary condition over super-channels, to guide the design for generating a dense convolutional kernel. The resulting network is called IGCV3. We empirically demonstrate that the combination of low-rank and sparse kernels boosts the performance and the superiority of our proposed approach to the state-of-the-arts, IGCV2 and MobileNetV2 over image classification on CIFAR and ImageNet and object detection on COCO.

Motivation & Objective

• Motivate the design of lightweight CNNs with less redundancy.
• Blend low-rank and structured sparsity to form dense convolutional kernels.
• Introduce a loose complementary (super-channel) condition to guide kernel composition.
• Empirically evaluate IGCV3 against IGCV2 and MobileNetV2 on vision benchmarks.

Proposed method

• Propose IGCV3 as a block that stacks channel-wise spatial convolution, low-rank group point-wise convolutions (two stages that reduce then restore width), with permutations to maintain dense connectivity.
• Represent low-rank group convolutions via block-sparse matrices and super-channels to manage differing input/output channel counts.
• Use a loose complementary condition over super-channels to guide the arrangement of branches in the two group convolutions.
• Provide an inverted IGCV3 variant (IGCV3-D) that follows an inverted bottleneck style to save memory during training/inference.
• Evaluate networks by comparing parameter counts, accuracy on CIFAR-10/100 and ImageNet, and object detection on COCO, against MobileNetV2 and IGCV2.

Experimental results

Research questions

• RQ1 Does integrating low-rank group convolutions with interleaved sparse structure improve accuracy for a fixed parameter budget vs. prior mobile architectures?
• RQ2 How does the loose complementary condition over super-channels impact the design and performance of IGCV3 blocks?
• RQ3 What are the trade-offs between deeper vs. wider IGCV3 configurations for classification and detection tasks?
• RQ4 How does IGCV3 perform on standard benchmarks (CIFAR, ImageNet, COCO) relative to IGCV2 and MobileNetV2?

Key findings

• IGCV3-D (deeper version with two low-rank group convolutions) achieves 94.96% on CIFAR-10 and 77.95% on CIFAR-100 with 2.2M params, outperforming IGCV2 and MobileNetV2 baselines at similar parameter counts.
• On ImageNet, IGCV3-D reaches 72.2% top-1 under comparable compute, surpassing several mobile-network baselines.
• IGCV3 with 50% of MobileNetV2 parameters still offers competitive or superior accuracy across CIFAR and ImageNet, due to the combination of low-rank and sparse design.
• IGCV3 outperforms MobileNetV2 in classification on CIFAR/imagenet and improves COCO detection when used as a backbone (SSDLite2), with fewer parameters.
• Deeper IGCV3 configurations benefit more from increased depth than merely widening the network, consistent with bottleneck/ResNet-style findings.
• Ablation studies show the second block with intermediate ReLUs and the two low-rank group convolutions effectively balance capacity and memory.