[Paper Review] Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution

Yunpeng Chen, Haoqi Fan+6 more|arXiv (Cornell University)|Apr 10, 2019

Advanced Neural Network ApplicationsComputer Science50 references149 citations

TL;DR

The paper introduces Octave Convolution (OctConv), a plug-and-play operation that factorizes feature maps into high- and low-frequency octaves to reduce spatial redundancy, lowering memory/compute while improving accuracy on image and video tasks.

ABSTRACT

In natural images, information is conveyed at different frequencies where higher frequencies are usually encoded with fine details and lower frequencies are usually encoded with global structures. Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies. In this work, we propose to factorize the mixed feature maps by their frequencies, and design a novel Octave Convolution (OctConv) operation to store and process feature maps that vary spatially "slower" at a lower spatial resolution reducing both memory and computation cost. Unlike existing multi-scale methods, OctConv is formulated as a single, generic, plug-and-play convolutional unit that can be used as a direct replacement of (vanilla) convolutions without any adjustments in the network architecture. It is also orthogonal and complementary to methods that suggest better topologies or reduce channel-wise redundancy like group or depth-wise convolutions. We experimentally show that by simply replacing convolutions with OctConv, we can consistently boost accuracy for both image and video recognition tasks, while reducing memory and computational cost. An OctConv-equipped ResNet-152 can achieve 82.9% top-1 classification accuracy on ImageNet with merely 22.2 GFLOPs.

Motivation & Objective

• Motivate and model that natural images contain information at multiple spatial frequencies requiring separate processing.
• Propose a generic, plug-and-play Octave Convolution unit that replaces vanilla convolutions without architectural changes.
• Show that OctConv reduces memory and FLOPs while boosting accuracy across 2D and 3D backbones on ImageNet and Kinetics.
• Demonstrate compatibility with group and depth-wise convolutions and analyze receptive field advantages and alignment considerations.

Proposed method

• Define octave feature representation by splitting input channels into high-frequency XH and low-frequency XL groups, with XL at half the spatial resolution (one octave).
• Devise Octave Convolution by decomposing kernels into intra- and inter-frequency components to update YH and YL with four computation paths.
• Compute YH = f(XH; WHH) + upsample(f(XL; LHH), 2) and YL = f(XL; LLL) + pool(f(XH; HLH), 2), enabling inter-frequency information exchange.
• Implement practical details using average pooling for down-sampling to avoid misalignment and maintain efficiency.
• Offer variants for group and depth-wise convolutions to integrate OctConv into existing architectures without major redesigns.

Experimental results

Research questions

• RQ1 Does replacing vanilla convolutions with OctConv improve accuracy for image and video recognition tasks?
• RQ2 What is the FLOPs/memory trade-off when using OctConv across different backbone architectures?
• RQ3 How does the low-frequency channel ratio α affect performance and efficiency?
• RQ4 Is OctConv compatible with group and depth-wise convolutions and other efficiency-oriented CNN designs?
• RQ5 What is the impact of OctConv on receptive field and information exchange between frequency groups?

Key findings

• OctConv-equipped networks consistently improve accuracy while reducing FLOPs across multiple backbones on ImageNet and Kinetics.
• The FLOPs-accuracy trade-off with OctConv is concave, with sweet spots around α = 0.125–0.25 offering substantial gains.
• OctConv yields practical speedups close to theoretical FLOP reductions, e.g., ResNet-50 showing notable speed improvements on CPUs.
• Low-frequency maps benefit from a doubled effective receptive field, enabling better contextual understanding without extra memory.
• Both intra- and inter-frequency exchange paths are important for maximizing performance, and OctConv benefits shallow networks more due to larger receptive field gains.
• Compared with MG-Conv and related multi-scale methods, OctConv achieves better FLOPs-accuracy with lower memory and computation.