[Paper Review] Harmonic Convolutional Networks based on Discrete Cosine Transform

Matej Uličný, Vladimir A. Krylov+1 more|arXiv (Cornell University)|Jan 18, 2020

Neural Networks and Applications4 citations

TL;DR

This paper proposes harmonic convolutional networks that replace standard convolutional layers with blocks learning weighted combinations of Discrete Cosine Transform (DCT) filters, leveraging DCT's energy compaction for efficient model compression and improved performance. The approach achieves consistent gains of 0.7–1.2% top-1 accuracy on ImageNet, 0.7–1.1 AP on object detection, and 1.1% IoU on semantic segmentation, with minimal computational overhead and strong generalization under varying lighting conditions.

ABSTRACT

Convolutional neural networks (CNNs) learn filters in order to capture local correlation patterns in feature space. We propose to learn these filters as combinations of preset spectral filters defined by the Discrete Cosine Transform (DCT). Our proposed DCT-based harmonic blocks replace conventional convolutional layers to produce partially or fully harmonic versions of new or existing CNN architectures. Using DCT energy compaction properties, we demonstrate how the harmonic networks can be efficiently compressed by truncating high-frequency information in harmonic blocks thanks to the redundancies in the spectral domain. We report extensive experimental validation demonstrating benefits of the introduction of harmonic blocks into state-of-the-art CNN models in image classification, object detection and semantic segmentation applications.

Motivation & Objective

• To improve CNN performance by incorporating spectral prior knowledge via DCT-based filter learning.
• To enable efficient model compression through truncation of high-frequency DCT components.
• To enhance training convergence and generalization, especially under limited data or varying lighting.
• To demonstrate consistent performance gains across diverse vision tasks including classification, detection, and segmentation.
• To provide a plug-and-play alternative to standard convolutional layers with minimal computational overhead.

Proposed method

• Replace standard convolutional layers with harmonic blocks that compute weighted sums of responses to DCT-filtered feature maps.
• Use windowed DCT on local receptive fields to extract spectral coefficients, then learn optimal weights for combining them.
• Leverage DCT's energy compaction property to compress models by truncating high-frequency spectral components with minimal accuracy loss.
• Train harmonic networks end-to-end, allowing backpropagation through the DCT and weighted sum operations.
• Enable conversion of pretrained standard CNNs into harmonic versions by reparameterizing convolutional filters in the DCT domain.
• Apply harmonic blocks to state-of-the-art architectures like ResNet, DenseNet, and DeepLabV3, with ablation on backbone conversion and pretraining.

Experimental results

Research questions

• RQ1 Can DCT-based harmonic blocks improve performance across diverse vision tasks such as image classification, object detection, and semantic segmentation?
• RQ2 Does learning filters as weighted combinations of DCT basis functions lead to faster training convergence and better generalization, especially under data scarcity or lighting variation?
• RQ3 To what extent can harmonic networks be compressed via high-frequency coefficient truncation without significant performance degradation?
• RQ4 How does the performance of harmonic networks compare to standard CNNs in terms of accuracy, inference speed, and memory footprint?
• RQ5 Can harmonic blocks be effectively integrated into existing CNN architectures like ResNet and DeepLabV3 without architectural overhaul?

Key findings

• On the ImageNet dataset, harmonic ResNet-50 and ResNet-101 achieve 0.7–1.2% higher top-1 accuracy than their standard counterparts.
• On the small NORB dataset, harmonic networks achieve state-of-the-art performance and generalize well to unseen lighting conditions.
• In object detection on Pascal VOC and MS COCO, harmonic versions of Faster R-CNN and Mask R-CNN improve AP by 0.7–1.1 points.
• For semantic segmentation on Pascal VOC 2012, harmonic DeepLabV3 with ResNet-101 backbone achieves a 1.1% absolute improvement in mean IoU over the baseline.
• The harmonic blocks introduce only 3–7% computational overhead compared to standard convolutions, with comparable memory footprint.
• The harmonic network shows significant improvements on specific classes such as "chair" (+9.5% IoU), "sheep" (+3.9%), and "boat" (+3.2%) in semantic segmentation.