[Paper Review] KernelWarehouse: Towards Parameter-Efficient Dynamic Convolution

Chao Li, Anbang Yao|arXiv (Cornell University)|Aug 16, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

KernelWarehouse proposes a parameter-efficient dynamic convolution method that redefines kernel representation by partitioning convolutional kernels into cells and sharing a global 'warehouse' of reusable kernel cells across layers. By enabling high-dimensional kernel mixing with minimal parameter growth, it achieves state-of-the-art accuracy—e.g., 81.05% top-1 on ImageNet with ResNet50—while reducing model size by up to 65.10% and improving accuracy by 2.29%.

ABSTRACT

Dynamic convolution learns a linear mixture of $n$ static kernels weighted with their sample-dependent attentions, demonstrating superior performance compared to normal convolution. However, existing designs are parameter-inefficient: they increase the number of convolutional parameters by $n$ times. This and the optimization difficulty lead to no research progress in dynamic convolution that can allow us to use a significant large value of $n$ (e.g., $n>100$ instead of typical setting $n<10$) to push forward the performance boundary. In this paper, we propose $KernelWarehouse$, a more general form of dynamic convolution, which can strike a favorable trade-off between parameter efficiency and representation power. Its key idea is to redefine the basic concepts of "$kernels$" and "$assembling$ $kernels$" in dynamic convolution from the perspective of reducing kernel dimension and increasing kernel number significantly. In principle, KernelWarehouse enhances convolutional parameter dependencies within the same layer and across successive layers via tactful kernel partition and warehouse sharing, yielding a high degree of freedom to fit a desired parameter budget. We validate our method on ImageNet and MS-COCO datasets with different ConvNet architectures, and show that it attains state-of-the-art results. For instance, the ResNet18|ResNet50|MobileNetV2|ConvNeXt-Tiny model trained with KernelWarehouse on ImageNet reaches 76.05%|81.05%|75.52%|82.51% top-1 accuracy. Thanks to its flexible design, KernelWarehouse can even reduce the model size of a ConvNet while improving the accuracy, e.g., our ResNet18 model with 36.45%|65.10% parameter reduction to the baseline shows 2.89%|2.29% absolute improvement to top-1 accuracy.

Motivation & Objective

• To address the parameter inefficiency of existing dynamic convolution methods that scale parameters by n times when using n dynamic kernels.
• To overcome the limitation of using only small n (e.g., n < 10) due to model size growth, enabling exploration of larger n (e.g., n > 100) for improved representation.
• To exploit both intra-layer and inter-layer parameter dependencies in convolutional kernels to enhance representation power without proportional parameter increase.
• To design a flexible, scalable architecture that allows for model compression and accuracy gains simultaneously.

Proposed method

• KernelWarehouse partitions each convolutional kernel into m disjoint kernel cells of equal size, reducing per-kernel dimensionality.
• It defines a shared 'warehouse' of n kernel cells (e.g., n = 108) that is reused across multiple convolutional layers, enabling parameter sharing across layers.
• Each kernel cell is computed as a linear mixture of the warehouse cells using input-dependent attentions, with a novel attention function: α_ij = z_ij / Σ_p |z_ip|.
• The attention mechanism is initialized using a temperature-scaled β_ij to encourage one-to-one correspondence between mixtures and warehouse cells, improving early training stability.
• The method integrates a new attention function that allows negative values to model adversarial attention relationships, improving representation learning.
• The architecture supports flexible hyperparameter tuning, enabling trade-offs between model size and accuracy through scaling factors (e.g., 1/2×, 1×, 4×).

Experimental results

Research questions

• RQ1 Can dynamic convolution achieve high representation power without proportionally increasing model parameters?
• RQ2 Can intra-layer and inter-layer parameter dependencies in convolutional kernels be leveraged to reduce parameter growth while maintaining or improving performance?
• RQ3 Does a shared, global warehouse of kernel cells enable better generalization and efficiency compared to per-layer kernel sets?
• RQ4 Can a novel attention function with negative values improve the learning of attention relationships in dynamic convolution?
• RQ5 Can KernelWarehouse simultaneously reduce model size and improve accuracy, especially when scaling the number of dynamic kernels?

Key findings

• KernelWarehouse achieves 81.05% top-1 accuracy on ImageNet using ResNet50, outperforming the baseline by 2.61 percentage points with only 28.05M parameters.
• The method reduces model size by 65.10% (to 9.2M parameters) while improving top-1 accuracy by 2.29% on ResNet18 compared to the baseline.
• On MobileNetV2, KernelWarehouse achieves 75.92% top-1 accuracy with 11.38M parameters, a 3.90% accuracy gain over the baseline, while reducing model size by 67.5%.
• The proposed attention function α_ij = z_ij / Σ_p |z_ip| outperforms Softmax, Sigmoid, and ReLU-based alternatives by 2.10–2.68% in top-1 accuracy.
• The attention initialization strategy with temperature scaling improves top-1 accuracy by 1.38% on ResNet18, demonstrating its importance for early training stability.
• Visualization shows that attentions learn structured, diagonal-dominant patterns, indicating effective one-to-one mapping between mixtures and warehouse cells, and stronger intra-layer dependencies than inter-layer ones.