[Paper Review] Dynamic Mobile-Former: Strengthening Dynamic Convolution with Attention and Residual Connection in Kernel Space

Seokju Yun, Youngmin Ro|arXiv (Cornell University)|Apr 13, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Dynamic Mobile-Former (DMF), a lightweight vision model that enhances dynamic convolution by integrating input-agnostic and input-dependent kernels via residual connections in kernel space, while using lightweight attention to guide kernel selection. DMF achieves 79.4% top-1 accuracy on ImageNet-1K with only 1/4 the FLOPs of PVT-Tiny, outperforming state-of-the-art models in efficiency and accuracy on detection and segmentation tasks.

ABSTRACT

We introduce Dynamic Mobile-Former(DMF), maximizes the capabilities of dynamic convolution by harmonizing it with efficient operators.Our Dynamic MobileFormer effectively utilizes the advantages of Dynamic MobileNet (MobileNet equipped with dynamic convolution) using global information from light-weight attention.A Transformer in Dynamic Mobile-Former only requires a few randomly initialized tokens to calculate global features, making it computationally efficient.And a bridge between Dynamic MobileNet and Transformer allows for bidirectional integration of local and global features.We also simplify the optimization process of vanilla dynamic convolution by splitting the convolution kernel into an input-agnostic kernel and an input-dependent kernel.This allows for optimization in a wider kernel space, resulting in enhanced capacity.By integrating lightweight attention and enhanced dynamic convolution, our Dynamic Mobile-Former achieves not only high efficiency, but also strong performance.We benchmark the Dynamic Mobile-Former on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection, and instanace segmentation.For example, our DMF hits the top-1 accuracy of 79.4% on ImageNet-1K, much higher than PVT-Tiny by 4.3% with only 1/4 FLOPs.Additionally,our proposed DMF-S model performed well on challenging vision datasets such as COCO, achieving a 39.0% mAP,which is 1% higher than that of the Mobile-Former 508M model, despite using 3 GFLOPs less computations.Code and models are available at https://github.com/ysj9909/DMF

Motivation & Objective

• To address the optimization instability and limited representational power in dynamic convolution by rethinking kernel space learning.
• To improve the efficiency and performance of lightweight vision models under strict FLOP and latency constraints.
• To enable dynamic convolution to scale effectively by decoupling input-agnostic and input-dependent kernels.
• To integrate global context from lightweight attention into kernel attention modules for better filter selection.
• To achieve a favorable accuracy-FLOP trade-off in mobile and real-time vision applications.

Proposed method

• Decomposes the convolution kernel into an input-agnostic component and an input-dependent component to decouple optimization and expand kernel space.
• Introduces a residual connection in kernel space that connects the input-dependent kernel to the input-agnostic kernel, stabilizing training and improving optimization.
• Uses sigmoid activation for attention scores to increase kernel space capacity, counterbalancing optimization difficulty through residual learning.
• Employs lightweight global attention to extract salient global features, which are used as input to the kernel attention module for better filter selection.
• Combines dynamic residual convolution with group convolution and depthwise separable convolutions to maintain computational efficiency.
• Utilizes temperature annealing and careful initialization (e.g., zero-initialization for static kernels) to stabilize training and improve convergence.

Figure 3 : A Dynamic Residual Convolution layer. Firstly, the global average pooled features of the input and the first global token are concatenated. These concatenated features are then passed through a kernel attention module to generate attention scores ( $\pi$ ) and obtain dynamic convolution w

Experimental results

Research questions

• RQ1 Can dynamic convolution be made more stable and scalable through kernel space decomposition and residual learning?
• RQ2 How does using global context from lightweight attention improve the performance of dynamic convolution?
• RQ3 Can a larger kernel space be effectively utilized without degrading training stability?
• RQ4 What is the impact of activation functions (sigmoid vs. softmax) on the performance and optimization of dynamic convolution?
• RQ5 How does the proposed dynamic residual convolution compare to standard dynamic convolution in terms of FLOPs, parameters, and accuracy?

Key findings

• DMF-S achieves 79.4% top-1 accuracy on ImageNet-1K, which is 4.3% higher than PVT-Tiny, while using only 1/4 of its FLOPs.
• DMF-S achieves 39.0% mAP on COCO instance segmentation, outperforming the Mobile-Former 508M model by 1% while using 3 GFLOPs less computation.
• The model with 8 static kernels and sigmoid-based attention scores achieves 73.6% top-1 accuracy, outperforming models with fewer kernels or softmax activation.
• Zero initialization of static kernels improves performance over random initialization, confirming the benefit of residual learning in kernel space.
• The ablation study confirms that sigmoid activation and residual connections significantly enhance model capacity and training stability.
• DMF demonstrates a strong accuracy-FLOP trade-off, outperforming both lightweight CNNs and vision transformer variants in real-time and mobile vision scenarios.