[Paper Review] 3D Depthwise Convolution: Reducing Model Parameters in 3D Vision Tasks

Rongtian Ye, Fangyu Liu+1 more|arXiv (Cornell University)|Aug 5, 2018

Advanced Vision and Imaging18 references4 citations

TL;DR

This paper proposes 3D depthwise convolution to drastically reduce model parameters in 3D vision tasks by factorizing standard 3D convolutions into depthwise and pointwise operations, achieving up to 95% parameter reduction with comparable performance to standard 3D convolutions in classification and reconstruction tasks, enabling deeper and more efficient 3D networks under resource constraints.

ABSTRACT

Standard 3D convolution operations require much larger amounts of memory and computation cost than 2D convolution operations. The fact has hindered the development of deep neural nets in many 3D vision tasks. In this paper, we investigate the possibility of applying depthwise separable convolutions in 3D scenario and introduce the use of 3D depthwise convolution. A 3D depthwise convolution splits a single standard 3D convolution into two separate steps, which would drastically reduce the number of parameters in 3D convolutions with more than one order of magnitude. We experiment with 3D depthwise convolution on popular CNN architectures and also compare it with a similar structure called pseudo-3D convolution. The results demonstrate that, with 3D depthwise convolutions, 3D vision tasks like classification and reconstruction can be carried out with more light-weighted neural networks while still delivering comparable performances.

Motivation & Objective

• To address the high computational and memory cost of standard 3D convolution in 3D vision tasks.
• To explore parameter-efficient alternatives to standard 3D convolutions for resource-constrained 3D neural networks.
• To propose and evaluate 3D depthwise convolution as a method to reduce model size while maintaining performance.
• To demonstrate the effectiveness of 3D depthwise convolution in both classification and 3D reconstruction tasks.
• To enable the use of deeper 3D ConvNets by reducing parameter count through depthwise factorization.

Proposed method

• Factorizes standard 3D convolution into two stages: depthwise convolution per input channel and pointwise 1×1×1 convolution across channels.
• Applies depthwise separable operations to 3D feature maps of shape (l, w, h, c), using c separate k×k×k filters on each channel before combining via 1×1×1 convolutions.
• Analyzes parameter efficiency mathematically, showing a reduction of over 10× in parameters compared to standard 3D convolutions.
• Compares 3D depthwise convolution with pseudo-3D convolution, which splits filters into spatial and temporal components, showing lower parameter count and comparable performance.
• Integrates 3D depthwise convolution into standard architectures like VGG and residual blocks, and applies it in 3D reconstruction decoders.
• Uses a residual block-based decoder (ResRec) with depthwise convolutions to build deeper, smaller models for 3D reconstruction.

Experimental results

Research questions

• RQ1 Can depthwise separable convolutions be effectively extended to 3D convolutional operations to reduce model parameters?
• RQ2 How does 3D depthwise convolution compare to pseudo-3D convolution in terms of parameter efficiency and performance?
• RQ3 Can 3D depthwise convolution maintain high performance in 3D classification and reconstruction tasks while significantly reducing model size?
• RQ4 Does using 3D depthwise convolution enable the training of deeper 3D ConvNets under memory and computational constraints?
• RQ5 What is the impact of model depth and architecture design on 3D reconstruction performance when using parameter-efficient convolutions?

Key findings

• 3D depthwise convolution reduces the number of parameters in 3D convolutional layers by up to 95% compared to standard 3D convolutions, with minimal performance drop in classification tasks.
• In 3D reconstruction, a deeper 3D depthwise ConvNet decoder (ResRec-16 dw) with fewer parameters outperformed a shallower standard 3D ConvNet decoder with more parameters, achieving 63.1 mIoU.
• The ResRec-16 dw model achieved 61.5 mIoU, outperforming the standard ResRec-16 (63.1 mIoU) when using depthwise convolutions, indicating strong efficiency and performance trade-offs.
• Qualitative results show that 3D depthwise convolution preserves finer details better than pseudo-3D convolution, which tends to produce smoother but less detailed surfaces.
• Deeper 3D ConvNets, such as ResRec-16, significantly outperform shallower models like 3D-R2N2, even when using the same encoder, highlighting the importance of depth in 3D vision tasks.
• The use of 3D depthwise convolution enables the construction of deeper, more complex 3D networks under strict computational and memory constraints, making previously infeasible models viable.