[Paper Review] Adaptable Deformable Convolutions for Semantic Segmentation of Fisheye Images in Autonomous Driving Systems

Clément Playout, Ola Ahmad+2 more|arXiv (Cornell University)|Feb 19, 2021

Advanced Neural Network Applications20 references17 citations

TL;DR

This paper proposes a novel adaptation method for semantic segmentation models to handle fisheye images using deformable convolutions, enabling high performance with minimal labeled fisheye data. By learning spatial offsets for convolutional filters without fine-tuning the main network weights, the approach achieves state-of-the-art performance on fisheye segmentation with as few as 50 training samples, significantly reducing the need for large annotated datasets.

ABSTRACT

Advanced Driver-Assistance Systems rely heavily on perception tasks such as semantic segmentation where images are captured from large field of view (FoV) cameras. State-of-the-art works have made considerable progress toward applying Convolutional Neural Network (CNN) to standard (rectilinear) images. However, the large FoV cameras used in autonomous vehicles produce fisheye images characterized by strong geometric distortion. This work demonstrates that a CNN trained on standard images can be readily adapted to fisheye images, which is crucial in real-world applications where time-consuming real-time data transformation must be avoided. Our adaptation protocol mainly relies on modifying the support of the convolutions by using their deformable equivalents on top of pre-existing layers. We prove that tuning an optimal support only requires a limited amount of labeled fisheye images, as a small number of training samples is sufficient to significantly improve an existing model's performance on wide-angle images. Furthermore, we show that finetuning the weights of the network is not necessary to achieve high performance once the deformable components are learned. Finally, we provide an in-depth analysis of the effect of the deformable convolutions, bringing elements of discussion on the behavior of CNN models.

Motivation & Objective

• To address the challenge of applying pre-trained CNNs to fisheye images, which exhibit strong geometric distortion due to ultra-wide field-of-view (FoV) lenses.
• To reduce dependency on large-scale, time-consuming annotation of real fisheye images by enabling model adaptation with few labeled samples.
• To develop a flexible, plug-and-play adaptation mechanism that preserves pre-trained weights while learning spatial offsets via deformable convolutions.
• To evaluate whether deformable convolutions can effectively model non-linear distortions in fisheye images without retraining the entire network.
• To explore the feasibility of self-supervised offset learning to further reduce annotation costs in future work.

Proposed method

• The method applies deformable convolutions on top of pre-trained, standard CNN layers to adapt them to fisheye images without modifying the original network weights.
• It introduces learnable spatial offsets for convolutional filters, allowing the network to dynamically adjust the sampling locations to account for barrel distortion.
• The deformable components are trained using backpropagation with cross-entropy loss, while the rest of the network remains frozen during adaptation.
• The approach is evaluated on simulated fisheye data with varying degrees of distortion (parameterized by f), using Cityscapes and BlendedMVS datasets.
• The adaptation is performed independently of the main network weights, with only batch normalization layers optionally fine-tuned.
• The method is validated across multiple training set sizes (1, 50, 100, 1000) to assess sample efficiency.

Experimental results

Research questions

• RQ1 Can deformable convolutions be effectively used to adapt a pre-trained CNN for semantic segmentation on fisheye images without retraining the main network?
• RQ2 How many labeled fisheye images are required to achieve significant performance gains using the proposed adaptation method?
• RQ3 Does learning only the deformable offsets, while keeping the original network weights frozen, yield performance comparable to full fine-tuning or retraining?
• RQ4 Can the proposed method generalize across different fisheye camera parameters (e.g., varying f values) without retraining?
• RQ5 Can the adaptation mechanism be extended to self-supervised learning to eliminate the need for annotated fisheye data?

Key findings

• Using only 50 labeled fisheye images, the adapted model achieved a mean Intersection over Union (mIoU) of 0.643 on the Cityscapes dataset with f=125, approaching the performance of a model retrained from scratch.
• The mIoU improved from 0.420 (non-adapted model) to 0.643 with adaptation, demonstrating a significant performance gain with minimal data.
• Even a single labeled fisheye image led to a drop in mIoU to 0.390, indicating that adaptation requires a minimum data threshold, but performance improves rapidly with small data increases.
• The adapted model outperformed a retrained baseline (fish-DL3) with mIoU of 0.615, achieving 0.676 mIoU when using the full 2675-image training set.
• The method achieved high performance without fine-tuning the main network weights, proving that offset learning alone is sufficient for effective adaptation.
• The results suggest that deformable convolutions can effectively model non-linear distortions in fisheye images, enabling robust adaptation with minimal data and computational cost.