[Paper Review] Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

Maoxun Yuan, Yinyan Wang+1 more|arXiv (Cornell University)|Sep 28, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a Translation-Scale-Rotation Alignment (TSRA) module to address cross-modal weakly misalignment in aerial RGB-infrared (RGB-IR) vehicle detection, where objects differ in position, scale, and orientation between modalities. By predicting and correcting these deviations through an alignment process and using a Modality-Selection strategy, the TSFADet detector achieves state-of-the-art 73.06% mAP on the DroneVehicle dataset, outperforming existing methods by addressing hardware and annotation-induced misalignment factors.

ABSTRACT

Integrating multispectral data in object detection, especially visible and infrared images, has received great attention in recent years. Since visible (RGB) and infrared (IR) images can provide complementary information to handle light variations, the paired images are used in many fields, such as multispectral pedestrian detection, RGB-IR crowd counting and RGB-IR salient object detection. Compared with natural RGB-IR images, we find detection in aerial RGB-IR images suffers from cross-modal weakly misalignment problems, which are manifested in the position, size and angle deviations of the same object. In this paper, we mainly address the challenge of cross-modal weakly misalignment in aerial RGB-IR images. Specifically, we firstly explain and analyze the cause of the weakly misalignment problem. Then, we propose a Translation-Scale-Rotation Alignment (TSRA) module to address the problem by calibrating the feature maps from these two modalities. The module predicts the deviation between two modality objects through an alignment process and utilizes Modality-Selection (MS) strategy to improve the performance of alignment. Finally, a two-stream feature alignment detector (TSFADet) based on the TSRA module is constructed for RGB-IR object detection in aerial images. With comprehensive experiments on the public DroneVehicle datasets, we verify that our method reduces the effect of the cross-modal misalignment and achieve robust detection results.

Motivation & Objective

• To identify and analyze the cross-modal weakly misalignment problem in aerial RGB-IR object detection, where objects exhibit deviations in position, scale, and orientation between visible and infrared images.
• To address misalignment caused by hardware limitations (e.g., sensor misalignment) and human annotation errors in paired RGB-IR aerial images.
• To develop a robust, end-to-end trainable two-stream feature alignment detector (TSFADet) that improves multispectral detection accuracy in aerial imagery.
• To evaluate the effectiveness of the TSRA module and its components—alignment process, Modality-Selection strategy, and Multi-task Jitter—on real-world aerial datasets.

Proposed method

• The TSRA module predicts translation, scale, and rotation deviations between RGB and IR feature maps using a learnable alignment process to correct cross-modal misalignment.
• A Modality-Selection (MS) strategy dynamically selects the most accurate bounding-box reference (RGB or IR) for each proposal to reduce error propagation from noisy annotations.
• The Multi-task Jitter technique augments training data by applying random deviations to bounding-boxes, improving the model's robustness to weakly misaligned inputs.
• The TSFADet detector integrates the TSRA module into a two-stream feature alignment framework, enabling end-to-end training for RGB-IR object detection.
• The method uses ResNet-50 as the backbone and is trained with standard data augmentations and a two-stage detection head for improved localization.
• The alignment process is applied per proposal, allowing fine-grained correction of position, scale, and angle discrepancies between modalities.

Experimental results

Research questions

• RQ1 What causes the cross-modal weakly misalignment problem in aerial RGB-IR object detection, particularly in terms of position, scale, and orientation deviations?
• RQ2 How can a deep learning module effectively correct for coupled translation, scale, and rotation deviations between RGB and infrared features in aerial images?
• RQ3 What is the impact of modality selection on alignment accuracy when one modality has more reliable annotations than the other?
• RQ4 Can Multi-task Jitter improve the robustness of the alignment module under varying levels of input misalignment?
• RQ5 Does the proposed TSRA-based detector achieve state-of-the-art performance on aerial RGB-IR object detection benchmarks?

Key findings

• The TSFADet detector achieves 73.06% mAP on the DroneVehicle dataset, outperforming all compared state-of-the-art single- and multi-modality detectors.
• The Modality-Selection strategy significantly improves detection performance by selecting the most accurate reference modality per proposal, reducing error from noisy annotations.
• The Multi-task Jitter technique enhances model robustness, leading to improved alignment accuracy and higher mAP by simulating diverse misalignment patterns during training.
• The TSRA module reduces the impact of cross-modal misalignment, especially in cases of arbitrary object orientation, scale variation, and position shift between modalities.
• When combined with Cascade R-CNN, the TSFADet achieves the highest mAP of 73.90%, demonstrating strong generalization and scalability.
• The method maintains competitive inference speed (18.6 FPS) on a single GPU, outperforming one-stage detectors in accuracy while remaining practical for real-time applications.