[Paper Review] DQ-DETR: DETR with Dynamic Query for Tiny Object Detection

Yi-Xin Huang, Hou-I Liu+2 more|arXiv (Cornell University)|Apr 4, 2024

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

DQ-DETR proposes a dynamic query mechanism for DETR to improve tiny object detection in aerial images by adaptively adjusting the number and positional encoding of object queries using a categorical counting module and density-aware feature enhancement. It achieves state-of-the-art mAP of 30.2% on the AI-TOD-V2 dataset, significantly outperforming prior methods.

ABSTRACT

Despite previous DETR-like methods having performed successfully in generic object detection, tiny object detection is still a challenging task for them since the positional information of object queries is not customized for detecting tiny objects, whose scale is extraordinarily smaller than general objects. Also, DETR-like methods using a fixed number of queries make them unsuitable for aerial datasets, which only contain tiny objects, and the numbers of instances are imbalanced between different images. Thus, we present a simple yet effective model, named DQ-DETR, which consists of three different components: categorical counting module, counting-guided feature enhancement, and dynamic query selection to solve the above-mentioned problems. DQ-DETR uses the prediction and density maps from the categorical counting module to dynamically adjust the number of object queries and improve the positional information of queries. Our model DQ-DETR outperforms previous CNN-based and DETR-like methods, achieving state-of-the-art mAP 30.2% on the AI-TOD-V2 dataset, which mostly consists of tiny objects. Our code will be available at https://github.com/hoiliu-0801/DQ-DETR.

Motivation & Objective

• To address the limitations of fixed-query DETR-like models in detecting tiny objects in aerial images with extreme instance imbalance.
• To improve detection recall in both sparse and dense scenes by dynamically adjusting the number of object queries.
• To enhance positional awareness of queries for tiny objects by fusing density maps with visual features.
• To reduce false positives in sparse images and false negatives in dense images through query number and position adaptation.
• To design a dataset-agnostic categorical counting module that tailors query count based on instance density per image.

Proposed method

• Introduces a categorical counting module that classifies the number of instances per image into four levels: N ≤ 10, 10 < N ≤ 100, 100 < N ≤ 500, and N > 500, based on dataset statistics.
• Uses the predicted count class to dynamically set the number of object queries in the transformer decoder, avoiding fixed-K limitations.
• Generates density maps from the counting module and fuses them with encoder features to enhance foreground representation for tiny objects.
• Applies the enhanced feature map to refine the positional embeddings of object queries, improving localization accuracy.
• Employs a learnable query selection mechanism that adjusts query positions based on the spatial distribution of predicted instances.
• Uses a pseudo-code pipeline that computes mean and variance of instance counts per image to automatically define counting categories without hand-designed thresholds.

Experimental results

Research questions

• RQ1 Can a dynamic query mechanism improve detection performance for tiny objects in aerial images with highly imbalanced instance counts?
• RQ2 How does dynamically adjusting the number of queries affect recall and precision in both sparse and dense scenes?
• RQ3 To what extent does fusing density maps with visual features improve localization of tiny objects?
• RQ4 Can a categorical counting module trained on dataset statistics generalize across different aerial datasets without manual threshold tuning?
• RQ5 How does the proposed method compare to fixed-query and dense-query DETR variants in terms of mAP and AP for small objects?

Key findings

• DQ-DETR achieves a state-of-the-art mAP of 30.2% on the AI-TOD-V2 dataset, surpassing previous SOTA methods by 16.6% in AP and 20.5% in APvt.
• The categorical counting module with four classes achieves 94.6% overall classification accuracy, with 97.7% accuracy for images with ≤10 objects.
• Using five classification classes instead of four reduces mAP by 1.4 points due to poor performance on rare high-density cases (N > 500), confirming the importance of class imbalance handling.
• The ablation study shows that the combination of counting-guided query adjustment and density-enhanced features improves AP by 4.3 points over the baseline DINO-DETR.
• Visualization confirms that DQ-DETR produces more informative feature maps with fewer false positives and false negatives compared to Deformable-DETR.
• The method effectively reduces both false negatives in dense images and false positives in sparse images by adapting query count and position to instance density.