[Paper Review] QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection

Chenhongyi Yang, Zehao Huang+1 more|arXiv (Cornell University)|Mar 16, 2021

Advanced Neural Network ApplicationsComputer Science73 references21 citations

TL;DR

QueryDet proposes a cascaded sparse query mechanism (CSQ) that accelerates high-resolution small object detection by first predicting coarse object locations on low-resolution features and then applying sparse computation on high-resolution features only at those locations. This reduces computation by up to 3.0× on COCO and 2.3× on VisDrone while improving mAP-small by 2.0 and achieving state-of-the-art performance.

ABSTRACT

While general object detection with deep learning has achieved great success in the past few years, the performance and efficiency of detecting small objects are far from satisfactory. The most common and effective way to promote small object detection is to use high-resolution images or feature maps. However, both approaches induce costly computation since the computational cost grows squarely as the size of images and features increases. To get the best of two worlds, we propose QueryDet that uses a novel query mechanism to accelerate the inference speed of feature-pyramid based object detectors. The pipeline composes two steps: it first predicts the coarse locations of small objects on low-resolution features and then computes the accurate detection results using high-resolution features sparsely guided by those coarse positions. In this way, we can not only harvest the benefit of high-resolution feature maps but also avoid useless computation for the background area. On the popular COCO dataset, the proposed method improves the detection mAP by 1.0 and mAP-small by 2.0, and the high-resolution inference speed is improved to 3.0x on average. On VisDrone dataset, which contains more small objects, we create a new state-of-the-art while gaining a 2.3x high-resolution acceleration on average. Code is available at https://github.com/ChenhongyiYang/QueryDet-PyTorch.

Motivation & Objective

• To address the high computational cost of detecting small objects in high-resolution feature maps.
• To reduce redundant computation in feature pyramid-based detectors by focusing only on likely small object locations.
• To maintain high detection accuracy for small objects while significantly accelerating inference.
• To enable efficient deployment of small object detectors on real-time systems like autonomous vehicles.
• To design a general-purpose method compatible with various detectors, backbones, and acceleration techniques.

Proposed method

• QueryDet uses a cascaded sparse query (CSQ) mechanism to predict coarse object locations on low-resolution feature maps.
• These predicted locations serve as queries to guide sparse computation on high-resolution features, avoiding full-convolutional processing.
• Sparse convolution is applied to the detection head, processing only features within a context patch (e.g., 5×5) around each query position.
• The query head is trained with focal loss to improve localization confidence in sparse regions.
• The method is compatible with both one-stage (e.g., RetinaNet, FCOS) and two-stage (e.g., Faster R-CNN) detectors.
• The approach is modular and can be combined with lightweight backbones, quantization, and knowledge distillation.

Experimental results

Research questions

• RQ1 Can a query-based mechanism reduce computation in high-resolution feature maps without sacrificing small object detection accuracy?
• RQ2 How much context (e.g., patch size) is needed to maintain high detection AP while enabling acceleration?
• RQ3 Can the proposed CSQ mechanism be generalized across different detector architectures, including anchor-free and two-stage detectors?
• RQ4 How does the method perform on datasets with a high density of small objects, such as VisDrone?
• RQ5 To what extent can the method be combined with other acceleration techniques like lightweight backbones?

Key findings

• On COCO, QueryDet improves mAP by 1.0 and mAP-small by 2.0 when using high-resolution features, with a 3.0× average speedup in high-resolution inference.
• On VisDrone, QueryDet achieves a new state-of-the-art mAP-small performance with a 2.3× average acceleration in high-resolution detection.
• With MobileNet V2, QueryDet achieves an average 4.1× speedup for high-resolution detection, demonstrating strong edge deployment potential.
• When applied to FCOS, a state-of-the-art anchor-free detector, QueryDet improves mAP-small and achieves a 1.8× speedup in high-resolution inference.
• In Faster R-CNN, CSQ increases inference speed from 17.57 to 19.03 FPS with only a minor drop in mAP-small (22.98 to 22.98), confirming effectiveness in two-stage detectors.
• Visualization shows that query heatmaps successfully localize small objects, though failure cases include false positives on large objects and missed detections due to insufficient context.