[Paper Review] Enhanced Training of Query-Based Object Detection via Selective Query Recollection

Fangyi Chen, Han Zhang+4 more|arXiv (Cornell University)|Dec 15, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Selective Query Recollection (SQR), a training strategy that enhances query-based object detectors by selectively forwarding intermediate queries from earlier decoding stages to later stages, mitigating cascading errors and improving training emphasis on later stages. SQR boosts AP by 1.4–2.8 across multiple models including Adamixer, DAB-DETR, and Deformable-DETR without altering inference speed.

ABSTRACT

This paper investigates a phenomenon where query-based object detectors mispredict at the last decoding stage while predicting correctly at an intermediate stage. We review the training process and attribute the overlooked phenomenon to two limitations: lack of training emphasis and cascading errors from decoding sequence. We design and present Selective Query Recollection (SQR), a simple and effective training strategy for query-based object detectors. It cumulatively collects intermediate queries as decoding stages go deeper and selectively forwards the queries to the downstream stages aside from the sequential structure. Such-wise, SQR places training emphasis on later stages and allows later stages to work with intermediate queries from earlier stages directly. SQR can be easily plugged into various query-based object detectors and significantly enhances their performance while leaving the inference pipeline unchanged. As a result, we apply SQR on Adamixer, DAB-DETR, and Deformable-DETR across various settings (backbone, number of queries, schedule) and consistently brings 1.4-2.8 AP improvement.

Motivation & Objective

• To investigate why query-based object detectors sometimes mispredict at the final decoding stage despite correct intermediate predictions.
• To identify two key limitations: insufficient training emphasis on later decoding stages and cascading errors from sequential refinement.
• To design a training strategy that improves supervision on later stages and reduces error propagation.
• To develop a plug-in solution that enhances performance without modifying the inference pipeline.
• To explore model compression via dense query recollection and recurrence, enabling parameter sharing across decoding stages.

Proposed method

• SQR cumulatively collects intermediate queries from all previous decoding stages as training progresses.
• It selectively forwards queries from only the prior two stages to each downstream stage, forming a Fibonacci-like supervision growth pattern.
• Each stage treats both original and recollected queries independently, enabling separate attention and loss computation.
• The method maintains the original sequential decoding structure during inference, ensuring no speed penalty.
• For model compression, SQR is extended to DQRR, where the final decoding stage is reused iteratively by feeding its output back as input during training.
• The final model uses a single shared stage for inference, reducing model size by 70% while maintaining high accuracy.

Figure 1 : The inference speed and AP for various networks on the MS-COCO val set. The red stars are the results trained with SQR. The blue circles are the results of baselines without SQR. SQR enhances the training of query-based object detectors while leaving the inference pipeline unchanged.

Experimental results

Research questions

• RQ1 Why do query-based object detectors sometimes degrade in performance at the final decoding stage despite correct predictions at earlier stages?
• RQ2 How does the sequential structure of decoding stages contribute to cascading errors during training?
• RQ3 Can training be improved by increasing supervision on later decoding stages without altering the inference process?
• RQ4 What is the optimal strategy for selecting which intermediate queries to recollect to balance performance and computational cost?
• RQ5 Can dense query recollection enable parameter sharing across decoding stages to reduce model size?

Key findings

• SQR improves AP by 1.4–2.8 points across multiple query-based detectors, including Adamixer, DAB-DETR, and Deformable-DETR, under various training settings.
• On Adamixer-R50 with 100 queries and 12 epochs, SQR increases AP from 42.5 to 44.4 (+1.9).
• With 7 decoder stages, SQR boosts Adamixer-R50 from 45.1 to 48.9 AP (+3.8), and Adamixer-R101 from 47.6 to 49.8 AP (+2.2).
• SQR achieves consistent gains across different backbones (R50, R101, Swin-B), query counts (100–300), and training schedules (12e, 36e, 50e).
• The DQRR variant, using dense query recollection and recurrence, reduces model size by 70% (1.6GB to 513MB) and achieves 42.8 AP with only 5 shared stages, outperforming the original 6-stage model (42.5 AP).
• The selective recollection strategy, forwarding only queries from the two prior stages, achieves better performance than full recollection while halving the computational cost.

Figure 2 : Are query-based object detectors always enhancing predictions stage-by-stage? The traffic light at stage 1 gets a confident score of 0.41, while from stage 2 to 5 the confidence gradually decreases to 0.21 (Upper); the remote at stage 3 was wrongly classified as a cell phone, and from sta