[Paper Review] Negative Frames Matter in Egocentric Visual Query 2D Localization

Mengmeng Xu, Cheng-Yang Fu+4 more|arXiv (Cornell University)|Aug 3, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes a faster, more accurate solution for egocentric visual query 2D localization by optimizing training efficiency and reducing false positives through negative frame training. By accelerating training from 15 days to under 24 hours and incorporating negative frames into the detector's training, the method achieves 0.17% spatial-temporal AP—31% higher than the baseline—ranking first on the Ego4D public leaderboard.

ABSTRACT

The recently released Ego4D dataset and benchmark significantly scales and diversifies the first-person visual perception data. In Ego4D, the Visual Queries 2D Localization task aims to retrieve objects appeared in the past from the recording in the first-person view. This task requires a system to spatially and temporally localize the most recent appearance of a given object query, where query is registered by a single tight visual crop of the object in a different scene. Our study is based on the three-stage baseline introduced in the Episodic Memory benchmark. The baseline solves the problem by detection and tracking: detect the similar objects in all the frames, then run a tracker from the most confident detection result. In the VQ2D challenge, we identified two limitations of the current baseline. (1) The training configuration has redundant computation. Although the training set has millions of instances, most of them are repetitive and the number of unique object is only around 14.6k. The repeated gradient computation of the same object lead to an inefficient training; (2) The false positive rate is high on background frames. This is due to the distribution gap between training and evaluation. During training, the model is only able to see the clean, stable, and labeled frames, but the egocentric videos also have noisy, blurry, or unlabeled background frames. To this end, we developed a more efficient and effective solution. Concretely, we bring the training loop from ~15 days to less than 24 hours, and we achieve 0.17% spatial-temporal AP, which is 31% higher than the baseline. Our solution got the first ranking on the public leaderboard. Our code is publicly available at https://github.com/facebookresearch/vq2d_cvpr.

Motivation & Objective

• To address inefficiencies in training and evaluation for egocentric visual query 2D localization.
• To reduce high false positive rates on background and negative frames, which degrade final retrieval performance.
• To close the distribution gap between pre-trained models (e.g., COCO) and egocentric video data (Ego4D).
• To improve model robustness on blurry, out-of-focus, or ambiguous frames by explicitly training on negative examples.
• To accelerate the research cycle through code refactoring and parallelization, enabling faster experimentation.

Proposed method

• Replaced the original 15-day training loop with a restructured, parallelized training and evaluation pipeline, reducing training time to under 24 hours.
• Introduced a new training strategy that includes negative frames not containing the query object, defined by IoU < 0.5 with ground truth or frame outside the response track.
• Applied hard-negative mining to select top-K negative proposals with highest loss, maintaining a 1:64 positive-to-negative ratio.
• Optimized the visual backbone by fine-tuning on Ego4D data to reduce domain shift from COCO pre-training.
• Used a Siamese ROI head with a two-branch feature extractor to compare query crops with video frames for detection.
• Applied tracker-based response track generation from the peak detection frame in both forward and backward temporal directions.

Experimental results

Research questions

• RQ1 How can training and evaluation efficiency be significantly improved in egocentric visual query 2D localization?
• RQ2 Why do false positives occur on negative frames, and how can they be mitigated during detection?
• RQ3 To what extent does including negative frames in training improve detection robustness and final retrieval performance?
• RQ4 How does backbone fine-tuning on egocentric data reduce the domain gap from COCO pre-training?
• RQ5 What is the impact of false positive errors on the final stAP metric, and how can they be quantitatively analyzed?

Key findings

• The proposed method reduced training time from 3600 hours to under 24 hours, achieving a 1500% speedup.
• Evaluation time was reduced from 400 hours to 12 hours, a 3200% improvement.
• Training with negative frames improved frame-level AP from 25.95 to 31.98 when evaluated on both positive and negative frames.
• The method achieved 0.17% spatial-temporal AP on the test set, a 31% improvement over the baseline’s 0.13%.
• The model surpassed the state-of-the-art by 0.02 in stAP25, 3.8% in success rate, and 4.1% in recovery ratio on the validation set.
• Error analysis revealed three main false positive types: similar appearance, ambiguous queries, and blurry frames—each alleviated by negative frame training.