[论文解读] Negative Frames Matter in Egocentric Visual Query 2D Localization

Mengmeng Xu, Cheng-Yang Fu+4 more|arXiv (Cornell University)|Aug 3, 2022

Advanced Image and Video Retrieval Techniques被引用 4

一句话总结

该论文通过优化训练效率并引入负样本帧训练，提出了一种更快、更准确的自指视觉查询2D定位方法。通过将训练时间从15天缩短至24小时以内，并在检测器训练中整合负样本帧，该方法在Ego4D公开排行榜上排名第一，实现了0.17%的时空AP，较基线模型高出31%。

ABSTRACT

The recently released Ego4D dataset and benchmark significantly scales and diversifies the first-person visual perception data. In Ego4D, the Visual Queries 2D Localization task aims to retrieve objects appeared in the past from the recording in the first-person view. This task requires a system to spatially and temporally localize the most recent appearance of a given object query, where query is registered by a single tight visual crop of the object in a different scene. Our study is based on the three-stage baseline introduced in the Episodic Memory benchmark. The baseline solves the problem by detection and tracking: detect the similar objects in all the frames, then run a tracker from the most confident detection result. In the VQ2D challenge, we identified two limitations of the current baseline. (1) The training configuration has redundant computation. Although the training set has millions of instances, most of them are repetitive and the number of unique object is only around 14.6k. The repeated gradient computation of the same object lead to an inefficient training; (2) The false positive rate is high on background frames. This is due to the distribution gap between training and evaluation. During training, the model is only able to see the clean, stable, and labeled frames, but the egocentric videos also have noisy, blurry, or unlabeled background frames. To this end, we developed a more efficient and effective solution. Concretely, we bring the training loop from ~15 days to less than 24 hours, and we achieve 0.17% spatial-temporal AP, which is 31% higher than the baseline. Our solution got the first ranking on the public leaderboard. Our code is publicly available at https://github.com/facebookresearch/vq2d_cvpr.

研究动机与目标

• 解决自指视觉查询2D定位中训练与评估效率低下的问题。
• 降低在背景和负样本帧上的高误报率，以提升最终检索性能。
• 弥合预训练模型（如COCO）与自指视频数据（Ego4D）之间的分布差距。
• 通过显式训练负样本示例，提升模型在模糊、失焦或模糊帧上的鲁棒性。
• 通过代码重构与并行化加速研究周期，实现更快的实验迭代。

提出的方法

• 将原始15天的训练循环替换为重构后的并行化训练与评估流水线，将训练时间缩短至24小时以内。
• 提出一种新训练策略，包含不包含查询对象的负样本帧，其定义为与真实框的IoU < 0.5，或位于响应轨迹之外的帧。
• 应用难负样本挖掘，选择损失最高的K个负样本提议，保持正负样本比例为1:64。
• 通过在Ego4D数据上微调视觉主干网络，减少来自COCO预训练的领域偏移。
• 采用孪生ROI头结构，结合双分支特征提取器，比较查询裁剪区域与视频帧之间的特征以实现检测。
• 基于峰值检测帧，沿前后两个时间方向生成基于追踪器的响应轨迹。

实验结果

研究问题

• RQ1 如何在自指视觉查询2D定位中显著提升训练与评估效率？
• RQ2 为何在负样本帧上会出现误报？如何在检测过程中缓解此类问题？
• RQ3 在训练中引入负样本帧在多大程度上能提升检测鲁棒性与最终检索性能？
• RQ4 在自指数据上微调主干网络在多大程度上能减少来自COCO预训练的领域差距？
• RQ5 误报错误对最终stAP指标有何影响？如何实现其定量分析？

主要发现

• 所提方法将训练时间从3600小时缩短至24小时以内，实现1500%的加速。
• 评估时间从400小时缩短至12小时，提升达3200%。
• 在包含正负样本帧的评估中，帧级AP从25.95提升至31.98。
• 该方法在测试集上实现了0.17%的时空AP，较基线模型的0.13%高出31%。
• 在验证集上，该模型在stAP25上超越当前最先进方法0.02，在成功率上提升3.8%，在恢复率上提升4.1%。
• 错误分析揭示了三种主要的误报类型：外观相似、查询模糊以及模糊帧，均通过负样本帧训练得到缓解。