[Paper Review] Quasi-Dense Similarity Learning for Multiple Object Tracking

Jiangmiao Pang, Linlu Qiu+5 more|arXiv (Cornell University)|Jun 11, 2020

Video Surveillance and Tracking Methods52 references18 citations

TL;DR

This paper proposes Quasi-Dense Similarity Learning, a contrastive learning method that densely samples hundreds of region proposals across image pairs to improve instance similarity learning for multiple object tracking. By leveraging quasi-dense positive and negative samples, it enables a simple nearest-neighbor search at inference, achieving state-of-the-art performance: 68.7 MOTA on MOT17 at 20.3 FPS without external data, and nearly 10-point MOTA gains on BDD100K and Waymo with fewer ID switches.

ABSTRACT

Similarity learning has been recognized as a crucial step for object tracking. However, existing multiple object tracking methods only use sparse ground truth matching as the training objective, while ignoring the majority of the informative regions on the images. In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of region proposals on a pair of images for contrastive learning. We can directly combine this similarity learning with existing detection methods to build Quasi-Dense Tracking (QDTrack) without turning to displacement regression or motion priors. We also find that the resulting distinctive feature space admits a simple nearest neighbor search at the inference time. Despite its simplicity, QDTrack outperforms all existing methods on MOT, BDD100K, Waymo, and TAO tracking benchmarks. It achieves 68.7 MOTA at 20.3 FPS on MOT17 without using external training data. Compared to methods with similar detectors, it boosts almost 10 points of MOTA and significantly decreases the number of ID switches on BDD100K and Waymo datasets. Our code and trained models are available at http://vis.xyz/pub/qdtrack.

Motivation & Objective

• To address the limitation of sparse supervision in existing similarity learning for multi-object tracking, which underutilizes informative image regions beyond ground truth boxes.
• To improve instance-level similarity learning by densely sampling region proposals across image pairs to provide richer supervision and hard negatives.
• To enable a simple, end-to-end tracking pipeline using only similarity matching and nearest-neighbor search, avoiding complex regression or motion priors.
• To reduce ID switches and false positives by incorporating unmatched objects (backdrops) and enforcing bi-directional consistency in matching.
• To build a plug-and-play tracking framework compatible with existing detectors, achieving high accuracy with minimal architectural changes.

Proposed method

• The method performs quasi-dense matching between hundreds of region proposals on a pair of consecutive frames, using both ground truth boxes and surrounding candidate regions as samples.
• It extends contrastive learning to handle multiple positive samples per query, enabling each proposal to distinguish all other proposals on the reference image simultaneously.
• A bi-directional softmax is applied over matching scores between current detections, past tracks, and backdrops to enforce consistency and suppress false positives.
• The framework integrates with standard detectors like Faster R-CNN and uses a lightweight embedding head with residual networks for feature extraction.
• Inference uses a simple nearest-neighbor search in the learned embedding space, eliminating the need for displacement regression or motion modeling.
• The entire system is trained end-to-end, simplifying training and deployment while maintaining high efficiency.

Experimental results

Research questions

• RQ1 Can dense sampling of region proposals across image pairs improve instance similarity learning in multi-object tracking beyond sparse supervision?
• RQ2 Does quasi-dense contrastive learning lead to a more discriminative feature space that enables accurate tracking via simple nearest-neighbor matching?
• RQ3 Can the proposed method reduce ID switches and false positives without relying on motion priors or regression heads?
• RQ4 How does the inclusion of backdrops (unmatched objects) in the matching process affect tracking robustness and consistency?
• RQ5 To what extent can a plug-and-play similarity learning module improve tracking performance across diverse benchmarks like MOT17, BDD100K, Waymo, and TAO?

Key findings

• QDTrack achieves 68.7 MOTA on MOT17 at 20.3 FPS without using any external training data, setting a new SOTA for this benchmark.
• On BDD100K and Waymo, QDTrack boosts MOTA by nearly 10 points compared to methods with similar detectors, significantly reducing ID switches.
• The method outperforms all existing tracking methods on all evaluated benchmarks: MOT, BDD100K, Waymo, and TAO.
• The use of backdrops in matching reduces false positives, as shown by visualizations where false positives are predominantly matched to backdrops.
• The simplicity of the inference pipeline—relying solely on nearest-neighbor search in the embedding space—demonstrates that strong tracking performance can be achieved without complex association modules.
• The framework is compatible with standard detectors and enables end-to-end training, simplifying the overall tracking pipeline.