[Paper Review] Integrated Object Detection and Tracking with Tracklet-Conditioned Detection

Zheng Zhang, Dazhi Cheng+3 more|arXiv (Cornell University)|Nov 27, 2018

Video Surveillance and Tracking Methods50 references49 citations

TL;DR

The paper tightly couples object detection and tracking by conditioning current-frame detection on prior tracklets, yielding state-of-the-art results and improved tracking stability on video benchmarks.

ABSTRACT

Accurate detection and tracking of objects is vital for effective video understanding. In previous work, the two tasks have been combined in a way that tracking is based heavily on detection, but the detection benefits marginally from the tracking. To increase synergy, we propose to more tightly integrate the tasks by conditioning the object detection in the current frame on tracklets computed in prior frames. With this approach, the object detection results not only have high detection responses, but also improved coherence with the existing tracklets. This greater coherence leads to estimated object trajectories that are smoother and more stable than the jittered paths obtained without tracklet-conditioned detection. Over extensive experiments, this approach is shown to achieve state-of-the-art performance in terms of both detection and tracking accuracy, as well as noticeable improvements in tracking stability.

Motivation & Objective

• Motivate the need for tighter integration of detection and tracking in video understanding.
• Develop a tracklet-conditioned detection framework that leverages prior tracklets to influence current-frame detections.
• Show that early integration improves both detection accuracy and tracking stability over traditional late-integration methods.

Proposed method

• Define a tracklet-conditioned detection formulation where P(c|b^t_i, {d^{t-1}_j}) combines per-frame detector scores with tracklet-based cues.
• Compute P_tr(c|d^{t-1}_j) from tracklet history and update tracklet embeddings E_s2(d^t) over time.
• Establish association weights w(b^t_i, d^{t-1}_j) using cosine similarity of embedding features to guide cross-frame labeling.
• Integrate the approach into a two-stage Faster R-CNN detector, applying tracklet conditioning in both the first-stage (RPN) and second-stage (Fast R-CNN).
• Train with joint detection and tracking losses that link frame t-1 detections to frame t ground-truths to optimize both detection and tracking.
• Use a bipartite matching-based association (Hungarian algorithm) with pseudo tracklets for new objects, and employ propagation and rescore steps as needed.

Experimental results

Research questions

• RQ1 Can tracklet information from previous frames improve per-frame object detection scores beyond appearance alone?
• RQ2 Does early (tracklet-conditioned) integration of detection and tracking outperform late integration approaches that fuse results after detection?
• RQ3 What are the effects of joint training on both detection accuracy and tracking stability across video sequences?
• RQ4 How do different components (first-stage vs second-stage conditioning) contribute to overall performance?

Key findings

• Achieves state-of-the-art metrics on ImageNet VID for both detection (mAP_det) and tracking (mAP_track).
• Integrated approach with tracklet-conditioned detection on both stages yields the best overall mAP: det 78.1%, track 67.9% on the validation set.
• Early integration improves tracking stability, producing smoother trajectories than late-integration baselines.
• The method yields notable gains over sequential detection and tracking baselines, demonstrating better coherence between detections and existing tracklets.
• On MOT 2015, the integrated detector is finetuned for video tracking, achieving competitive results with improved stability (reported in the paper’s experiments).