[Paper Review] Semi-TCL: Semi-Supervised Track Contrastive Representation Learning

Wei Li, Yuanjun Xiong+4 more|arXiv (Cornell University)|Jul 6, 2021

Face recognition and analysisComputer Science40 references33 citations

TL;DR

Semi-TCL learns track-level contrastive embeddings by matching detected instances to track representations, enabling semi-supervised learning from labeled and unlabeled videos for improved multi-object tracking.

ABSTRACT

Online tracking of multiple objects in videos requires strong capacity of modeling and matching object appearances. Previous methods for learning appearance embedding mostly rely on instance-level matching without considering the temporal continuity provided by videos. We design a new instance-to-track matching objective to learn appearance embedding that compares a candidate detection to the embedding of the tracks persisted in the tracker. It enables us to learn not only from videos labeled with complete tracks, but also unlabeled or partially labeled videos. We implement this learning objective in a unified form following the spirit of constrastive loss. Experiments on multiple object tracking datasets demonstrate that our method can effectively learning discriminative appearance embeddings in a semi-supervised fashion and outperform state of the art methods on representative benchmarks.

Motivation & Objective

• Motivate learning robust appearance embeddings for online multi-object tracking (MOT).
• Leverage temporal continuity in videos to improve instance representation via instance-to-track matching.
• Enable semi-supervised learning by utilizing unlabeled videos with pseudo-tracks.
• Integrate TCL into an end-to-end MOT framework built on CenterNet.
• Show that Semi-TCL achieves state-of-the-art results on MOT benchmarks.

Proposed method

• Define an instance-to-track distance using an aggregation G over tracklet embeddings.
• Propose Tracklet Contrastive Learning (TCL) with a contrastive loss over instances and sub-tracks: L_TCL = sum_i (-1/|S(i)|) sum_{\tilde{T}_j in S(i)} exp(f(I_i) · g̃_j / τ) / sum_{\tilde{T}_l in \tilde{T}} exp(f(I_i) · g̃_l / τ).
• Use a simple averaging aggregation G to compute g_j from tracklet features.
• Train with both labeled and pseudo-labeled unlabeled videos by sampling sub-tracks from tracks and using primitive trackers to generate pseudo-labels.
• Employ a joint detection+ID loss L_joint = L_det + L_id, where L_id is trained with TCL.

Experimental results

Research questions

• RQ1 Can an instance-to-track contrastive objective improve learning of appearance embeddings for MOT compared to instance-to-instance objectives?
• RQ2 Does TCL enable effective semi-supervised learning leveraging unlabeled videos for MOT?
• RQ3 How does batch size and use of unlabeled video data affect MOT performance?
• RQ4 What is the impact of pre-training strategy (CE vs. contrastive) on final MOT performance?
• RQ5 How does Semi-TCL perform on standard MOT benchmarks (IDF1, MOTA, IDS) relative to state-of-the-art?

Key findings

• Semi-TCL achieves state-of-the-art IDF1 across MOT15, MOT16, MOT17, and MOT20 benchmarks.
• On MOT16 and MOT17, Semi-TCL improves IDF1 by about 1% over prior SOTA.
• On MOT20, Semi-TCL raises IDF1 from 67.5% to 70.1%.
• Using unlabeled videos with tracklet density-based mining boosts IDF1 further (e.g., AVA-MEVA combinations).
• TCL outperforms both cross-entropy and instance-to-instance contrastive baselines in IDF1.
• Larger batch sizes (32, 96, 144) generally improve IDF1 and MOTA, indicating more contrastive pairs aid learning.
• Semi-TCL benefits from semi-supervised data; contrastive pre-training (SCL) yields larger IDF1 gains than CE pre-training.
• Unlabeled data from MEVA/AVA domains similar to MOT scenes yields larger gains than disparate domains.