[Paper Review] Learning to associate detections for real-time multiple object tracking

Michel Meneses, Leonardo Nogueira Matos+3 more|arXiv (Cornell University)|Jul 12, 2020

Video Surveillance and Tracking Methods33 references4 citations

TL;DR

This paper proposes SmartSORT, a real-time multiple object tracking method that uses a lightweight multilayer perceptron (MLP) neural network to learn a similarity function for associating detections across video frames. By combining deep appearance features with handcrafted motion features in a sliding window, the model predicts association costs efficiently, achieving state-of-the-art accuracy while running 58% faster than a similar baseline, making it highly effective for online and embedded applications.

ABSTRACT

With the recent advances in the object detection research field, tracking-by-detection has become the leading paradigm adopted by multi-object tracking algorithms. By extracting different features from detected objects, those algorithms can estimate the objects' similarities and association patterns along successive frames. However, since similarity functions applied by tracking algorithms are handcrafted, it is difficult to employ them in new contexts. In this study, it is investigated the use of artificial neural networks to learning a similarity function that can be used among detections. During training, the networks were introduced to correct and incorrect association patterns, sampled from a pedestrian tracking data set. For such, different motion and appearance features combinations have been explored. Finally, a trained network has been inserted into a multiple-object tracking framework, which has been assessed on the MOT Challenge benchmark. Throughout the experiments, the proposed tracker matched the results obtained by state-of-the-art methods, it has run 58\% faster than a recent and similar method, used as baseline.

Motivation & Objective

• To address the limitation of heuristic-based similarity functions in tracking-by-detection, which are context-dependent and hard to adapt to new scenarios.
• To develop a scalable and adaptable similarity function using machine learning that can generalize across different datasets and feature types.
• To enable real-time performance in online multi-object tracking by minimizing computational overhead while maintaining high accuracy.
• To explore the use of simple, compact neural networks instead of complex deep learning models for cost-effective tracking.
• To improve the efficiency of online tracking frameworks by replacing filtering-based motion modeling with a sliding window strategy combined with a learned regression model.

Proposed method

• A multilayer perceptron (MLP) neural network is trained to predict the association cost between detections using high-level appearance and motion features.
• Features include deep appearance embeddings from a pre-trained CNN and handcrafted motion features (e.g., position, velocity) from a Kalman filter.
• A sliding window mechanism captures temporal context by aggregating past detections, enabling sequence modeling without explicit recurrent layers.
• The MLP is trained on positive and negative association pairs sampled from a pedestrian tracking dataset, using backpropagation with supervised learning.
• The predicted costs are used to construct a bipartite graph, solved via linear programming to assign detections to tracks.
• The tracker operates online, processing each frame in real time with a single matrix operation for all pairwise costs.

Experimental results

Research questions

• RQ1 Can a lightweight neural network learn a robust similarity function for detection association in multi-object tracking that generalizes across datasets?
• RQ2 How does the performance of a learned similarity function compare to heuristic-based methods in terms of accuracy and speed?
• RQ3 Can a simple MLP-based regression model outperform more complex deep learning architectures in real-time tracking applications?
• RQ4 To what extent does the sliding window strategy with a non-recurrent model affect tracking stability and identity consistency?
• RQ5 How does the proposed method scale in terms of computational cost when using high-dimensional deep features?

Key findings

• SmartSORT achieved competitive tracking accuracy, with an IDP of 68.5% and FM of 15.7% on the MOT Challenge 2016 benchmark, comparable to state-of-the-art methods.
• The tracker ran 58% faster than a recent baseline method using similar deep features, demonstrating significant speed gains.
• Despite its simplicity, the MLP-based model achieved low regression error, indicating effective learning of association costs.
• SmartSORT was at least 59% faster than other trackers using deep visual features, including an LSTM-based method, while maintaining less than 3% lower accuracy.
• The method's main limitation was higher identity switches (ID) and fragmentation (FM), attributed to the sliding window's vulnerability to incorrect associations.
• The results confirm that a compact, learned regression model can achieve high cost-effectiveness in online tracking, especially suitable for embedded systems.