[论文解读] Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking

Jiawei He, Zehao Huang+2 more|arXiv (Cornell University)|Mar 30, 2021

Video Surveillance and Tracking Methods参考文献 72被引用 12

一句话总结

该论文提出GMTracker，一种可学习的图匹配框架，通过整合帧内关系与深度特征学习实现多目标跟踪。通过将轨迹片段与检测结果建模为通用图，并利用隐函数定理将图匹配问题松弛为可微分的连续二次规划，该方法在MOT16和MOT17上达到最先进性能，IDF1分别为66.9和68.7，显著提升了遮挡情况下的鲁棒性。

ABSTRACT

Data association across frames is at the core of Multiple Object Tracking (MOT) task. This problem is usually solved by a traditional graph-based optimization or directly learned via deep learning. Despite their popularity, we find some points worth studying in current paradigm: 1) Existing methods mostly ignore the context information among tracklets and intra-frame detections, which makes the tracker hard to survive in challenging cases like severe occlusion. 2) The end-to-end association methods solely rely on the data fitting power of deep neural networks, while they hardly utilize the advantage of optimization-based assignment methods. 3) The graph-based optimization methods mostly utilize a separate neural network to extract features, which brings the inconsistency between training and inference. Therefore, in this paper we propose a novel learnable graph matching method to address these issues. Briefly speaking, we model the relationships between tracklets and the intra-frame detections as a general undirected graph. Then the association problem turns into a general graph matching between tracklet graph and detection graph. Furthermore, to make the optimization end-to-end differentiable, we relax the original graph matching into continuous quadratic programming and then incorporate the training of it into a deep graph network with the help of the implicit function theorem. Lastly, our method GMTracker, achieves state-of-the-art performance on several standard MOT datasets. Our code will be available at https://github.com/jiaweihe1996/GMTracker .

研究动机与目标

• 通过整合轨迹片段与检测结果之间的帧内关系，解决现有多目标跟踪方法在严重遮挡下性能受限的问题。
• 通过统一特征学习与数据关联，克服端到端深度学习跟踪器在训练与推理之间的一致性问题。
• 通过连续松弛与隐函数定理使图匹配可微分，弥合基于优化的分配方法与深度学习之间的差距。
• 通过将帧间与帧内关系统一建模为通用图结构，提升数据关联的鲁棒性与准确性。
• 通过可微分QP层协同训练特征提取与分配，实现在标准多目标跟踪基准上的最先进性能。

提出的方法

• 将数据关联建模为轨迹片段图与检测图之间的通用图匹配问题，其中顶点表示轨迹片段与检测结果，边编码相似度。
• 通过二阶边-边相似度引入帧内关系，提升遮挡情况下的鲁棒性。
• 将离散图匹配问题松弛为连续二次规划（QP）形式，以支持端到端反向传播。
• 基于KKT条件与隐函数定理设计可微分QP层，用于在反向传播中计算输入特征的梯度。
• 将可微分QP层集成到深度图网络中，以端到端方式联合优化特征学习与数据关联。
• 使用标准反向传播端到端训练整个网络，使模型能够同时学习上下文感知特征与最优匹配。

Figure 1: An illustration of intra-graph relationship used in our graph matching formulation. We utilize the second-order edge-to-edge similarity to model the group activity, which is more robust under heavy occlusion. Note that not all vertices in tracklet graph can be matched to the detection grap

实验结果

研究问题

• RQ1 在严重遮挡条件下，建模轨迹片段与检测结果之间的帧内关系是否能提升多目标跟踪的鲁棒性？
• RQ2 如何使离散图匹配可微分，以支持与深度特征学习联合进行端到端训练？
• RQ3 将基于优化的分配方法与深度学习相结合，是否能相比纯端到端学习方法提升跟踪精度？
• RQ4 引入长期上下文与群体活动模式在多大程度上能增强多目标跟踪中的数据关联？
• RQ5 统一的、可微分的特征学习与分配框架是否在标准基准上优于解耦方法？

主要发现

• GMTracker在MOT17测试集上取得新的SOTA IDF1得分66.9，显著超越先前在线方法如Tracktor++v2（55.1）与GNNMatch（56.1）。
• 在MOT16上，GMTracker取得IDF1 68.7，显著优于此前SOTA方法LifT（65.6）与MPNTrack（61.7）。
• 该方法在MOT17上将HOTA提升至52.0，在MOT16上提升至55.2，表明整体跟踪精度与鲁棒性更优。
• 消融实验证实，通过二阶边相似度引入帧内关系，使MOT17的IDF1提升2.1%。
• 可微分QP层支持端到端训练，缩小了训练与推理之间的领域差距，提升了泛化能力。
• 模型以极少的额外参数实现优异性能，展现出在大规模基准上的高效性与可扩展性。

Figure 2: An example of the derivation from edge affinity matrix $\mathbf{M_{e}}$ to quadratic affinity matrix $\mathbf{M}$ .