[论文解读] Congested Crowd Instance Localization with Dilated Convolutional Swin Transformer

Junyu Gao, Maoguo Gong+1 more|arXiv (Cornell University)|Aug 2, 2021

Video Surveillance and Tracking Methods被引用 4

一句话总结

本文提出了一种空洞卷积Swin Transformer（DCST），用于在高度密集的群体场景中实现精确的实例定位，通过将空洞卷积整合到Swin Transformer主干网络中，增强大感受野的上下文特征学习能力。该方法在基准数据集上实现了SOTA性能，F1值达到77.5%，MAE为84.2，显著提升了遮挡和模糊区域的定位性能。

ABSTRACT

Crowd localization is a new computer vision task, evolved from crowd counting. Different from the latter, it provides more precise location information for each instance, not just counting numbers for the whole crowd scene, which brings greater challenges, especially in extremely congested crowd scenes. In this paper, we focus on how to achieve precise instance localization in high-density crowd scenes, and to alleviate the problem that the feature extraction ability of the traditional model is reduced due to the target occlusion, the image blur, etc. To this end, we propose a Dilated Convolutional Swin Transformer (DCST) for congested crowd scenes. Specifically, a window-based vision transformer is introduced into the crowd localization task, which effectively improves the capacity of representation learning. Then, the well-designed dilated convolutional module is inserted into some different stages of the transformer to enhance the large-range contextual information. Extensive experiments evidence the effectiveness of the proposed methods and achieve state-of-the-art performance on five popular datasets. Especially, the proposed model achieves F1-measure of 77.5\% and MAE of 84.2 in terms of localization and counting performance, respectively.

研究动机与目标

• 解决在小物体、相互遮挡和图像模糊导致性能下降的极端密集群体场景中，实现精确实例级定位的挑战。
• 通过增强上下文表征能力，克服传统模型在严重遮挡和模糊条件下提取鲁棒特征的局限性。
• 通过战略性地整合空洞卷积，提升视觉Transformer在密集群体场景中的特征提取能力。
• 在多个基准数据集上实现群体实例定位与计数任务的SOTA性能。

提出的方法

• 采用基于窗口的Swin Transformer作为主干编码器，以捕捉群体场景中的分层全局-局部表征。
• 在Swin Transformer的Stage 3和Stage 4中插入一种新型空洞卷积块（DCB），其空洞率分别为2和3，以扩展感受野。
• 在每个阶段后将Transformer的特征图重新组织为空间格式，以实现空洞卷积的有效应用。
• 使用特征金字塔网络（FPN）解码器生成与输入尺寸匹配的全分辨率分割图，用于实例级预测。
• 设计一种灵活的DCB模块，在保留局部细节的同时，编码多尺度下的长程上下文依赖关系。
• 通过消融实验优化DCB的放置位置和空洞率，以最大化定位与计数指标的性能。

Figure 1: Two problems in some typical crowd scenes: Tiny Objects & Mutual Occlusion and Blurred Regions, which cause that the current crowd localization models perform poorly in dense crowd scenes.

实验结果

研究问题

• RQ1 在Swin Transformer架构中何处插入空洞卷积块，可使密集人群场景下的定位性能达到最优？
• RQ2 DCB模块中何种空洞率配置能在局部细节保留与长程上下文编码之间取得最佳平衡？
• RQ3 阈值选择如何影响群体实例定位中定位F1值与计数MAE之间的权衡？
• RQ4 所提出的DCST在基准数据集上相较于标准Swin Transformer及其他SOTA模型，性能提升程度如何？

主要发现

• 所提出的DCST在五个基准数据集上实现了SOTA的F1值77.5%和MAE 84.2，展现出卓越的定位与计数性能。
• 将DCB插入Swin Transformer的Stage 3和Stage 4时性能最佳，在NWPU-Crowd验证集上F1值达到81.4%。
• 空洞率配置为{2, 3}的DCB实现了最优平衡，在NWPU数据集上取得81.4%的F1值、84.1%的精确率和79.0%的召回率。
• F1值最大化的阈值为0.44（F1值81.45%），而计数MAE最低的阈值为0.32（MAE 38.71），揭示了定位与计数目标之间的权衡。
• 该模型显著优于标准Swin Transformer及其他SOTA方法，尤其在处理微小、遮挡和模糊的群体区域方面表现突出。
• 消融实验表明，将DCB插入浅层阶段（1和2）会损害局部特征学习，而更深层插入则能更好地保留结构细节。

Figure 2: The flowchart of the proposed Dilated Convolutional Swin Transformer and FPN decoder (DCST+FPN). Notably, In Stage 3 and 4, the Dilated Convolutional Block (DCB) is applied to the top of the Swin Transformer Block (STB). By a Feature Pyramid Networks (FPN) decoder, the features from differ