[论文解读] Joint CNN and Transformer Network via weakly supervised Learning for efficient crowd counting

Fusen Wang, Kai Liu+4 more|arXiv (Cornell University)|Mar 12, 2022

Video Surveillance and Tracking Methods被引用 10

一句话总结

本文提出JCTNet，一种用于仅使用计数级标注的弱监督人群计数的联合CNN与Transformer网络，避免了昂贵的位置级标注。通过结合CNN进行局部语义特征提取与改进的Swin Transformer进行全局上下文建模，JCTNet在仅使用67–73%参数量的纯Transformer模型下实现了最先进性能，同时在弱监督下仍能有效聚焦于人群区域。

ABSTRACT

Currently, for crowd counting, the fully supervised methods via density map estimation are the mainstream research directions. However, such methods need location-level annotation of persons in an image, which is time-consuming and laborious. Therefore, the weakly supervised method just relying upon the count-level annotation is urgently needed. Since CNN is not suitable for modeling the global context and the interactions between image patches, crowd counting with weakly supervised learning via CNN generally can not show good performance. The weakly supervised model via Transformer was sequentially proposed to model the global context and learn contrast features. However, the transformer directly partitions the crowd images into a series of tokens, which may not be a good choice due to each pedestrian being an independent individual, and the parameter number of the network is very large. Hence, we propose a Joint CNN and Transformer Network (JCTNet) via weakly supervised learning for crowd counting in this paper. JCTNet consists of three parts: CNN feature extraction module (CFM), Transformer feature extraction module (TFM), and counting regression module (CRM). In particular, the CFM extracts crowd semantic information features, then sends their patch partitions to TRM for modeling global context, and CRM is used to predict the number of people. Extensive experiments and visualizations demonstrate that JCTNet can effectively focus on the crowd regions and obtain superior weakly supervised counting performance on five mainstream datasets. The number of parameters of the model can be reduced by about 67%~73% compared with the pure Transformer works. We also tried to explain the phenomenon that a model constrained only by count-level annotations can still focus on the crowd regions. We believe our work can promote further research in this field.

研究动机与目标

• 解决完全监督人群计数方法中需为每个人标注位置级边界框所带来的高标注成本问题。
• 克服CNN在人群场景中建模长距离依赖与全局上下文的局限性。
• 缓解纯Transformer模型的缺点，包括参数过多以及不合理的图像块划分导致个体被分割到不同token中。
• 开发一种弱监督人群计数框架，仅使用图像级计数标注即可实现高精度。
• 解释仅基于计数级监督训练的模型如何通过CNN与Transformer的互补学习仍能关注人群区域。

提出的方法

• JCTNet集成一个基于VGG16-BN前10层的CNN特征提取模块（CFM），用于从输入图像中提取局部语义特征。
• CFM生成的特征图被划分为图像块，并输入到改进的Swin Transformer块（MSTB）中，以实现全局上下文建模与长距离依赖学习。
• Transformer模块采用局部自注意力与跨窗口交互机制，以保持空间关系并提升特征表示能力。
• 计数回归模块（CRM）处理Transformer输出的最终token特征，以预测图像中总人数。
• 模型通过Smooth L1损失函数进行端到端训练，以最小化在计数级监督下的预测误差。
• 该架构通过利用CNN的归纳偏置来减少参数量，同时保留Transformer的全局建模能力。

实验结果

研究问题

• RQ1 仅基于图像级计数监督训练的弱监督人群计数模型，是否仍能关注实际人群区域？
• RQ2 如何联合优化CNN与Transformer以在降低模型复杂度的同时提升性能？
• RQ3 与仅使用CNN相比，Transformer中的自注意力机制是否能更好地区分人群与背景区域？
• RQ4 在所提出的架构中，嵌入维度与Swin Transformer块数量之间最优的权衡是什么？
• RQ5 为何该模型在缺乏位置级监督的情况下仍具有良好的泛化能力？

主要发现

• JCTNet在ShanghaiTech Part A数据集上实现了62.8的平均绝对误差（MAE），优于以往的弱监督方法。
• 与纯Transformer基线相比，该模型将参数量减少了67–73%，最佳配置下仅需2880万参数。
• 消融实验证实，CFM、TFM与CRM各组件逐步贡献于性能提升。
• 在ShanghaiTech Part A数据集上，使用Smooth L1损失相比L1损失将MAE从63.7降低至62.8，MSE从98.1降低至95.6。
• 可视化结果表明，即使在无位置级监督的情况下，模型仍能聚焦于人群区域（红色矩形），表明其注意力学习有效。
• CNN的归纳偏置与Transformer的全局建模能力相结合，使模型能够学习到前景与背景之间的对比特征，从而解释了其在弱监督下的鲁棒性。