[Paper Review] Improving the Learning of Multi-column Convolutional Neural Network for Crowd Counting

Zhi-Qi Cheng, Jun-Xiu Li+4 more|arXiv (Cornell University)|Sep 17, 2019

Video Surveillance and Tracking Methods65 references4 citations

TL;DR

This paper proposes Multi-column Mutual Learning (McML), a novel training strategy that reduces parameter redundancy and improves scale invariance in multi-column convolutional neural networks for crowd counting. By minimizing mutual information between columns and using asynchronous mutual learning, McML ensures each column learns distinct scale-specific features, significantly boosting accuracy across four benchmarks with relative MAE improvements up to 25.0%.

ABSTRACT

Tremendous variation in the scale of people/head size is a critical problem for crowd counting. To improve the scale invariance of feature representation, recent works extensively employ Convolutional Neural Networks with multi-column structures to handle different scales and resolutions. However, due to the substantial redundant parameters in columns, existing multi-column networks invariably exhibit almost the same scale features in different columns, which severely affects counting accuracy and leads to overfitting. In this paper, we attack this problem by proposing a novel Multi-column Mutual Learning (McML) strategy. It has two main innovations: 1) A statistical network is incorporated into the multi-column framework to estimate the mutual information between columns, which can approximately indicate the scale correlation between features from different columns. By minimizing the mutual information, each column is guided to learn features with different image scales. 2) We devise a mutual learning scheme that can alternately optimize each column while keeping the other columns fixed on each mini-batch training data. With such asynchronous parameter update process, each column is inclined to learn different feature representation from others, which can efficiently reduce the parameter redundancy and improve generalization ability. More remarkably, McML can be applied to all existing multi-column networks and is end-to-end trainable. Extensive experiments on four challenging benchmarks show that McML can significantly improve the original multi-column networks and outperform the other state-of-the-art approaches.

Motivation & Objective

• Address the critical problem of scale variation in crowd counting, where people's sizes vary drastically due to perspective effects.
• Identify that existing multi-column networks suffer from redundant parameters and learn nearly identical scale features across columns.
• Overcome overfitting and poor generalization by guiding each column to learn distinct, complementary scale representations.
• Develop a plug-and-play, end-to-end trainable strategy applicable to any existing multi-column crowd counting network.
• Improve feature diversity and generalization without requiring architectural changes or additional supervision.

Proposed method

• Introduce a statistical network to estimate mutual information between feature maps of different columns, serving as a proxy for scale correlation.
• Minimize mutual information between columns during training to encourage learning of dissimilar, scale-specific features.
• Design an asynchronous mutual learning scheme where each column is optimized in turn while others are fixed, promoting feature diversity.
• Apply the McML strategy in a plug-and-play manner to existing multi-column networks (e.g., MCNN, CSRNet, ic-CNN), enabling end-to-end training.
• Use Maximal Information Coefficient (MIC) and Structural Similarity (SSIM) to quantitatively analyze feature redundancy and similarity across columns.
• Leverage density map visualization and quantitative metrics (MAE, MSE) to evaluate the effectiveness of McML on diverse crowd scenes.

Experimental results

Research questions

• RQ1 Why do existing multi-column networks fail to learn diverse scale features despite having different receptive fields?
• RQ2 Can mutual information minimization effectively reduce redundancy and improve feature diversity among columns?
• RQ3 Does asynchronous mutual learning enhance generalization and reduce overfitting in multi-column crowd counting networks?
• RQ4 Can McML be universally applied to various multi-column architectures without architectural modification?
• RQ5 How does McML perform across datasets with varying crowd density and scale distribution?

Key findings

• McML significantly reduces mutual information (MIC) and structural similarity (SSIM) between columns, confirming reduced redundancy and improved feature diversity.
• On ShanghaiTech Part A, McML improves MAE by 25.0% for MCNN, 23.6% for CSRNet, and 5.6% for ic-CNN, demonstrating strong performance on dense crowds.
• On UCF_CC_50, McML achieves 17.6% relative MAE improvement for MCNN and 7.5% for CSRNet, showing effectiveness on small and sparse datasets.
• On UCSD, McML reduces both MAE and MSE, indicating improved accuracy and robustness in sparse crowd scenarios.
• On WorldExpo’10, McML outperforms SOTA methods that use perspective maps, even without such supervision, proving its generalization capability.
• Visualization shows McML-generated density maps have more distinct salient regions across columns, confirming better scale-specific feature learning.