[Paper Review] CLIP-EBC: CLIP Can Count Accurately through Enhanced Blockwise Classification

Yiming Ma, Víctor Sánchez+1 more|arXiv (Cornell University)|Mar 14, 2024

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper proposes CLIP-EBC, the first fully CLIP-based crowd-counting model that generates density maps by reformulating counting as enhanced blockwise classification using integer-valued bins. By replacing continuous interval bins with discrete integer bins and leveraging CLIP's vision-text alignment, CLIP-EBC achieves state-of-the-art performance, reducing mean absolute error to 55.0 and 6.3 on ShanghaiTech Part A and Part B, respectively, with up to 76.9% improvement over prior methods.

ABSTRACT

We propose CLIP-EBC, the first fully CLIP-based model for accurate crowd density estimation. While the CLIP model has demonstrated remarkable success in addressing recognition tasks such as zero-shot image classification, its potential for counting has been largely unexplored due to the inherent challenges in transforming a regression problem, such as counting, into a recognition task. In this work, we investigate and enhance CLIP's ability to count, focusing specifically on the task of estimating crowd sizes from images. Existing classification-based crowd-counting frameworks have significant limitations, including the quantization of count values into bordering real-valued bins and the sole focus on classification errors. These practices result in label ambiguity near the shared borders and inaccurate prediction of count values. Hence, directly applying CLIP within these frameworks may yield suboptimal performance. To address these challenges, we first propose the Enhanced Blockwise Classification (EBC) framework. Unlike previous methods, EBC utilizes integer-valued bins, effectively reducing ambiguity near bin boundaries. Additionally, it incorporates a regression loss based on density maps to improve the prediction of count values. Within our backbone-agnostic EBC framework, we then introduce CLIP-EBC to fully leverage CLIP's recognition capabilities for this task. Extensive experiments demonstrate the effectiveness of EBC and the competitive performance of CLIP-EBC. Specifically, our EBC framework can improve existing classification-based methods by up to 44.5% on the UCF-QNRF dataset, and CLIP-EBC achieves state-of-the-art performance on the NWPU-Crowd test set, with an MAE of 58.2 and an RMSE of 268.5, representing improvements of 8.6% and 13.3% over the previous best method, STEERER. The code and weights are available at https://github.com/Yiming-M/CLIP-EBC.

Motivation & Objective

• To address the underperformance of existing classification-based crowd-counting methods that use continuous interval bins and suffer from poor decision boundary learning.
• To bridge the gap between CLIP, a recognition model, and counting, a regression task, by reformulating counting as blockwise classification.
• To improve model robustness and generalization by using integer-valued bins instead of continuous intervals, reducing misclassification near bin borders.
• To enable CLIP-based density map generation for crowd counting, leveraging CLIP’s pre-trained vision and text encoders without fine-tuning the text encoder.
• To demonstrate that the EBC framework enhances not only CLIP-based models but also existing regression-based models when integrated.

Proposed method

• Propose the Enhanced Blockwise Classification (EBC) framework, which uses discrete integer-valued bins instead of continuous interval bins for count classification.
• Modify the CLIP image encoder by removing the global average pooling layer and replacing the linear projection with a 1×1 convolution to preserve spatial resolution for blockwise prediction.
• Use the frozen CLIP text encoder to generate text embeddings for each integer bin (e.g., '1 person', '2 people', ...), which are then compared via cosine similarity to image features.
• Generate a probability map by applying softmax over the cosine similarities between local image features and text embeddings for each bin.
• Construct the predicted density map by weighting each bin’s count value by its corresponding probability score and averaging across spatial locations.
• Optimize the model using a hybrid loss function DACE that combines classification loss and count regression loss, with a learnable weight λ to balance both terms.

Experimental results

Research questions

• RQ1 Can CLIP be effectively adapted for accurate crowd counting, despite being pre-trained for recognition tasks?
• RQ2 Does using integer-valued bins instead of continuous interval bins improve classification performance and decision boundary learning in blockwise crowd counting?
• RQ3 Can the EBC framework significantly enhance the performance of existing regression-based crowd-counting models when applied as a plug-in module?
• RQ4 Does CLIP-EBC, as a fully CLIP-based model, generate high-quality density maps and achieve state-of-the-art performance on benchmark datasets?
• RQ5 How does the choice of loss weighting (λ) between classification and regression components affect the final model performance?

Key findings

• CLIP-EBC achieves a mean absolute error (MAE) of 55.0 on ShanghaiTech Part A and 6.3 on Part B, setting a new state-of-the-art performance.
• The EBC framework improves existing models by up to 76.9% in performance gain, with consistent improvements across different backbones (VGG16, ResNet101, MobileNetV2, DenseNet201).
• Replacing continuous interval bins with integer-valued bins reduces MAE from 140.6 (baseline blockwise) to 88.3, a 37.1% improvement, demonstrating the effectiveness of discrete binning.
• The optimal loss weighting λ = 1.0 achieves the best performance (MAE = 77.9), while higher values like λ = 2.0 reduce generalization ability.
• The EBC framework is model-agnostic and boosts performance across multiple backbones, indicating its broad applicability beyond CLIP.
• Visualization results show that CLIP-EBC generalizes well across diverse crowd densities, from sparse to highly congested scenes, producing accurate and smooth density maps.