[Paper Review] Learning to Point and Count

Jie Shao, Dequan Wang+2 more|arXiv (Cornell University)|Dec 8, 2015

Explainable Artificial Intelligence (XAI)15 references3 citations

TL;DR

This paper introduces the point-and-count (P&C) problem as a new benchmark to challenge the 'what-and-where' deadlock in object detection, proposing two novel approaches: count-then-point (C2P) and point-then-count (P2C). It demonstrates that combining both methods improves robustness, reveals critical limitations in CNNs—such as data bias and scale sensitivity—and introduces the MNIST-LEGO dataset to enable controlled, scalable experimentation with infinite synthetic training data.

ABSTRACT

This paper proposes the problem of point-and-count as a test case to break the what-and-where deadlock. Different from the traditional detection problem, the goal is to discover key salient points as a way to localize and count the number of objects simultaneously. We propose two alternatives, one that counts first and then point, and another that works the other way around. Fundamentally, they pivot around whether we solve "what" or "where" first. We evaluate their performance on dataset that contains multiple instances of the same class, demonstrating the potentials and their synergies. The experiences derive a few important insights that explains why this is a much harder problem than classification, including strong data bias and the inability to deal with object scales robustly in state-of-art convolutional neural networks.

Motivation & Objective

• To address the 'what-and-where' deadlock in object detection by proposing a new task: point-and-count (P&C), where models must localize and count identical objects in a single image.
• To investigate whether solving 'what' (classification) or 'where' (localization) first leads to better performance in P&C tasks.
• To identify fundamental limitations in modern CNNs—such as strong data bias and poor scale generalization—through empirical analysis on real and synthetic data.
• To develop a new synthetic dataset, MNIST-LEGO, to enable controlled, scalable, and infinite training data for studying P&C problems.

Proposed method

• Proposes two distinct strategies: count-then-point (C2P), which first predicts object count via a learned classifier and then clusters activation heatmaps; and point-then-count (P2C), which uses top features of a pre-trained network as object signatures to filter heatmaps before clustering.
• Employs a pre-trained CNN to extract activation maps, then applies off-the-shelf clustering (e.g., mean-shift) on heatmap responses to localize salient points.
• Uses a dictionary of top-activated features from a pre-trained network as a signature to prune and refine heatmaps in the P2C approach.
• Introduces the MNIST-LEGO dataset by assembling MNIST digits into synthetic objects using simple construction rules, enabling control over object complexity, scale, and background noise.
• Applies a 4-layer CNN with NIN blocks and global average pooling for classification and feature extraction, achieving 85.4% top-1 accuracy on MNIST-LEGO.
• Evaluates performance using containment of predicted pointers within ground-truth bounding boxes, and analyzes failure modes such as small-object confusion and scale variance.

Experimental results

Research questions

• RQ1 Can solving 'where' before 'what' (P2C) or 'what' before 'where' (C2P) lead to better performance in point-and-count tasks?
• RQ2 What are the failure modes of state-of-the-art CNNs when applied to point-and-count, particularly regarding data bias and scale variance?
• RQ3 How do the two proposed approaches—C2P and P2C—interact, and can they be combined to improve robustness?
• RQ4 To what extent can synthetic datasets like MNIST-LEGO replicate real-world P&C challenges such as close-by object confusion and scale variation?
• RQ5 Can the insights from synthetic data transfer to real-world image understanding tasks?

Key findings

• The P2C approach, which requires no additional learning, can handle unbounded numbers of objects and performs well when classification is accurate, but is sensitive to clustering errors.
• The C2P approach, which learns to count first, outperforms P2C when one object class dominates, as it provides more informative clustering signals.
• Small and closely spaced objects lead to under-counting due to overlapping receptive fields and tangled activation maps, especially in deeper layers.
• Pre-trained CNNs exhibit strong data bias, favoring classes seen more frequently during training, which undermines fair counting performance.
• The MNIST-LEGO dataset successfully replicates real-world P&C failure modes such as big-O, parallel suppression, and close-by confusion, validating its utility for controlled experiments.
• Combining C2P and P2C strategies yields synergistic improvements, suggesting that integrated systems could outperform isolated approaches.