[Paper Review] Learning to Count Objects in Natural Images for Visual Question Answering

Yan Zhang, Jonathon Hare+1 more|arXiv (Cornell University)|Feb 15, 2018

Multimodal Machine Learning ApplicationsComputer Science25 references155 citations

TL;DR

The paper introduces a differentiable counting component that deduplicates object proposals to enable robust counting from soft-attention in VQA, achieving state-of-the-art accuracy on the number questions of VQA v2 and improving counting-focused metrics without harming other categories.

ABSTRACT

Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Motivation & Objective

• Identify why counting is hard for VQA with standard soft attention.
• Develop a differentiable counting component that deduplicates object proposals.
• Integrate the counting component with existing VQA models using attention.
• Demonstrate improved counting performance on VQA v2 and a toy counting task.
• Show robustness to overlapping proposals without sacrificing non-counting performance.

Proposed method

• Convert attention-weighted object proposals into a graph via the outer product A = a a^T to represent proposals as nodes and overlaps as edges.
• Remove intra-object duplicate edges by masking A with a distance matrix D based on IoU, using differentiable activations to handle partial overlaps (tilde A = f1(A) ⊙ f2(D)).
• Estimate the number of underlying objects by computing a similarity-based scaling s_i for each proposal, then form a count matrix C that scales edges accordingly (C = tildeA ⊙ s s^T + diag(s ⊙ f1(a ⊙ a))).
• Infer the count c from E by c = sqrt(|E|) where |E| = sum(C_ij); output a count vector o that encodes the count with interpolation between adjacent integers.
• Optionally compute a confidence factor from the attention and overlap statistics to scale the final output (õ = f8(p_a + p_D) · o).
• Integrate the counting component with a standard VQA model by feeding top-n attention weights and using the component’s output as an auxiliary feature.

Experimental results

Research questions

• RQ1 Can a differentiable counting mechanism operating on object proposals overcome the counting limitations of soft attention in VQA?
• RQ2 How can overlapping or duplicate object proposals be deduplicated in a differentiable manner to recover true object counts?
• RQ3 Does incorporating the counting component improve counting questions without degrading performance on other VQA categories?
• RQ4 How does the counting component perform on a toy counting task and on the VQA v2 number category in practice?

Key findings

• The counting component enables robust counting from attention maps and can be integrated with standard VQA models.
• On VQA v2, the counting-enabled model achieves higher accuracy on number questions than the baseline and can outperform some 8-model ensembles with a single model using the counting component.
• The approach improves counting-focused metrics, including a substantial 6.6% improvement on a difficult balanced pair metric for counting.
• Experiments on a toy counting task show the component outperforms a simple attention-sum baseline across varying overlap and noise conditions, especially with moderate overlaps.
• Qualitative analyses indicate the learned activation functions adapt to dataset parameters, supporting interpretability of the counting behavior.