[Paper Review] Cognitive Deficit of Deep Learning in Numerosity

Xiaolin Wu, Xi Zhang+1 more|arXiv (Cornell University)|Feb 9, 2018

Cognitive and developmental aspects of mathematical skills3 citations

TL;DR

This paper investigates whether deep learning (DL) can achieve human-like subitizing—the innate ability to instantly perceive small numbers of objects—using end-to-end training. Despite strong supervision, standard deep convolutional networks fail to generalize across visual variations in size, shape, and color. In contrast, a recurrent CNN with explicitly encoded morphological operations (erosion, connected components, counting) successfully subitizes by leveraging human-derived cognitive priors, demonstrating that DL’s cognitive deficit stems from optimization limitations, not architecture.

ABSTRACT

Subitizing, or the sense of small natural numbers, is an innate cognitive function of humans and primates; it responds to visual stimuli prior to the development of any symbolic skills, language or arithmetic. Given successes of deep learning (DL) in tasks of visual intelligence and given the primitivity of number sense, a tantalizing question is whether DL can comprehend numbers and perform subitizing. But somewhat disappointingly, extensive experiments of the type of cognitive psychology demonstrate that the examples-driven black box DL cannot see through superficial variations in visual representations and distill the abstract notion of natural number, a task that children perform with high accuracy and confidence. The failure is apparently due to the learning method not the CNN computational machinery itself. A recurrent neural network capable of subitizing does exist, which we construct by encoding a mechanism of mathematical morphology into the CNN convolutional kernels. Also, we investigate, using subitizing as a test bed, the ways to aid the black box DL by cognitive priors derived from human insight. Our findings are mixed and interesting, pointing to both cognitive deficit of pure DL, and some measured successes of boosting DL by predetermined cognitive implements. This case study of DL in cognitive computing is meaningful for visual numerosity represents a minimum level of human intelligence.

Motivation & Objective

• To test whether end-to-end deep learning can learn the abstract concept of natural numbers through visual numerosity perception.
• To investigate the limitations of data-driven deep learning in achieving cognitive abstraction beyond statistical inference.
• To explore whether incorporating human-derived cognitive priors—such as morphological operations—can enable deep learning to perform subitizing.
• To analyze why standard backpropagation fails to train a deterministic subitizing network, despite a correct architecture.
• To develop a modular, neurocognitively inspired deep learning system that generalizes across visual variations by decomposing subitizing into primitive cognitive functions.

Proposed method

• Designed a recurrent convolutional neural network (RCNN) with $3\times3$ convolutional kernels explicitly encoding morphological operations (erosion, connectivity preservation) for subitizing.
• Decomposed subitizing into three subnetworks: erosion (morphological atom), connected components detection, and counting—each trained separately.
• Used residual blocks with ReLU activations in the erosion subnetwork to enable stable learning of connectivity-preserving reduction.
• Constructed a modular DCNN by concatenating the learned subnetworks to form a complete subitizing system with generalization beyond i.i.d. data.
• Explored gradient descent training of the RCNN with both random and informed weight initialization, revealing non-convergence due to discontinuous loss near optimal solutions.
• Treated the RCNN as a biologically inspired architecture with a ventral visual pathway parallel, using recursive erosion as the core operation.

Experimental results

Research questions

• RQ1 Can end-to-end deep learning with standard backpropagation learn the abstract concept of natural numbers from visual stimuli exhibiting diverse shapes, sizes, and colors?
• RQ2 Why does deep learning fail to generalize in subitizing tasks despite strong supervision and high-quality training data?
• RQ3 Can a deep learning model achieve deterministic subitizing by incorporating human-derived cognitive primitives such as morphological operations?
• RQ4 Why does gradient descent fail to converge when training a prestructured RCNN with known optimal kernels, despite the architecture being correct?
• RQ5 To what extent can deep learning be augmented by preexisting cognitive constructs to achieve cognitive functions like subitizing?

Key findings

• Standard deep convolutional neural networks fail to subitize even with strong supervision, showing poor generalization across variations in object size, shape, color, and orientation.
• The failure is attributed to overreliance on statistical patterns in training data rather than scene semantics, indicating a fundamental cognitive deficit in pure data-driven learning.
• A recurrent CNN with morphologically encoded kernels successfully performs deterministic subitizing by abstracting number from visual variations, proving that the architecture is capable when guided by human insight.
• Training the RCNN with gradient descent fails to converge, even with informed initialization, due to a highly discontinuous loss function near the optimal solution, resembling integer programming.
• Decomposing subitizing into three subfunctions—erosion, connected components, and counting—enables separate training and modular composition, resulting in a generalizable subitizing system.
• The success of the modular approach demonstrates that deep learning can achieve cognitive abstraction only when augmented by preexisting cognitive priors, not through end-to-end learning alone.