[Paper Review] End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions

Zejin Lu, Adrien Doerig+5 more|arXiv (Cornell University)|Aug 18, 2023

Visual perception and processing mechanisms4 citations

TL;DR

This paper introduces All-Topographic Neural Networks (All-TNNs), a novel deep learning architecture that models the topographic organization of the primate visual cortex by replacing standard convolutions with spatially structured, topographically organized layers. Trained end-to-end on visual inputs, All-TNNs self-organize smooth orientation maps and cortical magnification in early layers and category-selective areas in later layers, while significantly outperforming convolutional networks in predicting human spatial biases in object recognition.

ABSTRACT

Computational models are an essential tool for understanding the origin and functions of the topographic organisation of the primate visual system. Yet, vision is most commonly modelled by convolutional neural networks that ignore topography by learning identical features across space. Here, we overcome this limitation by developing All-Topographic Neural Networks (All-TNNs). Trained on visual input, several features of primate topography emerge in All-TNNs: smooth orientation maps and cortical magnification in their first layer, and category-selective areas in their final layer. In addition, we introduce a novel dataset of human spatial biases in object recognition, which enables us to directly link models to behaviour. We demonstrate that All-TNNs significantly better align with human behaviour than previous state-of-the-art convolutional models due to their topographic nature. All-TNNs thereby mark an important step forward in understanding the spatial organisation of the visual brain and how it mediates visual behaviour.

Motivation & Objective

• To develop a deep learning model that captures the spatial topography of the primate visual cortex, which standard convolutional networks fail to represent.
• To bridge the gap between computational models of cortical map formation and human visual behavior by incorporating spatial organization into neural network architectures.
• To create a new dataset of human spatial biases in object recognition to directly evaluate model-behavior alignment.
• To demonstrate that topographic organization in neural networks leads to improved prediction of human visual behavior compared to standard convolutional models.

Proposed method

• Design All-Topographic Neural Networks (All-TNNs) that replace standard convolutional layers with topographically structured, spatially invariant feature maps to reflect cortical map principles.
• Implement end-to-end training of All-TNNs on natural image datasets to allow self-organization of topographic features without supervision.
• Integrate a novel human behavioral dataset measuring spatial biases in object recognition to evaluate model performance against human data.
• Train All-TNNs to learn smooth orientation maps and cortical magnification in the first layer, and category-selective responses in deeper layers.
• Use spatially structured weight initialization and connectivity patterns that reflect known principles of retinotopic and topographic mapping in the visual cortex.
• Compare All-TNNs directly with state-of-the-art convolutional neural networks (CNNs) on both topographic feature emergence and human behavioral prediction.

Experimental results

Research questions

• RQ1 Can a deep neural network architecture with topographic organization self-organize key features of primate visual topography, such as orientation maps and cortical magnification?
• RQ2 Does replacing convolutions with topographic layers improve the alignment of deep learning models with human visual behavior?
• RQ3 To what extent do All-TNNs predict human spatial biases in object recognition compared to standard CNNs?
• RQ4 Can topographic organization in neural networks lead to the emergence of category-selective areas similar to those in the human visual cortex?
• RQ5 How does the inclusion of a human behavioral dataset enhance the evaluation of cortical map models in vision?

Key findings

• All-TNNs successfully self-organize smooth orientation maps and cortical magnification in their first layer, mirroring key features of the primate visual cortex.
• All-TNNs develop category-selective responses in deeper layers, suggesting the emergence of functional visual areas analogous to human extrastriate cortex.
• All-TNNs show significantly better alignment with human spatial biases in object recognition than state-of-the-art convolutional neural networks.
• The topographic architecture enables the model to predict human behavioral data more accurately, demonstrating the functional relevance of spatial organization.
• The novel human behavioral dataset reveals systematic spatial biases in object recognition that are better captured by topographic models.
• The results indicate that topographic organization is not only biologically plausible but also functionally advantageous for modeling human visual perception.