[Paper Review] Deep learning in a bilateral brain with hemispheric specialization

Chandramouli Rajagopalan, David Rawlinson+2 more|arXiv (Cornell University)|Sep 9, 2022

Hemispheric Asymmetry in Neuroscience4 citations

TL;DR

This paper proposes a bilateral artificial neural network where the left hemisphere specializes in specific features and the right in general features, using distinct training objectives. A weighted attention mechanism fuses these complementary representations, significantly outperforming single-branch and symmetric baselines on a hierarchical image classification task, demonstrating that hemispheric specialization enhances performance and offers a biologically inspired inductive bias for AI systems.

ABSTRACT

The brains of all bilaterally symmetric animals on Earth are divided into left and right hemispheres. The anatomy and functionality of the hemispheres have a large degree of overlap, but there are asymmetries, and they specialise in possesses different attributes. Other authors have used computational models to mimic hemispheric asymmetries with a focus on reproducing human data on semantic and visual processing tasks. We took a different approach and aimed to understand how dual hemispheres in a bilateral architecture interact to perform well in a given task. We propose a bilateral artificial neural network that imitates lateralisation observed in nature: that the left hemisphere specialises in local features and the right in global features. We used different training objectives to achieve the desired specialisation and tested it on an image classification task with two different CNN backbones: ResNet and VGG. Our analysis found that the hemispheres represent complementary features that are exploited by a network head that implements a type of weighted attention. The bilateral architecture outperformed a range of baselines of similar representational capacity that do not exploit differential specialisation, with the exception of a conventional ensemble of unilateral networks trained on dual training objectives for local and global features. The results demonstrate the efficacy of bilateralism, contribute to the discussion of bilateralism in biological brains, and the principle may serve as an inductive bias for new AI systems.

Motivation & Objective

• To investigate how dual hemispheres with specialized functions can complement each other to improve performance in a cognitive task.
• To test the hypothesis that differential specialization in left and right artificial neural networks enhances classification accuracy on hierarchical image tasks.
• To explore whether hemispheric specialization, inspired by biological brains, can serve as a useful inductive bias in artificial intelligence.
• To provide a computationally testable mechanism for how bilateralism might support intelligence in both biological and artificial systems.

Proposed method

• The model uses two ResNet-9 convolutional neural networks as left and right hemispheres, each trained with different objectives: one optimized for specific classes, the other for general classes.
• The left hemisphere is trained to classify fine-grained, specific categories (e.g., penguin, seal), while the right is trained on broader, general categories (e.g., sea creature).
• A shared, learnable fully connected layer acts as a weighted attention mechanism, fusing features from both hemispheres to produce a final prediction.
• The architecture is evaluated on a hierarchical image classification benchmark with both general and specific class labels.
• Performance is compared against multiple baselines of equivalent representational capacity, including single-branch networks and symmetric dual-branch models.
• The model’s internal representations are analyzed to assess the nature of features extracted by each hemisphere and their complementarity.

Experimental results

Research questions

• RQ1 How do left and right hemispheres with specialized training objectives contribute to improved classification performance in a hierarchical image task?
• RQ2 What mechanism enables the bilateral system to outperform symmetric or single-branch architectures?
• RQ3 To what extent do the left and right hemispheres extract complementary features—specific versus general—consistent with biological hemispheric specialization?
• RQ4 Can a simple weighted attention mechanism effectively integrate specialized representations from two distinct neural pathways?
• RQ5 Does hemispheric specialization in artificial networks yield performance gains beyond the explicit objectives of individual hemispheres?

Key findings

• The bilateral architecture outperformed all comparable baselines, including single-branch and symmetric dual-branch models, on both general and specific image classification tasks.
• The left hemisphere specialized in extracting local, fine-grained features, while the right focused on non-local, broader contextual patterns, confirming functional complementarity.
• The weighted attention mechanism effectively fused specialized representations, enabling the system to leverage both specificity and generality for improved accuracy.
• The performance gain was not merely additive but emerged from the interaction of specialized features, indicating synergy between hemispheres.
• The results support the hypothesis that hemispheric specialization—specificity in the left, generality in the right—can be a powerful inductive bias in artificial neural networks.
• The model provides testable neuroscientific predictions, such as differential TMS effects on performance depending on hemisphere impairment, and suggests bilateralism should be considered in future cognitive modeling.