[Paper Review] Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks

Fenglei Fan, Yingxin Li+3 more|arXiv (Cornell University)|Jan 23, 2023

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper proposes integrating biological neuronal diversity into artificial neural networks to enhance efficiency, interpretability, and memory—key limitations in current AI. By modeling heterogeneous neurons (e.g., quadratic, polynomial, dendritic) inspired by brain biology, the authors demonstrate that diverse neuron types can outperform standard ReLU-based networks in complexity and generalization, advancing NeuroAI through theory, design, and neuroinformatics.

ABSTRACT

Throughout history, the development of artificial intelligence, particularly artificial neural networks, has been open to and constantly inspired by the increasingly deepened understanding of the brain, such as the inspiration of neocognitron, which is the pioneering work of convolutional neural networks. Per the motives of the emerging field: NeuroAI, a great amount of neuroscience knowledge can help catalyze the next generation of AI by endowing a network with more powerful capabilities. As we know, the human brain has numerous morphologically and functionally different neurons, while artificial neural networks are almost exclusively built on a single neuron type. In the human brain, neuronal diversity is an enabling factor for all kinds of biological intelligent behaviors. Since an artificial network is a miniature of the human brain, introducing neuronal diversity should be valuable in terms of addressing those essential problems of artificial networks such as efficiency, interpretability, and memory. In this Primer, we first discuss the preliminaries of biological neuronal diversity and the characteristics of information transmission and processing in a biological neuron. Then, we review studies of designing new neurons for artificial networks. Next, we discuss what gains can neuronal diversity bring into artificial networks and exemplary applications in several important fields. Lastly, we discuss the challenges and future directions of neuronal diversity to explore the potential of NeuroAI.

Motivation & Objective

• To address critical limitations in artificial neural networks—such as low efficiency, poor interpretability, and limited memory—by drawing on biological neuronal diversity.
• To explore how incorporating multiple neuron types (e.g., quadratic, polynomial, dendritic) can improve network performance beyond standard homogeneous ReLU-based units.
• To establish a theoretical and methodological foundation for heterogeneous networks, including optimization, generalization, and scalability analysis.
• To promote the development of neuroinformatics for artificial neurons, enabling standardized modeling, knowledge graphs, and multi-module synergy.
• To catalyze the next generation of AI by integrating neuroscience insights into artificial networks, advancing the emerging field of NeuroAI.

Proposed method

• Proposes a framework for designing artificial neurons with diverse computational mechanisms—e.g., quadratic, polynomial, and dendritic computation—inspired by biological neuron morphology and function.
• Introduces the QuadraLib library for efficient optimization and design exploration of quadratic networks, enabling scalable training of non-linear neuron types.
• Adapts and extends existing deep learning tools (e.g., ReLinear training algorithm) to support novel neuron architectures such as quadratic autoencoders.
• Applies tensor decomposition techniques to sparsify deep polynomial networks, improving efficiency without sacrificing expressivity.
• Develops a neuroinformatics pipeline for artificial neurons, including ontology modeling, database construction, and knowledge graph integration to support neuron fitness and connectivity analysis.
• Proposes theoretical extensions to existing deep learning theory (e.g., neural tangent kernel, double descent) to assess generalization and optimization in heterogeneous networks.

Figure 1: The rapid growth of the number of articles on the research of new neurons in deep learning. The data are based on the search in the Web of Science on December 12, 2022, with the time range from 2000 to 2022 according to the keyword “Deep Learning New Neurons”.

Experimental results

Research questions

• RQ1 To what extent can introducing biologically inspired neuronal diversity improve the efficiency, interpretability, and memory capacity of artificial neural networks?
• RQ2 How do heterogeneous neuron types (e.g., quadratic, polynomial, dendritic) compare to standard ReLU neurons in terms of expressivity and generalization?
• RQ3 What theoretical foundations are needed to analyze optimization, generalization, and scalability in heterogeneous neural networks, given the departure from the 'one-for-all' homogeneous paradigm?
• RQ4 How can neuroinformatics tools be developed to standardize, organize, and model artificial neurons and their interactions for multi-module synergy?
• RQ5 What are the practical and theoretical implications of replacing homogeneous activation functions with diverse, functionally specialized neurons in deep learning architectures?

Key findings

• Neuronal diversity in artificial networks can significantly enhance expressivity and complexity, with depth being more effective than width in maximizing complexity measures.
• The QuadraLib library, which supports quadratic networks, won the best paper award at MLSys2022, demonstrating practical feasibility and efficiency gains.
• Quadratic and polynomial networks can be sparsified via tensor decomposition, enabling efficient training and deployment while preserving representational power.
• Dendritic computation models, as detailed in the book 'Dendrite', provide a biologically plausible mechanism for non-linear integration, supporting advanced information processing.
• Theoretical analysis suggests that heterogeneous networks may outperform homogeneous ones in efficiency, though generalization and optimization theories for such networks remain underdeveloped.
• Neuroinformatics of artificial neurons—through ontology, databases, and knowledge graphs—can enable systematic design, interpretation, and synergy of diverse neuron types in artificial networks.

Figure 2: Neurons exhibiting an extraordinary morphological diversity. The shape classifications of neurons include unipolar, bipolar, pseudounipolar, multipolar, and so on.