[Paper Review] When, where, and how to add new neurons to ANNs

Kaitlin Maile, Emmanuel Rachelson+2 more|arXiv (Cornell University)|Feb 17, 2022

Neural Networks and Applications4 citations

TL;DR

This paper introduces NORTH*, a neurogenesis framework that dynamically adds neurons to artificial neural networks using orthogonality-based triggers and informed initializations. By determining when, where, and how to grow networks during training, NORTH* achieves compact, high-performing architectures that converge efficiently without manual architecture tuning.

ABSTRACT

Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.

Motivation & Objective

• To address the underexplored problem of neurogenesis—dynamically adding neurons during training—by decomposing it into triggers and initializations.
• To develop a framework that answers the core questions of when, where, and how to add neurons in a way that improves network efficiency and performance.
• To evaluate whether orthogonality of activations or weights can serve as effective, scalable triggers and initialization strategies for neurogenesis.
• To demonstrate that dynamic neurogenesis can produce compact, high-performing networks that rival or exceed static architectures without extensive hyperparameter tuning.

Proposed method

• Decomposes neurogenesis into two components: triggers (when and where to add neurons) and initializations (how to set weights of new neurons).
• Proposes activation-orthogonality-based triggers that measure the orthogonality of post-activation vectors across mini-batches to detect when new neurons are needed.
• Introduces weight-orthogonality-based triggers that assess the orthogonality of weight matrices to guide layer-specific neuron addition.
• Employs gradient-based triggers that evaluate the magnitude of gradients in the direction of new neuron parameters to detect learning potential.
• Uses orthogonality-preserving initialization for new neurons, ensuring that fan-in and fan-out weights are set to maintain orthogonal relationships with existing network components.
• Applies triggers after every gradient update, enabling dynamic, adaptive neuron addition throughout training without pre-defined schedules.

Experimental results

Research questions

• RQ1 Can orthogonality of activations or weights serve as effective, scalable triggers for dynamic neuron addition in deep networks?
• RQ2 How does the choice of initialization strategy—particularly orthogonality-based vs. random or gradient-based—affect the performance and convergence of dynamically grown networks?
• RQ3 Can neurogenesis strategies that grow networks during training achieve better compactness and performance than static architectures without manual architecture search?
• RQ4 How do different trigger frequencies and hyperparameters (e.g., number of candidates, activation buffer size) affect the efficiency and effectiveness of neurogenesis?
• RQ5 Can neurogenesis be extended to continual learning or unsupervised settings where gradient information may be unreliable?

Key findings

• NORTH* strategies achieve compact yet performant networks across multiple supervised learning tasks, outperforming baseline static architectures in terms of model efficiency and accuracy.
• The use of activation and weight orthogonality as triggers leads to dynamic neuron addition that avoids redundant neurons, especially in early training stages.
• NORTH* outperforms gradient-based neurogenesis methods in GPU efficiency, though the latter may be more effective on CPU due to computational cost trade-offs.
• The framework enables function-preserving neuron addition, where new neurons do not immediately alter network output but learn through gradient updates.
• Orthogonality-based initialization reduces the risk of disruptive changes to the loss landscape, unlike functional initialization methods such as those in Firefly or NeST.
• The results suggest that neurogenesis can reduce the environmental cost of neural architecture search by eliminating the need for expensive super-networks or repeated hand-tuning.