[Paper Review] Universality and individuality in neural dynamics across large populations of recurrent networks

Niru Maheswaranathan, Alex H. Williams+3 more|PubMed|Jul 19, 2019

Functional Brain Connectivity Studies55 references39 citations

TL;DR

The paper probes how RNN dynamics vary across architectures, finding that representational geometry is architecture-dependent while the topological structure of fixed points and linearized dynamics shows a stronger universality across models solving the same tasks.

ABSTRACT

Task-based modeling with recurrent neural networks (RNNs) has emerged as a popular way to infer the computational function of different brain regions. These models are quantitatively assessed by comparing the low-dimensional neural representations of the model with the brain, for example using canonical correlation analysis (CCA). However, the nature of the detailed neurobiological inferences one can draw from such efforts remains elusive. For example, to what extent does training neural networks to solve common tasks uniquely determine the network dynamics, independent of modeling architectural choices? Or alternatively, are the learned dynamics highly sensitive to different model choices? Knowing the answer to these questions has strong implications for whether and how we should use task-based RNN modeling to understand brain dynamics. To address these foundational questions, we study populations of thousands of networks, with commonly used RNN architectures, trained to solve neuroscientifically motivated tasks and characterize their nonlinear dynamics. We find the geometry of the RNN representations can be highly sensitive to different network architectures, yielding a cautionary tale for measures of similarity that rely on representational geometry, such as CCA. Moreover, we find that while the geometry of neural dynamics can vary greatly across architectures, the underlying computational scaffold-the topological structure of fixed points, transitions between them, limit cycles, and linearized dynamics-often appears universal across all architectures.

Motivation & Objective

• Motivate how task-based RNN modeling relates to brain dynamics across diverse architectures.
• Quantify how representational geometry and dynamical topology differ across RNN architectures solving neuroscience-inspired tasks.
• Identify which dynamical properties are universal versus architecture-specific.
• Provide methods to compare RNN dynamics beyond simple representational similarity measures.

Proposed method

• Train thousands of RNNs across four architectures (Vanilla, UGRNN, GRU, LSTM) with relu or tanh activations, various sizes, and regularization.
• Use tasks inspired by neuroscience (3-bit memory, sine wave generation, context-dependent integration) to elicit diverse dynamics.
• Assess representational similarity with SVCCA and CKA, focusing on geometry of network representations.
• Analyze dynamics via fixed points, their stability, and linearized dynamics around fixed points.
• Construct fixed-point graphs from transition probabilities to quantify topology of computation.
• Use multidimensional scaling (MDS) to visualize network-network dissimilarities based on SVCCA or fixed-point topology.

Experimental results

Research questions

• RQ1 How does the geometry of neural representations vary with RNN architecture and activation function when solving the same task?
• RQ2 Are there universal dynamical structures (e.g., fixed points, limit cycles, linearized dynamics) across architectures solving the same tasks?
• RQ3 Do representational geometry and functional similarity align across trained networks of different architectures or activations?
• RQ4 To what extent do fixed-point topologies provide a universal account of computation across architectures?
• RQ5 What are the cautions of using representational similarity metrics like SVCCA/CKA when comparing networks across architectures?

Key findings

• Representational geometry is highly sensitive to architecture and activation function.
• Fixed-point topology and linearized dynamics around fixed points show greater universality across architectures.
• Trained and untrained networks of the same nonlinearity can be more similar than networks of different nonlinearities, illustrating limits of SVCCA/CKA.
• Across tasks, most architectures converge to qualitatively similar dynamical solutions in terms of fixed points and line attractors.
• Gated architectures differ in the dimensionality of the linear modes used for integration, indicating architecture-specific strategies within a universal framework.
• Topological analyses (fixed-point graphs) reveal more uniformity across architectures than geometry-based measures.