[论文解读] Universality and individuality in neural dynamics across large populations of recurrent networks

Niru Maheswaranathan, Alex H. Williams+3 more|PubMed|Jul 19, 2019

Functional Brain Connectivity Studies参考文献 55被引用 39

一句话总结

该论文探究 RNN 动态如何随架构而异，发现 表征几何对架构敏感，而固定点的拓扑结构与线性化动力学在解决同一任务的模型中显示出更强的普适性。

ABSTRACT

Task-based modeling with recurrent neural networks (RNNs) has emerged as a popular way to infer the computational function of different brain regions. These models are quantitatively assessed by comparing the low-dimensional neural representations of the model with the brain, for example using canonical correlation analysis (CCA). However, the nature of the detailed neurobiological inferences one can draw from such efforts remains elusive. For example, to what extent does training neural networks to solve common tasks uniquely determine the network dynamics, independent of modeling architectural choices? Or alternatively, are the learned dynamics highly sensitive to different model choices? Knowing the answer to these questions has strong implications for whether and how we should use task-based RNN modeling to understand brain dynamics. To address these foundational questions, we study populations of thousands of networks, with commonly used RNN architectures, trained to solve neuroscientifically motivated tasks and characterize their nonlinear dynamics. We find the geometry of the RNN representations can be highly sensitive to different network architectures, yielding a cautionary tale for measures of similarity that rely on representational geometry, such as CCA. Moreover, we find that while the geometry of neural dynamics can vary greatly across architectures, the underlying computational scaffold-the topological structure of fixed points, transitions between them, limit cycles, and linearized dynamics-often appears universal across all architectures.

研究动机与目标

• Motivate how task-based RNN modeling relates to brain dynamics across diverse architectures.
• Quantify how representational geometry and dynamical topology differ across RNN architectures solving neuroscience-inspired tasks.
• Identify which dynamical properties are universal versus architecture-specific.
• Provide methods to compare RNN dynamics beyond simple representational similarity measures.

提出的方法

• Train thousands of RNNs across four architectures (Vanilla, UGRNN, GRU, LSTM) with relu or tanh activations, various sizes, and regularization.
• Use tasks inspired by neuroscience (3-bit memory, sine wave generation, context-dependent integration) to elicit diverse dynamics.
• Assess representational similarity with SVCCA and CKA, focusing on geometry of network representations.
• Analyze dynamics via fixed points, their stability, and linearized dynamics around fixed points.
• Construct fixed-point graphs from transition probabilities to quantify topology of computation.
• Use multidimensional scaling (MDS) to visualize network-network dissimilarities based on SVCCA or fixed-point topology.

实验结果

研究问题

• RQ1 How does the geometry of neural representations vary with RNN architecture and activation function when solving the same task?
• RQ2 Are there universal dynamical structures (e.g., fixed points, limit cycles, linearized dynamics) across architectures solving the same tasks?
• RQ3 Do representational geometry and functional similarity align across trained networks of different architectures or activations?
• RQ4 To what extent do fixed-point topologies provide a universal account of computation across architectures?
• RQ5 What are the cautions of using representational similarity metrics like SVCCA/CKA when comparing networks across architectures?

主要发现

• Representational geometry is highly sensitive to architecture and activation function.
• Fixed-point topology and linearized dynamics around fixed points show greater universality across architectures.
• Trained and untrained networks of the same nonlinearity can be more similar than networks of different nonlinearities, illustrating limits of SVCCA/CKA.
• Across tasks, most architectures converge to qualitatively similar dynamical solutions in terms of fixed points and line attractors.
• Gated architectures differ in the dimensionality of the linear modes used for integration, indicating architecture-specific strategies within a universal framework.
• Topological analyses (fixed-point graphs) reveal more uniformity across architectures than geometry-based measures.