[Paper Review] Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis

Mitchell Ostrow, Adam Eisen+2 more|arXiv (Cornell University)|Jun 16, 2023

Neural dynamics and brain function8 citations

TL;DR

The paper introduces Dynamical Similarity Analysis (DSA), a data-driven method to compare neural systems at the level of dynamics using Koopman embeddings and a novel vector-field Procrustes metric, showing dynamical equivalence where geometric similarity fails.

ABSTRACT

How can we tell whether two neural networks utilize the same internal processes for a particular computation? This question is pertinent for multiple subfields of neuroscience and machine learning, including neuroAI, mechanistic interpretability, and brain-machine interfaces. Standard approaches for comparing neural networks focus on the spatial geometry of latent states. Yet in recurrent networks, computations are implemented at the level of dynamics, and two networks performing the same computation with equivalent dynamics need not exhibit the same geometry. To bridge this gap, we introduce a novel similarity metric that compares two systems at the level of their dynamics, called Dynamical Similarity Analysis (DSA). Our method incorporates two components: Using recent advances in data-driven dynamical systems theory, we learn a high-dimensional linear system that accurately captures core features of the original nonlinear dynamics. Next, we compare different systems passed through this embedding using a novel extension of Procrustes Analysis that accounts for how vector fields change under orthogonal transformation. In four case studies, we demonstrate that our method disentangles conjugate and non-conjugate recurrent neural networks (RNNs), while geometric methods fall short. We additionally show that our method can distinguish learning rules in an unsupervised manner. Our method opens the door to comparative analyses of the essential temporal structure of computation in neural circuits.

Motivation & Objective

• Motivate the need to compare neural computations at the level of dynamics rather than geometry.
• Develop a general, data-driven metric (DSA) that assesses dynamical similarity between systems.
• Demonstrate DSA’s ability to distinguish conjugate (topologically equivalent) dynamics from non-conjugate dynamics across diverse case studies.
• Show that DSA can reveal learning-rule differences in neural networks in an unsupervised manner.
• Promote applications of DSA for model validation and interpretation in neuroscience and ML.

Proposed method

• Embed nonlinear dynamics into a globally linear representation using delay embeddings (HAVOK) to obtain a high-dimensional linear system.
• Estimate reduced-rank DMD matrices A_x, A_y from the Koopman embedding of each system.
• Introduce Procrustes Analysis over Vector Fields: minimize over orthogonal C the Frobenius norm ||A_x − C A_y C^{-1}||_F to quantify dynamical similarity.
• Use angular form of the distance via arccos of normalized traces to obtain a metric on vector fields.
• Prove metric properties and relate DSA to conjugacy and Koopman eigenspectra (2-Wasserstein connection in special cases).
• Validate with four case studies showing geometry vs dynamics dissociation, topology invariance, and unsupervised learning-rule disentanglement.

Experimental results

Research questions

• RQ1 Can two neural networks with different latent geometries implement the same underlying dynamics (conjugate systems)?
• RQ2 Can two systems exhibit different dynamics despite similar geometric trajectories, and vice versa?
• RQ3 How does DSA respond to topological and geometric transformations of dynamical systems?
• RQ4 Can DSA disentangle learning rules from trajectory data in an unsupervised manner?
• RQ5 What is the relationship between DSA and Koopman operator theory in capturing dynamical similarity?

Key findings

• DSA identifies dynamical similarity between RNNs with different geometries but equivalent dynamics, where Procrustes on geometry fails.
• DSA distinguishes learning-rule differences in unsupervised analyses, clustering networks by learning rule and enabling reasonable classification accuracy.
• DSA remains invariant under smooth geometric deformations that preserve attractor topology (ring attractor) but detects topology changes (ring vs line attractor).
• DSA can separate dynamical types that share similar condition-averaged trajectories when noise and trial structure are present.
• Ablation shows delay embedding and HAVOK (DMD) are both necessary for correctly capturing dynamics, beyond purely temporal or geometric analyses.
• DSA can disentangle learning rules across different architectures/tasks, enabling unsupervised inference of dynamical strategies.