[Paper Review] Synaptic Weight Distributions Depend on the Geometry of Plasticity

Roman Pogodin, Jonathan Cornford+4 more|arXiv (Cornell University)|May 30, 2023

Neuroscience and Neuropharmacology ResearchNeuroscience3 citations

TL;DR

This paper demonstrates that synaptic weight distributions in the brain are shaped by the underlying geometry of plasticity, not just the loss function. Using mirror descent theory, it shows that log-normal weight distributions—commonly observed experimentally—imply non-Euclidean synaptic geometry, challenging the assumption of Euclidean gradient descent in neuroscience models.

ABSTRACT

A growing literature in computational neuroscience leverages gradient descent and learning algorithms that approximate it to study synaptic plasticity in the brain. However, the vast majority of this work ignores a critical underlying assumption: the choice of distance for synaptic changes - i.e. the geometry of synaptic plasticity. Gradient descent assumes that the distance is Euclidean, but many other distances are possible, and there is no reason that biology necessarily uses Euclidean geometry. Here, using the theoretical tools provided by mirror descent, we show that the distribution of synaptic weights will depend on the geometry of synaptic plasticity. We use these results to show that experimentally-observed log-normal weight distributions found in several brain areas are not consistent with standard gradient descent (i.e. a Euclidean geometry), but rather with non-Euclidean distances. Finally, we show that it should be possible to experimentally test for different synaptic geometries by comparing synaptic weight distributions before and after learning. Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.

Motivation & Objective

• To challenge the widespread assumption in computational neuroscience that synaptic plasticity uses Euclidean geometry (i.e., L2-norm distance) in weight space.
• To investigate whether experimentally observed synaptic weight distributions, particularly log-normal distributions, are consistent with standard gradient descent under Euclidean geometry.
• To develop a theoretical framework based on mirror descent that links synaptic geometry to the resulting distribution of synaptic weights.
• To provide a method for experimentally testing candidate synaptic geometries by comparing synaptic weight distributions before and after learning.
• To demonstrate that non-Euclidean geometries—such as those induced by negative entropy or p-norms—are more consistent with empirical data than Euclidean geometry.

Proposed method

• Applies mirror descent theory to model synaptic plasticity, where the choice of divergence (e.g., Kullback-Leibler, negative entropy, p-norms) defines the synaptic geometry.
• Derives that in the dual space, total synaptic weight changes are approximately Gaussian under mild assumptions, regardless of the loss function.
• Uses this duality to show that if synaptic weight distributions are log-normal in the original space, they must arise from a non-Euclidean geometry in the dual space.
• Empirically validates the framework using deep neural network training with various p-norms and negative entropy, observing weight change magnitudes under different geometries.
• Compares model predictions to experimental data from spine volume measurements (e.g., from [34]), fitting mixture models in dual space to match observed log-normal distributions.
• Proposes experimental tests: comparing pre- and post-learning synaptic weight distributions to rule out candidate geometries based on their predicted distributional outcomes.

Experimental results

Research questions

• RQ1 Does the observed log-normal distribution of synaptic weights in the brain imply a non-Euclidean geometry of synaptic plasticity?
• RQ2 Can the geometry of synaptic plasticity be inferred from the distribution of synaptic weights, independent of the loss function?
• RQ3 Is standard gradient descent with Euclidean geometry consistent with experimentally observed synaptic weight distributions?
• RQ4 Can synaptic geometry be experimentally tested by comparing synaptic weight distributions before and after learning?
• RQ5 What are the implications of non-Euclidean synaptic geometry for models of brain learning and plasticity?

Key findings

• Log-normal synaptic weight distributions, commonly observed in cortical and hippocampal regions, are inconsistent with standard gradient descent under Euclidean geometry.
• Synaptic weight distributions depend on the underlying geometry of plasticity, as shown by mirror descent theory, even when the loss function is held constant.
• The dual space of synaptic changes is approximately Gaussian under any valid geometry, and matching this to experimentally observed distributions identifies the true geometry.
• Negative entropy and p-norm geometries (e.g., 3-norm) can reproduce experimentally observed log-normal and mixture distributions of synaptic weights.
• Weight changes in trained deep networks were typically below 0.2 in magnitude, with negative entropy changes remaining under 0.02, supporting the validity of the small-change approximation.
• The framework enables experimental prediction: comparing pre- and post-learning synaptic weight distributions can rule out candidate geometries, offering a testable hypothesis for future neuroscience research.