[Paper Review] Brief technical note on linearizing recurrent neural networks (RNNs) before vs after the pointwise nonlinearity

Marino Pagan, Adrian Valente+2 more|arXiv (Cornell University)|Sep 7, 2023

Neural Networks and Applications4 citations

TL;DR

This paper compares linearization of recurrent neural networks (RNNs) in activation space (pre-nonlinearities) versus activity space (post-nonlinearities), showing that while both linearizations are related via a diagonal gain matrix, they differ in how context-dependent input modulation is captured. The key finding is that context-dependent input modulation—critical for modeling cognitive context effects—is only observable in activity-space linearization, not in activation-space linearization.

ABSTRACT

Linearization of the dynamics of recurrent neural networks (RNNs) is often used to study their properties. The same RNN dynamics can be written in terms of the ``activations" (the net inputs to each unit, before its pointwise nonlinearity) or in terms of the ``activities" (the output of each unit, after its pointwise nonlinearity); the two corresponding linearizations are different from each other. This brief and informal technical note describes the relationship between the two linearizations, between the left and right eigenvectors of their dynamics matrices, and shows that some context-dependent effects are readily apparent under linearization of activity dynamics but not linearization of activation dynamics.

Motivation & Objective

• To clarify the distinction between linearizing RNN dynamics in terms of unit activations (pre-nonlinearity) versus activities (post-nonlinearity).
• To establish the mathematical relationship between the two linearizations, particularly through a diagonal gain matrix derived from the derivative of the nonlinearity.
• To investigate how context-dependent input modulation—arising from changes in unit gains—is represented differently in the two linearization frameworks.
• To demonstrate that certain dynamical effects, such as context-dependent input modulation, are only detectable in activity-space linearization, not in activation-space linearization.
• To guide researchers in selecting the appropriate linearization framework based on the dynamical property of interest, especially in context-dependent network modeling.

Proposed method

• Formalizes RNN dynamics using two equivalent formulations: one in terms of activations (x) before the pointwise nonlinearity g, and one in terms of activities (r) after g.
• Derives the linearized dynamics around a fixed point for both formulations, resulting in linear systems with dynamics matrices Wg′(x₀) for activations and g′(x₀)W for activities.
• Introduces a diagonal gain matrix D = diag(g′(x₀)) to relate the two linearizations, showing that the activation-space dynamics matrix is WD and the activity-space matrix is DW.
• Analyzes the eigensystem relationships between the two linearized dynamics, proving that right eigenvectors of DW correspond to scaled right eigenvectors of WD via D⁻¹.
• Applies the framework to context-dependent RNNs by introducing a constant context vector c_R, which alters the fixed point and thus the gain matrix D_R.
• Demonstrates that in activity-space linearization, the effective input is scaled by D_R (i.e., D_R u), making context-dependent input modulation visible; in activation-space, the input remains u, independent of context.

Experimental results

Research questions

• RQ1 How do the linearized dynamics of RNNs differ when linearized before versus after the pointwise nonlinearity?
• RQ2 What is the mathematical relationship between the dynamics matrices obtained from activation-space versus activity-space linearization?
• RQ3 How are the left and right eigenvectors of the two linearized dynamics matrices related?
• RQ4 Can context-dependent modulation of external inputs be observed in both linearization frameworks?
• RQ5 Why is activity-space linearization better suited for studying context-dependent input modulation in RNNs?

Key findings

• The linearized dynamics in activation space (WD) and activity space (DW) are related by a diagonal scaling matrix D = diag(g′(x₀)), meaning the two systems are equivalent up to a unit-specific gain transformation.
• Right eigenvectors of the activity-space matrix DW are related to those of the activation-space matrix WD via ρ_x = D⁻¹ρ_r, and left eigenvectors satisfy s_x = D⁻¹s_r, preserving the dot product between left and right eigenvectors.
• Context-dependent input modulation—where the same input vector u is effectively scaled by D_R in different contexts—is only visible in the activity-space linearization, where the input becomes D_R u.
• In contrast, the activation-space linearization retains the input as u, independent of the context, thus failing to capture context-dependent modulation of inputs.
• This difference implies that activity-space linearization is necessary for studying context-dependent input effects in RNNs, such as those found in cognitive modeling and working memory tasks.
• The paper concludes that while both linearizations describe the same underlying trajectories, they are not interchangeable when analyzing context-dependent dynamics, especially input modulation.