[Paper Review] Dopamine: Brain Modes, Not Brains

Shervin Ghasemlou|arXiv (Cornell University)|Feb 12, 2026

Neurological disorders and treatments0 citations

TL;DR

The paper introduces TauGate, an activation-space PEFT method that freezes base weights and learns per-neuron thresholds and gains to gate neuron activations, enabling interpretable mode switching with a small parameter budget. A MNIST rotation mode study demonstrates competitive accuracy with few trainable parameters and interpretable gating.

ABSTRACT

Parameter-efficient fine-tuning (PEFT) methods such as \lora{} adapt large pretrained models by adding small weight-space updates. While effective, weight deltas are hard to interpret mechanistically, and they do not directly expose \emph{which} internal computations are reused versus bypassed for a new task. We explore an alternative view inspired by neuromodulation: adaptation as a change in \emph{mode} -- selecting and rescaling existing computations -- rather than rewriting the underlying weights. We propose \methodname{}, a simple activation-space PEFT technique that freezes base weights and learns per-neuron \emph{thresholds} and \emph{gains}. During training, a smooth gate decides whether a neuron's activation participates; at inference the gate can be hardened to yield explicit conditional computation and neuron-level attributions. As a proof of concept, we study ``mode specialization'' on MNIST (0$^\circ$) versus rotated MNIST (45$^\circ$). We pretrain a small MLP on a 50/50 mixture (foundation), freeze its weights, and then specialize to the rotated mode using \methodname{}. Across seeds, \methodname{} improves rotated accuracy over the frozen baseline while using only a few hundred trainable parameters per layer, and exhibits partial activation sparsity (a minority of units strongly active). Compared to \lora{}, \methodname{} trades some accuracy for substantially fewer trainable parameters and a more interpretable ``which-neurons-fire'' mechanism. We discuss limitations, including reduced expressivity when the frozen base lacks features needed for the target mode.

Motivation & Objective

• Motivate adaptive computation without weight rewiring by drawing an analogy to neuromodulation.
• Develop TauGate to learn per-neuron thresholds and gains while freezing base weights.
• Demonstrate mode specialization on MNIST 0° vs 45° rotation with a small parameter budget.
• Compare TauGate to bias-only tuning, LoRA, and full fine-tuning in a controlled setting.
• Provide insights on interpretability and limitations of activation-space PEFT.

Proposed method

• Define TauGate as per-neuron thresholds and gains with a smooth gate g = sigmoid(s(z - tau)).
• Freeze the base weights and train only (tau, gamma) per layer.
• Optionally harden gates at inference to yield explicit conditional computation.
• Encourage sparsity via a gate-activation regularizer.
• Quantify parameter count and compare to baselines (e.g., BitFit, LoRA, Full FT).
• Provide a reproducible MNIST rotation experiment using a small MLP and DirectML.

Figure 1: Test accuracy on MNIST (0 ∘ ) and rotated MNIST (45 ∘ ) after specializing to the rotated mode. TauGate improves over the frozen foundation baseline with a small parameter budget and exhibits partial activation sparsity (“High-act frac” in Table 1 ).

Experimental results

Research questions

• RQ1 Can activation-space gating via TauGate achieve mode-specific specialization with a frozen backbone?
• RQ2 How does TauGate compare to weight-space PEFT methods in accuracy and parameter efficiency?
• RQ3 Does TauGate provide interpretable neuron-level attributions through hard-gated subnetworks?
• RQ4 What are the trade-offs and limitations of gating-based PEFT in limited-data toy settings?

Key findings

• TauGate adds 512 trainable parameters and achieves 0.850 accuracy on both 0° and 45° MNIST after specializing to the rotated mode.
• Compared to frozen backbone, TauGate improves rotated-mode accuracy while preserving original mode performance.
• BitFit with 266 trainable params yields 0.851 (0°) and 0.852 (45°).
• LoRA (r=8) uses 10,448 trainable params and reaches 0.855 (0°) and 0.864 (45°).
• Full fine-tuning uses 118,282 trainable params and yields 0.783 (0°) and 0.887 (45°).
• TauGate shows partial activation sparsity with 0.28 fraction of high-activation units, indicating mode-specific gating.