[论文解读] RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers

Md Zesun Ahmed Mia, Malyaban Bal+1 more|arXiv (Cornell University)|Jan 1, 2026

Advanced Memory and Neural Computing被引用 0

一句话总结

RMAAT 引入了 astrocyte-inspired 的内存压缩与带记忆回放训练的递归记忆增强型变换器，以高效处理长上下文序列，在 Long Range Arena 基准测试上以更低的内存占用实现具有竞争力的准确性。

ABSTRACT

The quadratic complexity of self-attention mechanism presents a significant impediment to applying Transformer models to long sequences. This work explores computational principles derived from astrocytes-glial cells critical for biological memory and synaptic modulation-as a complementary approach to conventional architectural modifications for efficient self-attention. We introduce the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), an architecture integrating abstracted astrocyte functionalities. RMAAT employs a recurrent, segment-based processing strategy where persistent memory tokens propagate contextual information. An adaptive compression mechanism, governed by a novel retention factor derived from simulated astrocyte long-term plasticity (LTP), modulates these tokens. Attention within segments utilizes an efficient, linear-complexity mechanism inspired by astrocyte short-term plasticity (STP). Training is performed using Astrocytic Memory Replay Backpropagation (AMRB), a novel algorithm designed for memory efficiency in recurrent networks. Evaluations on the Long Range Arena (LRA) benchmark demonstrate RMAAT's competitive accuracy and substantial improvements in computational and memory efficiency, indicating the potential of incorporating astrocyte-inspired dynamics into scalable sequence models.

研究动机与目标

• Motivate and address the quadratic complexity of self-attention for long sequences in Transformers.
• Propose astrocyte-inspired mechanisms to compress memory and modulate attention across segments.
• Develop AMRB training to reduce memory footprint in recurrent Transformer training.
• Demonstrate competitive accuracy with substantial memory and computational efficiency gains on LRA.

提出的方法

• Propose RMAAT with segmented processing and persistent memory tokens to carry context across segments.
• Introduce astromorphic attention with Write and Read modes inspired by neuron-astrocyte interactions.
• Derive a memory retention factor from astrocyte LTP-inspired dynamics to adaptively compress memory tokens.
• Define Astrocytic Memory Replay Backpropagation (AMRB) to train the recurrent architecture with reduced memory usage.
• Ground relative positional encoding in astrocyte STP dynamics to encode spatial context.
• Provide a two-layer neuron-astrocyte network implementation for efficient O(N) attention within segments.

Figure 1: Conceptual illustration of RMAAT processing through time unrolling. Processing within each segment incorporates mechanisms inspired by STP. The recurrent propagation of astrocytic memory tokens ( $mem_{t}$ ) integrates context across many segments, drawing inspiration from LTP principles f

实验结果

研究问题

• RQ1 How can astrocyte-inspired dynamics be integrated into Transformer-like architectures to improve long-context processing?
• RQ2 Can memory tokens with adaptive compression maintain performance while reducing memory and compute?
• RQ3 Does the AMRB training algorithm provide memory efficiency advantages over standard BPTT in recurrent transformers?
• RQ4 What is the impact of astrocyte-derived relative positioning on attention effectiveness in long sequences?

主要发现

• RMAAT achieves competitive accuracy on the Long Range Arena benchmark across multiple tasks.
• RMAAT uses significantly lower peak memory than iso-architecture recurrent baselines (e.g., RMT).
• On the Long Range Arena tasks, RMAAT attains strong performance especially on longer-context tasks like Retrieval and Pathfinder.
• AMRB enables memory-efficient training by replaying recomputed forward passes using a compact memory token sequence, reducing storage requirements.
• The memory retention factor provides adaptive, bio-inspired context compression that degrades memory less for newer information, enabling long-range context propagation.
• Ablation shows removal of memory compression (akin to RMT) degrades accuracy, highlighting the synergy between compression and AMRB.

Figure 2: Overview of the Astromorphic Transformer architecture. This diagram illustrates the integration of bioplausible bidirectional feedback mechanisms within a two-layered neuron-astrocyte network, emulating the Self-Attention of the transformer encoder. The synaptic weights $W_{K}$ , $W_{Q}$ ,