[Paper Review] The Transient Nature of Emergent In-Context Learning in Transformers

Aaditya K. Singh, Stephanie C. Y. Chan+4 more|arXiv (Cornell University)|Nov 14, 2023

Neural Networks and Applications4 citations

TL;DR

This paper demonstrates that in-context learning (ICL) in transformers is often transient, emerging during training only to be overtaken by in-weights learning (IWL) as training continues, despite decreasing loss. The authors show that L2 regularization can stabilize ICL, preventing its disappearance and offering a path to persistent ICL without early stopping.

ABSTRACT

Transformer neural networks can exhibit a surprising capacity for in-context learning (ICL) despite not being explicitly trained for it. Prior work has provided a deeper understanding of how ICL emerges in transformers, e.g. through the lens of mechanistic interpretability, Bayesian inference, or by examining the distributional properties of training data. However, in each of these cases, ICL is treated largely as a persistent phenomenon; namely, once ICL emerges, it is assumed to persist asymptotically. Here, we show that the emergence of ICL during transformer training is, in fact, often transient. We train transformers on synthetic data designed so that both ICL and in-weights learning (IWL) strategies can lead to correct predictions. We find that ICL first emerges, then disappears and gives way to IWL, all while the training loss decreases, indicating an asymptotic preference for IWL. The transient nature of ICL is observed in transformers across a range of model sizes and datasets, raising the question of how much to "overtrain" transformers when seeking compact, cheaper-to-run models. We find that L2 regularization may offer a path to more persistent ICL that removes the need for early stopping based on ICL-style validation tasks. Finally, we present initial evidence that ICL transience may be caused by competition between ICL and IWL circuits.

Motivation & Objective

• To investigate whether in-context learning (ICL) in transformers is a persistent or transient phenomenon during training.
• To examine the conditions under which ICL emerges and subsequently disappears, even as training loss decreases.
• To explore whether in-weights learning (IWL) asymptotically dominates ICL due to optimization dynamics.
• To evaluate whether regularization techniques like L2 can stabilize ICL and prevent its transience.
• To investigate the potential competition between ICL and IWL circuits in the transformer residual stream.

Proposed method

• The authors train transformers on synthetic, bursty few-shot classification datasets where both ICL and IWL are viable strategies for achieving high accuracy.
• They use separate evaluation heads to measure ICL and IWL performance independently throughout training, tracking accuracy over time.
• They monitor training loss and evaluate model behavior across multiple model sizes and data distributions to assess generalizability.
• They apply L2 regularization during training to test its effect on stabilizing ICL and preventing its disappearance.
• They analyze the competition between ICL and IWL circuits in the residual stream, hypothesizing that resource constraints favor IWL asymptotically.
• They extend experiments to language model token embeddings to test ICL transience in settings closer to real LLMs.

Experimental results

Research questions

• RQ1 Does in-context learning (ICL) persist asymptotically in transformers, or does it emerge and then fade during training?
• RQ2 Why does ICL disappear even as training loss continues to decrease, indicating ongoing optimization?
• RQ3 Is there a competition between ICL and in-weights learning (IWL) circuits that explains the transient nature of ICL?
• RQ4 Can regularization techniques such as L2 regularization stabilize ICL and prevent its disappearance?
• RQ5 To what extent does the data distribution (e.g., burstiness, class skew) influence the emergence and transience of ICL?

Key findings

• ICL first emerges during training but then disappears entirely, even as training loss continues to decrease, indicating that ICL is not asymptotically preferred.
• In-weights learning (IWL) consistently overtakes ICL as training progresses, suggesting that IWL is asymptotically favored under standard cross-entropy training.
• The transient behavior of ICL is observed across multiple model sizes and datasets, indicating it is a general phenomenon in transformers.
• L2 regularization effectively stabilizes ICL, preventing its disappearance and enabling persistent in-context learning without early stopping.
• The results suggest that competition between ICL and IWL circuits in the residual stream may underlie the transience of ICL, with IWL gaining dominance due to the soft attention mechanism's limitations in matching context.
• Preliminary evidence indicates that ICL transience may be mitigated by regularization, though this comes at the cost of reduced IWL performance.