[Paper Review] Language Models Struggle to Use Representations Learned In-Context

Michael A. Lepori, Tal Linzen+2 more|arXiv (Cornell University)|Feb 4, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper shows that open-weight LMs largely fail to deploy in-context representations to next-token prediction or adaptive world modeling, though frontier reasoning models fare somewhat better in limited settings.

ABSTRACT

Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks. We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.

Motivation & Objective

• Investigate whether LMs can induce rich in-context representations of novel semantics.
• Assess if these in-context representations can be flexibly deployed for downstream tasks like next-token prediction.
• Introduce and evaluate a new task, adaptive world modeling (AWM), to probe deployability of in-context semantics.
• Compare open-weight LMs with frontier reasoning models on deployment of in-context representations.

Proposed method

• Replicate in-context representation learning using a graph-tracing task on 2D grids and 1D lines with 16/25 states.
• Measure topology alignment of token representations via Dirichlet Energy (DE) and Distance Correlation (DC).
• Test next-token prediction under two conditions: Instruction (delayed use) vs Prefilled (immediate use).
• Introduce adaptive world modeling (AWM): combine graph tracing with few-shot rules mapping states to new states.
• Analyze whether in-context representations remain inert during AWM and whether explicit topology helps some models.

Figure 1: (Top Left) Example of an N-by-N state space topology used to generate a random walk. (Right) Examples of next token prediction prompts in the Instruction or Prefilled condition. Prompt formatting tokens are not bolded for readability. In the Instruction condition, models need to deploy in-

Experimental results

Research questions

• RQ1 Can open-weight LMs deploy in-context learned representations for next-token prediction when required to delay usage?
• RQ2 Are in-context representations sufficient to support adaptive world modeling tasks?
• RQ3 Do frontier reasoning models leverage in-context semantics more effectively than open-weight models, and under what conditions?
• RQ4 How does presenting topology explicitly in prompts affect few-shot rule learning in AWM?

Key findings

• Open-weight LMs struggle to use in-context representations for next-token prediction when the random walk is in the user prompt (Instruction Condition).
• Open-weight LMs show limited or no deployability of in-context representations for Adaptive World Modeling across configurations.
• Some frontier reasoning models achieve nontrivial accuracy on one-dimensional topologies but fail on two-dimensional (grid) topologies in AWM.
• When topology is explicitly described in prompts, frontier models reach ceiling performance on few-shot learning, indicating reliance on externalized topology information.
• Representations learned in-context are often inert and not flexibly deployable for downstream tasks, even with latent topology encoding.
• Explicit topology descriptions in prompts can improve few-shot rule learning, suggesting representations may be deployable when readily available.

Figure 2: Example of in-context representation learning over a 5-by-5 grid topology.