[Paper Review] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

Kenneth Li, Aspen Hopkins+4 more|arXiv (Cornell University)|Oct 24, 2022

Topic Modeling60 citations

TL;DR

The paper shows that a GPT variant trained to predict legal moves in Othello develops emergent nonlinear internal representations of the board state, demonstrates their causal influence on predictions via interventions, and introduces latent saliency maps for interpretability.

ABSTRACT

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

Motivation & Objective

• Investigate whether sequence models trained on a synthetic task develop internal world representations beyond surface statistics.
• Examine if the emergent board-state representation is nonlinear and causally affects predictions.
• Develop and validate intervention techniques to manipulate internal representations and assess their causal role.
• Introduce latent saliency maps to visualize and explain model predictions in board-game settings.

Proposed method

• Train an 8-layer GPT variant (Othello-GPT) with 60-token vocabulary representing Othello board tiles to predict the next legal move from partial transcripts.
• Use two datasets: a championship dataset with expert human moves and a large synthetic dataset of random legal moves.
• Evaluate the model’s ability to predict legal moves against validation data (top-1 legality error rates).
• Probe internal activations with linear and nonlinear classifiers to infer board-state representations from intermediate layers.
• Apply an intervention technique that gradient-descent optimizes activations to force counterfactual board states and observes changes in predictions.
• Construct latent saliency maps by attributing prediction changes to state changes of individual tiles via interventions.

Experimental results

Research questions

• RQ1 Do GPT-style sequence models learning from transcripts of a board game develop internal, nonlinear representations of the game state (the world)?
• RQ2 Are these emergent representations causally related to the model’s next-move predictions?
• RQ3 Can activation interventions reveal and manipulate the model’s internal world model.
• RQ4 Can latent saliency maps provide interpretable visualizations of how board states influence predictions?

Key findings

• Othello-GPT predicts legal moves with very low error on synthetic data (0.01%) and championship data (5.17%), far better than untrained baselines (93.29%).
• Linear probes poorly recover board state (high error rates across layers; e.g., all >20%), while nonlinear probes (2-layer MLP) achieve substantially lower errors, indicating a nonlinear internal representation.
• Interventions that alter internal activations to change a single board tile state can causally affect next-move predictions, with best results at intervening layer Ls=4 (average errors 0.12 and 0.06 on natural and unnatural subsets).
• Latent saliency maps show region-specific tile saliency consistent with legality patterns in the synthetic model and reveal more complex patterns in the championship-trained model.
• The latent-saliency visualization distinguishes representations learned under synthetic versus championship training, illustrating the method’s interpretability utility.