[Paper Review] Transcendence: Generative Models Can Outperform The Experts That Train Them

Edwin Zhang, Vincent Zhu+6 more|arXiv (Cornell University)|Jun 17, 2024

Philosophy and History of ScienceArts and Humanities3 citations

TL;DR

This paper introduces 'transcendence'—a phenomenon where autoregressive transformer models trained on human-generated chess games outperform all individual human experts in the training data. By leveraging low-temperature sampling to implicitly perform majority voting across diverse expert behaviors, the model denoises errors and achieves superior performance, validated empirically on chess datasets with ratings up to 1500.

ABSTRACT

Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of transcendence: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. We theoretically prove that transcendence can be enabled by low-temperature sampling, and rigorously assess this claim experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.

Motivation & Objective

• To formalize the phenomenon of 'transcendence' in generative models, where models outperform the best individual experts in their training data.
• To investigate how low-temperature sampling enables generative models to perform implicit majority voting over diverse expert behaviors, reducing errors and improving performance.
• To validate theoretically and empirically that transcendence arises from the denoising effect of ensemble-like averaging via temperature control.
• To examine the necessity of dataset diversity for effective transcendence, showing that homogeneous or low-diversity data prevents performance gains.
• To lay a theoretical and empirical foundation for studying transcendence in broader domains beyond chess, such as NLP and vision.

Proposed method

• Train an autoregressive transformer (ChessFormer) on public chess game transcripts, with training data restricted to players up to a specified rating threshold (e.g., 1000, 1300, 1500).
• Use low-temperature sampling during inference to amplify high-probability moves, effectively emulating a majority vote over expert trajectories.
• Theoretical analysis proves that under low-temperature sampling, a model trained on a single expert making random errors can surpass that expert by denoising.
• Extend the theoretical framework to show that transcendence is enabled when training data includes diverse experts with non-uniform strengths and biases.
• Empirically evaluate model performance using Glicko-2 ratings, comparing model performance against the maximum rating of any human player in the training set.
• Visualize performance gains by analyzing reward changes across game states, identifying that improvements are concentrated in a small subset of critical positions.

Figure 1: Ratings of our autoregressive decoder-only transformer, ChessFormer, over several different temperatures. We refer to our models as “ChessFormer <Maximum Glicko-2 rating seen during training>" to easily distinguish between different models in subsequent sections. Each model is trained only

Experimental results

Research questions

• RQ1 Can a generative model trained via imitation learning on human data outperform the best individual expert in the dataset?
• RQ2 What mechanism enables generative models to transcend their expert sources, and under what conditions does this occur?
• RQ3 To what extent does low-temperature sampling enable a denoising effect that mimics ensemble averaging over expert behaviors?
• RQ4 How critical is diversity in the training data for achieving transcendence, and what happens when data is less diverse?
• RQ5 Does the performance gain from transcendence stem from improvements on a few high-impact positions, or is it uniformly distributed?

Key findings

• ChessFormer models trained on human games up to rating 1300 achieved Glicko-2 ratings that significantly exceeded the maximum rating of any human player in the training data, demonstrating transcendence.
• Low-temperature sampling was essential for transcendence, as models only surpassed expert ratings when inference temperature was reduced, confirming the theoretical role of majority voting.
• Performance gains were concentrated on a small subset of game states—particularly critical positions—indicating that transcendence arises from strategic improvements in high-stakes scenarios.
• When trained on less diverse datasets, ChessFormer failed to transcend, confirming that diversity among expert behaviors is a necessary condition for effective denoising and majority voting.
• Theoretical analysis proved that a model trained on a single expert making random errors can outperform that expert under low-temperature sampling, establishing a formal basis for the phenomenon.
• The study establishes a deep connection between low-temperature sampling in generative models and ensemble methods in machine learning, showing implicit model ensembling through distributional smoothing.

Figure 2: Visualizing the denoising effects of low temperature on the action distribution: an example of ChessFormer shifting probability mass towards the high reward move of trapping the queen with the rook as the temperature $\tau$ decreases. Opacity of the red arrows represent the probability mas