[论文解读] Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens

Byung-Doh Oh, William Schuler|arXiv (Cornell University)|Apr 22, 2023

Topic Modeling被引用 7

一句话总结

本研究探讨了基于Transformer的语言模型中训练数据规模与模型容量如何影响 surprisal 对人类阅读时间的预测能力。研究发现，经过约20亿个标记训练的模型所生成的surprisal估计值与人类阅读数据的拟合度最佳，此后由于过度优化，性能开始下降。在较小模型中发现的‘临界点’表明，必须具备足够的模型容量才能捕捉到类人的预测行为。

ABSTRACT

Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a 'tipping point' at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.

研究动机与目标

• 解决关于语言模型质量与surprisal对人类阅读时间预测能力之间关系的相互矛盾的发现。
• 研究不同规模的训练数据和模型容量如何影响surprisal估计值与人类阅读时间的拟合程度。
• 确定是否存在一个数据量或模型容量的阈值，使得surprisal性能开始偏离人类处理模式。
• 检查较小模型是否表现出一个‘临界点’，即困惑度降低反而导致与人类阅读时间拟合度变差。

提出的方法

• 系统性地训练了多种基于Pythia的Transformer语言模型变体，其模型容量和训练数据规模均经过系统调整。
• 利用自回归下一个词预测方法，在不同训练阶段计算每个模型变体的surprisal估计值。
• 将surprisal预测结果与两个行为数据集进行评估：自定节奏阅读时间（Natural Stories语料库）和眼动追踪后继续阅读的时长（Dundee语料库）。
• 在分层线性回归模型中使用对数转换后的阅读时间数据，评估surprisal对阅读时间方差的解释能力。
• 追踪模型困惑度与模型拟合度（ΔLL）在训练过程中的关系，以识别性能的转折点。
• 通过比较不同模型大小和训练阶段的结果，分离出数据量和模型容量的影响。

Figure 1: Increase in regression model log-likelihood due to including each surprisal estimate from Pythia variants as a function of training steps (top) and perplexity (middle; the stars indicate the fully trained versions after 143,000 steps), as well as perplexity as a function of training steps

实验结果

研究问题

• RQ1 基于Transformer的语言模型在训练的哪个阶段，其surprisal估计值对人类阅读时间的预测能力最佳？
• RQ2 模型容量在语言模型困惑度与阅读时间预测拟合度之间的关系中起何种调节作用？
• RQ3 在较小模型中是否存在一个‘临界点’，即困惑度降低反而导致与人类阅读时间拟合度变差？
• RQ4 为何大型预训练模型表现出困惑度与预测能力之间的负相关关系，而较小模型最初却表现出正相关？
• RQ5 训练数据量在多大程度上驱动了模型surprisal与人类处理难度之间的偏差？

主要发现

• 大多数当代基于Transformer的语言模型在训练约20亿个标记后，其surprisal估计值与人类阅读时间的拟合度最佳。
• 超过20亿个标记后，进一步训练导致预测拟合度下降，表明过度优化使surprisal偏离了人类预期。
• 较小的模型变体（如2-3-192）在收敛时表现出‘临界点’，即困惑度持续降低反而导致与阅读时间的拟合度变差。
• 2-3-192模型变体代表一个关键阈值：低于此容量，模型无法有效学习类人的预期；高于此容量，模型开始过度拟合统计规律。
• 只有在大型模型经过大量训练后，才会出现困惑度与模型拟合度之间严格单调递减的关系，证实了先前发现的负相关性。
• 模型容量并未显著改变困惑度与拟合度之间的关系；所有变体均遵循连续曲线，表明模型在学习预测超出人类水平期望时具有共享的归纳偏置。

Figure 2: Increase in regression model log-likelihood due to including each surprisal estimate from newly-trained LM variants as a function of training steps (top) and perplexity (middle; the stars indicate the fully trained versions after 10,000 steps), as well as perplexity as a function of traini