[论文解读] On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

Ethan Wilcox, Jon Gauthier+3 more|arXiv (Cornell University)|Jun 2, 2020

Topic Modeling参考文献 22被引用 107

一句话总结

论文评估多样化语言模型预测人类实时阅读行为的能力，发现较低的困惑度通常带来更好的心理测量预测能力，深层 Transformer 模型表现出色，而句法知识在困惑度之外几乎不增添预测价值。

ABSTRACT

Human reading behavior is tuned to the statistics of natural language: the time it takes human subjects to read a word can be predicted from estimates of the word's probability in context. However, it remains an open question what computational architecture best characterizes the expectations deployed in real time by humans that determine the behavioral signatures of reading. Here we test over two dozen models, independently manipulating computational architecture and training dataset size, on how well their next-word expectations predict human reading time behavior on naturalistic text corpora. We find that across model architectures and training dataset sizes the relationship between word log-probability and reading time is (near-)linear. We next evaluate how features of these models determine their psychometric predictive power, or ability to predict human reading behavior. In general, the better a model's next-word expectations, the better its psychometric predictive power. However, we find nontrivial differences across model architectures. For any given perplexity, deep Transformer models and n-gram models generally show superior psychometric predictive power over LSTM or structurally supervised neural models, especially for eye movement data. Finally, we compare models' psychometric predictive power to the depth of their syntactic knowledge, as measured by a battery of syntactic generalization tests developed using methods from controlled psycholinguistic experiments. Once perplexity is controlled for, we find no significant relationship between syntactic knowledge and predictive power. These results suggest that different approaches may be required to best model human real-time language comprehension behavior in naturalistic reading versus behavior for controlled linguistic materials designed for targeted probing of syntactic knowledge.

研究动机与目标

• 评估模型驱动的 surprisal 在自然语料库中对人类阅读时间的预测作用。
• 比较架构（LSTM、RNNG、Transformer、n-gram）及训练数据规模。
• 确定模型困惑度与心理测量预测能力之间的关系。
• 检验句法泛化是否在困惑度之外解释额外的方差。

提出的方法

• 在四个逐步增大的 BLLIP 语料库（XS、SM、MD、LG）上训练一组语言模型（LSTM、RNNG、Transformer GPT-2、5-gram）。
• 对于子词 Transformer 模型，使用 BPE 编码及词级变体来估计词概率。
• 通过在回归人类阅读量度对 surprisal 的单词级 Delta LogLik 来评估心理测量预测能力，同时控制长度和频率。
• 使用广义加法模型和线性回归，对 Dundee 眼动追踪、Brown 自整段阅读和 Natural Stories SPRT 的阅读时间数据进行评估。
• 以 34 项有针对性的句法测试（SG 分数）量化句法知识，并将其与预测能力相关联。
• 比较困惑度与预测能力，并分析体系结构特定效应。

实验结果

研究问题

• RQ1 在不同模型与训练数据下，词级 surprisal 与阅读时间之间的关系是否保持线性？
• RQ2 模型的困惑度如何与其预测人类阅读行为的能力相关？
• RQ3 架构与句法知识是否在困惑度之外解释了预测能力的额外方差？
• RQ4 在自然语料阅读数据与受控句法测试之间，预测能力是否存在差异？

主要发现

• 在不同架构和数据规模下， surprisal 与阅读时间呈现近似线性的关系。
• 更好的下一个词预测（更低困惑度）通常会提高心理测量预测能力（Delta LogLik），跨语料库一致。
• 深层 Transformer 模型展现出最强的心理测量预测能力；在某些情况下，n-gram 模型在某些方面的表现甚至超出仅以困惑度为基础的预期。
• 在控制困惑度后，句法知识（SG 分数）对预测能力的方差不显著解释。
• 在推动句法泛化的因素与推动自然语料阅读预测能力的因素之间存在分离。