[论文解读] What Syntactic Structures block Dependencies in RNN Language Models?

Ethan Wilcox, Roger Lévy+1 more|arXiv (Cornell University)|May 24, 2019

Topic Modeling参考文献 18被引用 14

一句话总结

本文研究了RNN语言模型是否通过测试其在复杂句法结构中维持填充空缺-空缺依赖关系和性别一致预期的能力，来学习句法岛屿约束。通过受控实验，作者表明最先进的LSTM模型渐进式地学习了左分支岛屿和并列岛屿，但未能学习句子主语岛屿，显示出对句法层级结构的敏感性，而不仅仅是句法复杂性的敏感性。

ABSTRACT

Recurrent Neural Networks (RNNs) trained on a language modeling task have been shown to acquire a number of non-local grammatical dependencies with some success. Here, we provide new evidence that RNN language models are sensitive to hierarchical syntactic structure by investigating the filler--gap dependency and constraints on it, known as syntactic islands. Previous work is inconclusive about whether RNNs learn to attenuate their expectations for gaps in island constructions in particular or in any sufficiently complex syntactic environment. This paper gives new evidence for the former by providing control studies that have been lacking so far. We demonstrate that two state-of-the-art RNN models are are able to maintain the filler--gap dependency through unbounded sentential embeddings and are also sensitive to the hierarchical relationship between the filler and the gap. Next, we demonstrate that the models are able to maintain possessive pronoun gender expectations through island constructions---this control case rules out the possibility that island constructions block all information flow in these networks. We also evaluate three untested islands constraints: coordination islands, left branch islands, and sentential subject islands. Models are able to learn left branch islands and learn coordination islands gradiently, but fail to learn sentential subject islands. Through these controls and new tests, we provide evidence that model behavior is due to finer-grained expectations than gross syntactic complexity, but also that the models are conspicuously un-humanlike in some of their performance characteristics.

研究动机与目标

• 确定RNN语言模型是否学习句法岛屿约束，或仅对句法复杂性敏感。
• 测试模型是否能通过岛屿结构维持填充空缺-空缺依赖关系和性别一致预期。
• 评估模型对先前未测试的岛屿约束的敏感性：并列岛屿、左分支岛屿和句子主语岛屿。
• 区分RNN行为中的一般句法复杂性效应与细粒度句法结构敏感性。

提出的方法

• 设计了受控的句子对，包含填充空缺-空缺依赖关系和性别一致预期，仅在岛屿结构上有所不同。
• 使用两个最先进的LSTM语言模型（Google和Gulordava）测量岛屿条件与非岛屿条件之间的概率差异。
• 以wh-授权交互作用和性别预期衰减作为句法敏感性的指标。
• 在无界句子嵌入上进行实验，以测试通过复杂嵌套结构的依赖关系维持能力。
• 在四种岛屿类型上评估模型：wh-岛屿、复杂名词短语岛屿、附加语岛屿，以及三种新型岛屿（并列、左分支、句子主语）。
• 应用统计检验（例如，p < 0.01，p < 0.05）以评估模型在控制条件与岛屿条件之间响应差异的显著性。

实验结果

研究问题

• RQ1 RNN语言模型是否在句法岛屿中衰减对空缺的预期，还是这种效应仅源于一般句法复杂性？
• RQ2 RNN能否通过无界句子嵌入维持填充空缺-空缺依赖关系？
• RQ3 RNN是否对句法层级结构敏感，如通过岛屿约束所体现的那样，还是无法区分结构复杂性与非结构复杂性？
• RQ4 RNN是否以不同程度成功地学习左分支岛屿、并列岛屿和句子主语岛屿？

主要发现

• Google和Gulordava模型在无界句子嵌入中维持了填充空缺-空缺依赖关系，表明其具备强大的长距离依赖学习能力。
• 在左分支岛屿条件下，模型显示出wh-授权交互作用的显著降低（Google模型p < 0.001，Gulordava模型p < 0.05），表明其学习了左分支岛屿约束。
• 模型对并列岛屿表现出渐进式学习，显示出部分敏感性，但未完全衰减预期。
• 在任一模型中，句子主语岛屿均未观察到wh-授权或性别预期的显著降低，表明其未能学习该约束。
• 所有岛屿类型中，性别一致预期均保持完整，排除了信息流普遍受阻的可能性，支持了对细粒度句法结构的敏感性。
• 模型学习了七种岛屿约束中的五种（Gulordava模型为六种），表明其对句法层级结构具有选择性敏感性，而非对一般复杂性的敏感。