[论文解读] Solving Verbal Comprehension Questions in IQ Test by Knowledge-Powered Word Embedding

Huazheng Wang, Fei Tian+3 more|arXiv (Cornell University)|May 29, 2015

Topic Modeling参考文献 38被引用 20

一句话总结

本文提出了一种知识增强型词嵌入框架，通过联合建模词义与语义关系（如同义、反义）来解决智商测试中的语言理解问题。通过利用从词典中获取的关系知识训练多义词嵌入，并结合类型特定的求解器，该方法在平均表现上超越了基线模型与人类工作者，达到当前最优水平。

ABSTRACT

Intelligence Quotient (IQ) Test is a set of standardized questions designed to evaluate human intelligence. Verbal comprehension questions appear very frequently in IQ tests, which measure human's verbal ability including the understanding of the words with multiple senses, the synonyms and antonyms, and the analogies among words. In this work, we explore whether such tests can be solved automatically by artificial intelligence technologies, especially the deep learning technologies that are recently developed and successfully applied in a number of fields. However, we found that the task was quite challenging, and simply applying existing technologies (e.g., word embedding) could not achieve a good performance, mainly due to the multiple senses of words and the complex relations among words. To tackle these challenges, we propose a novel framework consisting of three components. First, we build a classifier to recognize the specific type of a verbal question (e.g., analogy, classification, synonym, or antonym). Second, we obtain distributed representations of words and relations by leveraging a novel word embedding method that considers the multi-sense nature of words and the relational knowledge among words (or their senses) contained in dictionaries. Third, for each type of questions, we propose a specific solver based on the obtained distributed word representations and relation representations. Experimental results have shown that the proposed framework can not only outperform existing methods for solving verbal comprehension questions but also exceed the average performance of the Amazon Mechanical Turk workers involved in the study. The results indicate that with appropriate uses of the deep learning technologies we might be a further step closer to the human intelligence.

研究动机与目标

• 探究深度学习是否能够自动解决需要理解多义性与复杂词间关系的智商测试语言理解问题。
• 解决标准词嵌入在捕捉多义词义与关系知识方面（尤其是罕见词义）的局限性。
• 开发一个统一框架，用于分类问题类型、学习词义对与关系的联合表示，并应用类型特定的求解器。
• 在语言智商题上的表现超越现有词嵌入方法与人类基线水平。
• 证明引入外部词汇知识（如词典）可增强对罕见与歧义词义的表征学习。

提出的方法

• 训练一个问题类型分类器以识别问题类别（如类比、同义、反义），从而支持类型特定的求解策略。
• 提出一种新型词嵌入模型，通过聚类上下文窗口并将之与词典中的词义示例关联，为每个词义对学习独立的向量表示。
• 将关系知识（如同义、反义）整合进嵌入目标函数，实现词义与关系表征的联合学习。
• 模型采用类似跳字模型的目标函数，并使用分层Softmax，但扩展以处理多义性与关系约束。
• 针对每类问题设计专用求解器——例如，类比求解器通过最小化问题中词义对与候选答案之间向量距离来工作。
• 该框架利用外部资源（如WordRep）获取关系知识，并通过无监督聚类在上下文中消歧词义。

实验结果

研究问题

• RQ1 深度学习模型能否有效解决需要理解多义性与复杂语义关系的智商测试语言理解问题？
• RQ2 从词汇资源中引入关系知识是否能提升词嵌入质量，尤其是对罕见词义？
• RQ3 结合问题分类、多义表征学习与类型特定求解器的统一框架能否超越现有方法与人类表现？
• RQ4 词义与关系表征的联合学习在类比、同义与反义任务上的表现提升程度如何？
• RQ5 所提方法是否具有泛化能力，可在无需大规模标注数据的情况下适用于不同类型的语言智商问题？

主要发现

• 所提出的RK模型在测试集上整体准确率达到50.86%，显著优于人类基线（46.23%）。
• RK在整体准确率上较人类表现基线（HP）提升4.63个百分点，表现出更优性能。
• 在同义词问题上，模型准确率达到60.78%；在分类任务上达到52.83%，显著优于HP及其他基线模型。
• 关系知识的整合使性能优于仅使用单义或多元词嵌入的模型，表明关系归纳偏置有助于提升罕见词义的表征学习。
• 在所有问题类型上，该方法均优于GloVe、SG-1/SG-2以及MS-1/MS-2/MS-3，且在同义与分类任务上具有统计显著优势。
• 问题分类器准确率达到93.1%，有效实现问题到对应求解器的路由。