[论文解读] Imparting Interpretability to Word Embeddings.

Aykut Koç, İhsan Utlu+2 more|arXiv (Cornell University)|Jul 19, 2018

Topic Modeling被引用 4

一句话总结

本文提出一种方法，通过将语义相关的词语沿从罗热词典派生的预定义维度对齐，为词嵌入赋予可解释性，采用对嵌入目标函数的加法修改。该方法在不牺牲语义一致性的前提下，提升了维度级别的可解释性，经基准测试和人工评估验证。

ABSTRACT

As an ubiquitous method in natural language processing, word embeddings are extensively employed to map semantic properties of words into a dense vector representation. They capture semantic and syntactic relations among words but the vectors corresponding to the words are only meaningful relative to each other. Neither the vector nor its dimensions have any absolute, interpretable meaning. We introduce an additive modification to the objective function of the embedding learning algorithm that encourages the embedding vectors of words that are semantically related to a predefined concept to take larger values along a specified dimension, while leaving the original semantic learning mechanism mostly unaffected. In other words, we align words that are already determined to be related, along predefined concepts. Therefore, we impart interpretability to the word embedding by assigning meaning to its vector dimensions. The predefined concepts are derived from an external lexical resource, which in this paper is chosen as Roget's Thesaurus. We observe that alignment along the chosen concepts is not limited to words in the Thesaurus and extends to other related words as well. We quantify the extent of interpretability and assignment of meaning from our experimental results. Manual human evaluation results have also been presented to further verify that the proposed method increases interpretability. We also demonstrate the preservation of semantic coherence of the resulting vector space by using word-analogy and word-similarity tests. These tests show that the interpretability-imparted word embeddings that are obtained by the proposed framework do not sacrifice performances in common benchmark tests.

研究动机与目标

• 解决词嵌入中向量维度缺乏内在语义的问题。
• 通过与外部词汇资源对齐，使词嵌入维度承载可解释的语义概念。
• 在提升可解释性的同时，保持词嵌入空间的语义一致性和性能。
• 证明可解释性可扩展至词汇资源之外的语义相关、未登录词。
• 通过定量基准测试和人工评估验证该方法。

提出的方法

• 在词嵌入学习的目标函数中引入加法修改，以促使与预定义概念相关的词语在特定维度上具有更高的值。
• 预定义概念源自罗热词典，该词典提供了结构化的词汇资源以实现概念对齐。
• 该方法选择性地修改优化目标，强调指定维度上的语义相似性，同时保留原始学习机制。
• 对齐过程在训练期间应用，使嵌入空间能够学习到对应于概念的可解释方向。
• 该方法无需重新训练整个模型，仅通过添加正则化项修改目标函数即可实现。
• 通过标准自然语言处理基准测试和人工标注，对所得嵌入进行评估，以衡量其可解释性和语义质量。

实验结果

研究问题

• RQ1 能否通过外部词汇资源为词嵌入的各个维度赋予可解释的语义？
• RQ2 可解释性在多大程度上能从词汇资源中的显式词语扩展到语义相关的未登录词？
• RQ3 所提出的方法在标准基准测试中测量的词嵌入空间语义一致性是否得以保持？
• RQ4 与标准嵌入相比，人工评估者如何评价修改后嵌入的可解释性？
• RQ5 能否在不降低词类比和相似性任务性能的前提下增强可解释性？

主要发现

• 该方法通过将罗热词典中预定义概念相关的词语对齐，成功为词嵌入维度赋予了可解释的语义。
• 可解释性不仅限于词典中的词语，还扩展到语义相关的未登录词，表明对齐具有泛化能力。
• 词类比和词相似性测试表明，嵌入空间的语义一致性得以保持，性能与标准词嵌入相当。
• 人工评估证实，与标准嵌入相比，修改后的嵌入被感知为显著更具可解释性。
• 对目标函数的加法修改有效提升了可解释性，且未损害所学向量空间的质量。
• 定量结果表明，可在保持标准自然语言处理基准测试最先进性能的同时，系统性地为词嵌入赋予可解释性。