[论文解读] Evaluating the Complementarity of Taxonomic Relation Extraction Methods Across Different Languages

Roger Granada, Renata Vieira+2 more|arXiv (Cornell University)|Nov 8, 2018

Natural Language Processing Techniques参考文献 75被引用 4

一句话总结

本文在英语和葡萄牙语语料库上评估了七种最先进的分类关系抽取方法，比较了它们在精确率、召回率和互补性方面的表现。研究发现，分布方法虽然召回率高但精确率低，而基于模式的方法虽然精确率高但召回率低，将方法结合——尤其是模式与文档蕴含结合——可产生互补结果，从而提升整体性能。

ABSTRACT

Modern information systems are changing the idea of "data processing" to the idea of "concept processing", meaning that instead of processing words, such systems process semantic concepts which carry meaning and share contexts with other concepts. Ontology is commonly used as a structure that captures the knowledge about a certain area via providing concepts and relations between them. Traditionally, concept hierarchies have been built manually by knowledge engineers or domain experts. However, the manual construction of a concept hierarchy suffers from several limitations such as its coverage and the enormous costs of its extension and maintenance. Ontology learning, usually referred to the (semi-)automatic support in ontology development, is usually divided into steps, going from concepts identification, passing through hierarchy and non-hierarchy relations detection and, seldom, axiom extraction. It is reasonable to say that among these steps the current frontier is in the establishment of concept hierarchies, since this is the backbone of ontologies and, therefore, a good concept hierarchy is already a valuable resource for many ontology applications. The automatic construction of concept hierarchies from texts is a complex task and much work have been proposing approaches to better extract relations between concepts. These different proposals have never been contrasted against each other on the same set of data and across different languages. Such comparison is important to see whether they are complementary or incremental. Also, we can see whether they present different tendencies towards recall and precision. This paper evaluates these different methods on the basis of hierarchy metrics such as density and depth, and evaluation metrics such as Recall and Precision. Results shed light over the comprehensive set of methods according to the literature in the area.

研究动机与目标

• 在不同语言的相同数据上比较多种分类关系抽取方法的性能。
• 评估方法在关系预测中是否具有互补性或冗余性。
• 通过深度、宽度和密度等指标，分析自动生成本体结构的特性。
• 评估方法选择对多语言语料库中精确率、召回率和F1值的影响。
• 探索结合方法以提升整体抽取质量的潜力。

提出的方法

• 实现了七种分类关系抽取方法：分布方法（如TF、DF、SLQS、DSim）、基于模式的方法（Patt）、文档蕴含方法（DocSub）和层次聚类方法（HClust）。
• 在平行和可比的英语与葡萄牙语语料库上，使用与标准答案的关系进行自动评估，对方法进行评估。
• 通过计算精确率、召回率和F1值来评估方法质量，结果按语言进行分析。
• 通过计算深度、宽度和密度等层次指标，表征自动生成本体的结构特性。
• 通过计算不同方法间的关系集合的交集与并集，分析互补性，识别重叠和唯一预测。
• 对本体应用传递约减，以在归一化后更好地评估深度和结构清晰度。

实验结果

研究问题

• RQ1 在英语和葡萄牙语语料库中，哪种方法在精确率和召回率方面表现最佳？
• RQ2 相同的方法在不同语言中是否表现一致，还是语言特异性因素会影响性能？
• RQ3 不同方法生成的本体在深度和宽度方面在结构上是否相似，还是存在显著差异？
• RQ4 不同方法的结果在多大程度上具有互补性，哪些组合能取得最佳性能？
• RQ5 方法特有的偏差（如高精确率与高召回率）如何影响整体质量与互补性？

主要发现

• 分布方法（如TF、DF、SLQS、DSim）召回率很高（通常超过70%），但精确率较低（低于40%），表明存在大量误报。
• 基于模式的方法（Patt）精确率很高（超过80%），但召回率极低（低于20%），原因是文本中模式数量有限。
• 分布方法生成的本体显著更深（最大路径长度通常 >1）且更窄，而DocSub和HClust生成的本体更宽但不那么深。
• 基于文档蕴含的方法（DocSub）和层次聚类方法（HClust）与基于模式的方法（Patt）表现出强互补性，尤其在组合后能显著提升F1值。
• DSim方法生成了大量逆向关系（如“car is-a vehicle”和“vehicle is-a car”），表明需要进行归一化或过滤。
• 应用传递约减后发现，分布方法生成的本体结构更复杂、更深，而DocSub和HClust生成的本体结构更简单、更宽泛。