[论文解读] Doc2Vec on the PubMed corpus: study of a new approach to generate related articles

Emeric Dynomant, Stéfan Darmoni+6 more|arXiv (Cornell University)|Nov 26, 2019

Topic Modeling参考文献 18被引用 7

一句话总结

本研究评估了Doc2Vec，特别是PV-DBOW架构，作为PubMed的统计PubMed相关文章（pmra）模型的替代方案。通过在PubMed摘要上对六个超参数进行网格搜索以优化性能，该方法在无需预先索引的情况下学习文档嵌入，实现了与pmra相当的基于MeSH的相似性——尽管人工评估显示其感知准确性较低，表明在评估者意见分歧和模型可解释性方面仍需进一步研究。

ABSTRACT

PubMed is the biggest and most used bibliographic database worldwide, hosting more than 26M biomedical publications. One of its useful features is the "similar articles" section, allowing the end-user to find scientific articles linked to the consulted document in term of context. The aim of this study is to analyze whether it is possible to replace the statistic model PubMed Related Articles (pmra) with a document embedding method. Doc2Vec algorithm was used to train models allowing to vectorize documents. Six of its parameters were optimised by following a grid-search strategy to train more than 1,900 models. Parameters combination leading to the best accuracy was used to train models on abstracts from the PubMed database. Four evaluations tasks were defined to determine what does or does not influence the proximity between documents for both Doc2Vec and pmra. The two different Doc2Vec architectures have different abilities to link documents about a common context. The terminological indexing, words and stems contents of linked documents are highly similar between pmra and Doc2Vec PV-DBOW architecture. These algorithms are also more likely to bring closer documents having a similar size. In contrary, the manual evaluation shows much better results for the pmra algorithm. While the pmra algorithm links documents by explicitly using terminological indexing in its formula, Doc2Vec does not need a prior indexing. It can infer relations between documents sharing a similar indexing, without any knowledge about them, particularly regarding the PV-DBOW architecture. In contrary, the human evaluation, without any clear agreement between evaluators, implies future studies to better understand this difference between PV-DBOW and pmra algorithm.

研究动机与目标

• 探究Doc2Vec是否可作为无需依赖人工MeSH索引的替代方案，用于生成相关文章，替代PubMed的pmra模型。
• 评估文档长度、词汇内容及术语相似性对Doc2Vec与pmra模型相似性评分的影响。
• 通过自动化与人工评估方法，评估Doc2Vec PV-DBOW与PV-DM相对于pmra的性能表现。
• 理解基于MeSH的自动化评估与人工对相关文章相关性评估之间存在差异的原因。

提出的方法

• 在PubMed摘要上训练超过1,900个Doc2Vec模型，通过六组超参数的网格搜索以优化性能。
• 采用PV-DBOW与PV-DM架构生成文档向量，其中PV-DBOW仅通过文档向量预测词语。
• 通过四项任务评估模型性能：MeSH术语重叠、词汇/词干相似性、文档长度相关性以及人工相关性评估。
• 基于MeSH术语重叠优化模型参数，假设摘要在语义上可代表全文内容。
• 通过四位标注员进行人工评估，以判断各模型返回的前10篇相关文章的相关性。
• 将Doc2Vec的余弦相似度得分与pmra的统计评分进行比较，后者结合了MeSH术语与精英主题频率。

实验结果

研究问题

• RQ1 Doc2Vec PV-DBOW能否在无需人工MeSH索引的情况下，实现与PubMed pmra模型相当的识别相关生物医学文章的性能？
• RQ2 文档长度、共享词汇与共享词干如何影响Doc2Vec与pmra模型的相似性评分？
• RQ3 PV-DBOW架构在捕捉基于MeSH的语义相似性方面，与pmra相比表现如何？
• RQ4 尽管基于MeSH的性能相似，为何人工评估显示Doc2Vec PV-DBOW的感知准确性显著低于pmra？

主要发现

• Doc2Vec的PV-DBOW架构实现了与pmra相当的基于MeSH的相似性，证明语义关系可在无需预先索引的情况下被学习。
• Doc2Vec PV-DBOW与MeSH术语重叠具有强相关性，并且更倾向于将长度相近的文档归为一类，与pmra表现一致。
• 人工评估显示pmra优于Doc2Vec PV-DBOW，pmra的平均排名为第7位，而PV-DBOW为第14位。
• 人工评估中的标注者一致性仅为中等水平，表明共识度较低，对人工评估在此类任务中的可靠性提出质疑。
• 尽管未进行人工索引，Doc2Vec PV-DBOW仍能仅基于摘要与标题成功推断文档间的关系，表明其具备强大的语义学习能力。
• 本研究揭示了基于MeSH的自动化评估与人工判断之间存在脱节，提示需进一步探究评估方法与模型可解释性。