[论文解读] Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Cristina-Iulia Bucur, Tobias Kuhn+1 more|arXiv (Cornell University)|Oct 8, 2019

Scientific Computing and Data Management被引用 6

一句话总结

本文提出了 Linkflows，一种用于科学出版中细粒度、相互链接的同行评审评论的语义模型，通过正式链接和元数据实现对评审者意图的精确表示。评估结果表明，在评审时捕捉语义可减少与真实情况的分歧，相较于事后提取方法；更大的评审群体通过‘群体智慧’效应实现更高的一致性。

ABSTRACT

Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determine if a fine-grained model of the scientific publishing workflow can help us make the reviewing processes better organized and more accurate, by ensuring that review comments are created with formal links and semantics from the start. Our contributions include a novel model called Linkflows that allows for such detailed and semantically rich representations of reviews and the reviewing processes. We evaluate our approach on a manually curated dataset from several recent Computer Science journals and conferences that come with open peer reviews. We gathered ground-truth data by contacting the original reviewers and asking them to categorize their own review comments according to our model. Comparing this ground truth to answers provided by model experts, peers, and automated techniques confirms that our approach of formally capturing the reviewers' intentions from the start prevents substantial discrepancies compared to when this information is later extracted from the plain-text comments. In general, our analysis shows that our model is well understood and easy to apply, and it revealed the semantic properties of such review comments.

研究动机与目标

• 解决科学出版中传统同行评审存在的效率低下、不一致和不透明问题。
• 通过在撰写时捕捉评论的语义意图，减少对评审评论解释的差异。
• 评估通过语义元数据形式化评审评论是否能提升下游评估中的准确性和一致性。
• 探索在重构评审意图方面，同行群体或自动化工具是否优于专家。
• 证明在万维网上以链接化、语义增强的数据形式表示同行评审的可行性和优势。

提出的方法

• 设计 Linkflows 模型，通过正式链接关联到文章特定部分，并附加语义元数据（如情感、可操作性、影响程度）来表示评审评论。
• 从计算机科学期刊和会议的开放同行评审中构建人工标注的数据集，并由原始评审者提供真实标注。
• 采用多主体评估框架，将真实情况与模型专家、同行及自动化情感分析工具的响应进行比较。
• 应用统计检验（Wilcoxon signed-rank test）评估各组之间一致程度差异的显著性。
• 通过将同行划分为更小单位并比较分歧得分，检验‘群体智慧’假说。
• 利用语义网络原则，包括 HTTP URI 和溯源追踪，确保评审数据可机器处理且可审计。

实验结果

研究问题

• RQ1 与事后分析相比，细粒度、语义增强的同行评审评论模型是否能减少解释上的差异？
• RQ2 在重构评审意图方面，模型专家与同行及自动化工具的一致性如何比较？
• RQ3 在聚合同行判断时，‘群体智慧’效应是否能提高一致性？
• RQ4 自动化情感分析工具与真实标注在多大程度上保持一致？
• RQ5 Linkflows 模型能否有效应用于计算机科学领域的实际开放同行评审？

主要发现

• 事后重构评审评论的语义会导致与真实情况的显著分歧，凸显了在源头捕捉意图的必要性。
• 模型专家在解读评审评论方面并未显著优于同行，表明专业知识本身并不能保证更高准确性。
• 较大的同行群体与真实情况的分歧低于较小群体，证实了‘群体智慧’效应的存在。
• 三人一组的同行群体平均分歧高于整个同行群体，支持在更大样本中存在集体智能。
• 自动化情感分析工具的表现与模型专家相当，但仍与真实情况存在显著分歧。
• Linkflows 模型能够实现对评审评论的精确、机器可处理的表示，减少歧义并提升科学评估的可追溯性。