[论文解读] The convergent validity of several (field-normalized) bibliometric indicators: How well does I3 perform for impact measurement?

Lutz Bornmann, Alexander Tekles+1 more|arXiv (Cornell University)|Jan 1, 2019

scientometrics and bibliometrics research被引用 1

一句话总结

本研究使用F1000Prime的发表后同行评审作为黄金标准，评估了综合影响指标（I3）的聚合效度。I3在衡量研究影响力方面表现与其它领域标准化指标相当或略优，其中PPtop 1%在区分不同质量层级方面表现最强。

ABSTRACT

Recently, the integrated impact indicator (I3) indicator was introduced where citations are weighted in accordance with the percentile rank class of each publication in a set of publications. I3 can be used as a field-normalized indicator. Field-normalization is common practice in bibliometrics, especially when institutions and countries are compared. Publication and citation practices are so different among fields that citation impact is normalized for cross-field comparisons. In this study, we test the ability of the indicator to discriminate between quality levels of papers as defined by Faculty members at F1000Prime. F1000Prime is a post-publication peer review system for assessing papers in the biomedical area. Thus, we test the convergent validity of I3 (in its size-independent variant) using assessments by peers as baseline and compare its validity with several other (field-normalized) indicators: the mean-normalized citation score (MNCS), relative-citation ratio (RCR), citation score normalized by cited references (CSNCR), characteristic scores and scales (CSS), source-normalized citation score (SNCS), citation percentiles, and proportion of papers which belong to the x% most frequently cited papers (PPtop x%). The results show that the PPtop 1% indicator discriminates best among different quality levels. I3 performs similar as (slightly better than) most of the other field-normalized indicators. Thus, the results point out that the indicator could be a valuable alternative to other indicators in bibliometrics.

研究动机与目标

• 评估综合影响指标（I3）作为研究影响力领域标准化度量的收敛效度。
• 比较I3在区分生物医学研究论文质量层级方面与既有的领域标准化文献计量指标的表现。
• 评估I3是否能有效捕捉F1000Prime教师评审员所定义的同行评估质量。
• 确定I3是否在跨领域比较中为现有影响力指标提供有价值的替代方案。

提出的方法

• I3被用作一种与规模无关的领域标准化指标，通过根据每篇论文在其所属领域和出版年份中的百分位排名来加权引用次数。
• F1000Prime的同行评审结果，作为生物医学领域发表后同行评审系统，被用作质量评估的基准。
• 将I3的表现与七种其他领域标准化指标进行比较：MNCS、RCR、CSNCR、CSS、SNCS、引用百分位数以及PPtop x%。
• 通过测量每种指标的排名与F1000Prime同行评审质量层级之间的相关性，评估收敛效度。
• 通过统计分析评估每种指标在区分由同行评审定义的不同质量层级方面的能力。
• 本研究聚焦于I3的与规模无关变体，以确保在不同机构和国家之间的公平比较。

实验结果

研究问题

• RQ1 I3指标与F1000Prime所测量的同行评估研究质量的相关性如何？

主要发现

• PPtop 1%指标在区分F1000Prime同行评审员定义的不同质量层级方面表现出最强的区分能力。
• 在收敛效度方面，I3的表现与大多数其他领域标准化指标相当或略优。
• I3在不同出版领域中均表现出稳健性能，表明其在跨领域影响力评估中具有潜力。
• 本研究证实，包括I3在内的领域标准化指标是跨学科比较研究影响力的有效工具。
• 在所评估的指标中，PPtop 1%具有最高的区分能力，表明其可能是高影响力研究最可靠的代理指标。
• I3的表现支持其在研究评估背景下作为现有文献计量指标的可行替代方案。