[Paper Review] Quantifying Success in Science: An Overview

Xiaomei Bai, Hanxiao Pan+4 more|arXiv (Cornell University)|Aug 10, 2020

scientometrics and bibliometrics research110 references4 citations

TL;DR

This paper provides a comprehensive review of scholarly impact evaluation, categorizing and analyzing metrics for paper, scholar, and journal impact. It identifies key challenges such as collaboration impact patterns, implicit success factors, dynamic network modeling, and citation inflation, proposing solutions through unified standards, heterogeneous networks, and dynamic embedding techniques to improve scientific evaluation systems.

ABSTRACT

Quantifying success in science plays a key role in guiding funding allocations, recruitment decisions, and rewards. Recently, a significant amount of progresses have been made towards quantifying success in science. This lack of detailed analysis and summary continues a practical issue. The literature reports the factors influencing scholarly impact and evaluation methods and indices aimed at overcoming this crucial weakness. We focus on categorizing and reviewing the current development on evaluation indices of scholarly impact, including paper impact, scholar impact, and journal impact. Besides, we summarize the issues of existing evaluation methods and indices, investigate the open issues and challenges, and provide possible solutions, including the pattern of collaboration impact, unified evaluation standards, implicit success factor mining, dynamic academic network embedding, and scholarly impact inflation. This paper should help the researchers obtaining a broader understanding of quantifying success in science, and identifying some potential research directions.

Motivation & Objective

• To systematically categorize and review existing evaluation indices for scholarly impact across papers, scholars, and journals.
• To identify critical limitations in current evaluation methods, including bias toward academic age and citation inflation.
• To explore emerging challenges such as collaboration impact, implicit success factors, and dynamic academic networks.
• To propose solutions like unified evaluation standards, implicit factor mining, and dynamic network embedding for more accurate impact assessment.
• To guide future research by identifying open issues in scholarly impact quantification, especially in interdisciplinary and evolving academic contexts.

Proposed method

• Categorizes scholarly impact evaluation into three domains: paper impact, scholar impact, and journal impact using bibliometric and scientometric methods.
• Reviews counting-based methods (e.g., citations, h-index, JIF) and network-based algorithms (e.g., PageRank, HITS) for impact quantification.
• Proposes dynamic academic network embedding to model evolving scholarly relationships over time, improving representation of node importance.
• Introduces the concept of higher-order academic networks to mitigate citation inflation and enhance robustness of impact measures.
• Advocates for unified evaluation standards and ground-truth datasets to benchmark and compare different impact evaluation methods.
• Employs heterogeneous academic networks to integrate and quantify co-author collaboration impact, addressing limitations of traditional metrics.

Experimental results

Research questions

• RQ1 How can collaboration impact be systematically quantified in scholarly networks, and what patterns emerge over time?
• RQ2 What implicit success factors—beyond academic age and field—contribute to scholarly impact, and how can they be mined?
• RQ3 How can dynamic academic networks be effectively modeled to reflect evolving citation and co-authorship patterns?
• RQ4 To what extent does scholarly impact inflation distort cross-temporal comparisons of research output?
• RQ5 What unified standards and evaluation frameworks are needed to fairly compare different impact assessment methods?

Key findings

• Existing evaluation indices such as h-index and JIF are limited by bias toward academic age and susceptibility to citation manipulation.
• Network-based methods like PageRank and HITS improve impact assessment by modeling structural relationships in citation and co-authorship networks.
• Scholarly impact inflation, driven by exponential growth in publications, undermines the reliability of citation-based metrics over time.
• Dynamic academic network embedding is essential for capturing temporal changes in scholarly influence but remains underdeveloped.
• Implicit success factors—such as mentorship and learning habits—play a significant but underexplored role in academic success.
• Unified evaluation standards and higher-order network models are critical to resolving inconsistencies and improving fairness in scholarly impact assessment.