[Paper Review] How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations

Lutz Bornmann, Werner Marx|arXiv (Cornell University)|Feb 15, 2013

scientometrics and bibliometrics research84 references4 citations

TL;DR

This paper proposes a percentile-based citation evaluation framework for natural and life sciences researchers, replacing flawed metrics like the h-index with standardized percentiles of citations within scientific subfields. By normalizing citation impact relative to field-specific benchmarks, the method enables fairer, multidimensional comparisons of individual research performance across disciplines.

ABSTRACT

Although bibliometrics has been a separate research field for many years, there is still no uniformity in the way bibliometric analyses are applied to individual researchers. Therefore, this study aims to set up proposals how to evaluate individual researchers working in the natural and life sciences. 2005 saw the introduction of the h index, which gives information about a researcher's productivity and the impact of his or her publications in a single number (h is the number of publications with at least h citations); however, it is not possible to cover the multidimensional complexity of research performance and to undertake inter-personal comparisons with this number. This study therefore includes recommendations for a set of indicators to be used for evaluating researchers. Our proposals relate to the selection of data on which an evaluation is based, the analysis of the data and the presentation of the results.

Motivation & Objective

• Address the lack of standardization in bibliometric evaluations of individual researchers in the natural and life sciences.
• Overcome the limitations of single-number metrics like the h-index, which fail to capture the multidimensional nature of research performance.
• Develop a consistent, field-normalized method for evaluating researchers based on citation impact.
• Provide practical guidelines for data selection, analysis, and presentation of evaluation results to ensure transparency and fairness.
• Enable meaningful interpersonal comparisons across different scientific subfields by accounting for varying citation practices.

Proposed method

• Use percentiles of citations within discipline-specific subfields to normalize individual citation impact.
• Calculate the relative citation impact of a researcher’s publications by comparing them to the distribution of citations in their field and publication year.
• Apply field-normalized citation percentiles to publications, rather than relying on raw citation counts or aggregated indices.
• Combine multiple percentile-based indicators (e.g., top 10%, 25%, 50% percentile publications) to reflect different dimensions of research impact.
• Use citation data from databases like Web of Science or Scopus, filtered by scientific field and publication year to ensure accuracy.
• Present results using visual and tabular formats that highlight a researcher’s performance relative to peers in the same field and time period.

Experimental results

Research questions

• RQ1 How can citation impact be meaningfully compared across researchers in different scientific subfields with varying citation cultures?
• RQ2 What are the limitations of existing metrics like the h-index in capturing the multidimensional nature of research performance?
• RQ3 How can citation data be normalized to ensure fair evaluation of individual researchers across disciplines?
• RQ4 What set of indicators can provide a comprehensive and transparent evaluation of individual research impact?
• RQ5 How can citation percentiles be practically applied in institutional or funding evaluations of researchers?

Key findings

• Percentile-based citation metrics provide a more equitable and field-specific method for evaluating individual researchers compared to absolute citation counts or the h-index.
• The method enables valid comparisons across disciplines by accounting for differences in citation practices across scientific fields.
• Using citation percentiles allows for a multidimensional assessment of research impact, including top-performing publications and overall citation performance.
• The approach reduces bias from field-dependent citation rates and publication output, improving fairness in research evaluation.
• The framework supports transparent and reproducible evaluations when combined with standardized data sources and clear methodological guidelines.
• The study demonstrates that percentile-based metrics can effectively replace or supplement traditional metrics in research assessment contexts.