[Paper Review] Superiority of the $h$-index over the Impact Factor for Physics

Casey W. Miller|ArXiv.org|Aug 17, 2006

Big Data Technologies and Applications17 citations

TL;DR

This paper argues that the h-index is a superior metric to the journal Impact Factor (IF) for evaluating physics journals, as it better reflects peer-reviewed research quality and reduces biases from multidisciplinary and review-focused journals. Using citation data from physics articles in multidisciplinary journals like Nature and Science, the study shows that Physical Review Letters ranks higher under the h-index than these top-tier journals, aligning with physicists' perception of prestige.

ABSTRACT

Focusing specifically on physics periodicals, I show that the journal Impact Factor is not correlated with Hirsch's $h$-index. This implies that the Impact Factor is not a good measure of research quality or influence because the $h$-index is a reflection of peer review, and thus a strong indicator of research quality. The impact gap between multidisciplinary journals and physics-only journals is significantly reduced when $h$ is used instead of the Impact Factor. Additionally, the impact of journals specializing in review articles is inherently deflated using $h$ because of the limited number of annual publications in such periodicals. Finally, a reordering of the top ranking journals occurs with $h$ when only the physics articles of multidisciplinary journals are considered, falling more in line with the average physicist's interpretation of a journal's prestige.

Motivation & Objective

• To challenge the validity of the journal Impact Factor (IF) as a measure of research quality in physics.
• To evaluate whether the h-index provides a more accurate and peer-review-aligned indicator of journal influence in physics.
• To quantify and compare the impact of physics-only journals versus multidisciplinary journals using the h-index and a new variant, the ℏ-index.
• To resolve discrepancies in journal rankings by filtering physics-specific content from multidisciplinary journals like Nature and Science.
• To demonstrate that the h-index better reflects the true scientific influence of journals such as Physical Review Letters, which is historically regarded as the most prestigious physics journal.

Proposed method

• The h-index was calculated for journals by ranking articles by citation count and identifying the highest rank h where the top h articles each have at least h citations.
• The ℏ-index was introduced as a geometric variant of the h-index, defined as ℏ = √(α/2), where α is the total citation count, to better capture the influence of highly cited and moderately cited articles.
• A text-based filtering method was used to identify physics articles in multidisciplinary journals by scanning bibliographies for physics-specific reference strings (e.g., 'phys rev', 'j phys b') and excluding non-physics references.
• The filter was tuned by adjusting the percentage of physics references per article, and the h and ℏ indices were computed for subsets of articles meeting each threshold.
• Linear fits were applied to data points at 20–50% physics reference thresholds to extrapolate the h and ℏ indices for pure physics content in journals like Nature and Science.
• The study compared the resulting h and ℏ indices of journals such as Physical Review Letters, Nature, and Science to assess relative prestige and consistency with expert opinion.

Experimental results

Research questions

• RQ1 Is the journal Impact Factor a reliable indicator of research quality in physics, given its lack of public data and susceptibility to manipulation?
• RQ2 How does the h-index compare to the Impact Factor in ranking physics journals, particularly in terms of reflecting peer-reviewed quality and scientific influence?
• RQ3 To what extent are multidisciplinary journals like Nature and Science overrepresented in Impact Factor rankings due to non-physics content and review articles?
• RQ4 Can the h-index and ℏ-index provide a more accurate and consistent ranking of physics journals, especially when isolating physics-specific content from multidisciplinary journals?
• RQ5 Does the h-index better align with the expert perception of journal prestige, as seen in the historical standing of Physical Review Letters?

Key findings

• The journal Impact Factor is not correlated with the h-index in physics, indicating that IF fails to reflect true research quality or influence.
• The h-index for Physics Review Letters (h = 263) is higher than for Nature’s physics subset (h = 285 when extrapolated), confirming its superior standing in physics.
• The ℏ-index for Physical Review Letters (ℏ = 869) significantly exceeds that of Nature’s physics subset (ℏ = 499), further validating its higher scientific impact.
• The h-index reduces the inflated impact of review journals like Reviews of Modern Physics, which have high IFs due to low publication volume and high citation rates.
• The analysis shows that multidisciplinary journals like Nature and Science have disproportionately high IFs due to non-physics content, which skews their perceived influence in physics.
• The h-index for the physics subset of Nature, extrapolated to 100% physics content, yields h_phys = 285 and ℏ_phys = 499, demonstrating that physics-only journals are more influential than IF rankings suggest.