[Paper Review] Open data to evaluate academic researchers: an experiment with the Italian Scientific Habilitation

Angelo Di Iorio, Silvio Peroni+1 more|arXiv (Cornell University)|Feb 8, 2019

scientometrics and bibliometrics research17 citations

TL;DR

This paper evaluates the feasibility of replacing commercial citation databases with open data in academic evaluation by simulating Italy's national professor qualification process (Scientific Habilitation). Using only open citation data, the study finds current open citation coverage is insufficient to replicate official results, highlighting a critical gap in open scholarly infrastructure despite growing initiatives like I4OC.

ABSTRACT

The need for scholarly open data is ever increasing. While there are large repositories of open access articles and free publication indexes, there are still a few examples of free citation networks and their coverage is partial. One of the results is that most of the evaluation processes based on citation counts rely on commercial citation databases. Things are changing under the pressure of the Initiative for Open Citations (I4OC), whose goal is to campaign for scholarly publishers to make their citations as totally open. This paper investigates the growth of open citations with an experiment on the Italian Scientific Habilitation, the National process for University Professor qualification which instead uses data from commercial indexes. We simulated the procedure by only using open data and explored similarities and differences with the official results. The outcomes of the experiment show that the amount of open citation data currently available is not yet enough for obtaining similar results.

Motivation & Objective

• To assess whether open citation data can reliably replace commercial citation databases in national academic evaluation processes.
• To investigate the feasibility of using open data in high-stakes academic evaluations, such as professorship qualification.
• To compare results from open data simulations with official evaluations based on commercial indexes.
• To identify gaps in current open citation coverage and their impact on evaluation outcomes.
• To support the broader goal of open scholarly infrastructure by empirically testing open data in a real-world evaluation context.

Proposed method

• The authors simulated the Italian Scientific Habilitation process using only open citation data from sources like Crossref and OpenCitations.
• They collected publication and citation data for candidates from open repositories and citation networks.
• The simulation replicated the official evaluation procedure, including citation count aggregation and normalization.
• Results from the open data simulation were compared with the official results from the National Scientific Habilitation process.
• The study used standard bibliometric indicators such as citation counts and h-index, computed exclusively from open data.
• Coverage and completeness of open citation data were analyzed to assess their impact on evaluation outcomes.

Experimental results

Research questions

• RQ1 Can open citation data produce results comparable to those from commercial citation databases in a national academic evaluation?
• RQ2 What is the current level of completeness and coverage of open citation data for use in high-stakes academic evaluations?
• RQ3 How do differences in citation data sources affect the ranking and qualification of academic candidates?
• RQ4 To what extent does the lack of comprehensive open citation data hinder the adoption of open evaluation systems?
• RQ5 What are the practical implications of relying on open data for national academic qualification processes?

Key findings

• The simulation using only open citation data failed to reproduce the official results of the Italian Scientific Habilitation process.
• Current open citation coverage is insufficient to ensure reliable or comparable outcomes in large-scale academic evaluations.
• Significant gaps exist in the availability and completeness of open citation data, especially for older publications and certain disciplines.
• The study confirms that while open citation initiatives like I4OC are progressing, they have not yet reached critical mass for replacing commercial databases.
• The discrepancy between official and open-data-based evaluations underscores the need for improved open citation infrastructure.
• The results highlight the dependency of academic evaluation on commercial data and the risks of relying on incomplete open alternatives.