[Paper Review] A maturity model for catalogues of semantic artefacts

Óscar Corcho, Fajar J. Ekaputra+5 more|arXiv (Cornell University)|May 11, 2023

Big Data and Business Intelligence33 references4 citations

TL;DR

This paper proposes a 12-dimensional maturity model to assess catalogues of semantic artefacts—such as ontologies, vocabularies, and metadata schemas—focusing on governance, interoperability, and long-term sustainability. The model, validated through expert assessment of 26 real-world catalogues, includes 43 measurable features across dimensions like metadata, openness, PID, and transparency, offering a standardized framework to evaluate and improve semantic artefact catalogues for FAIR and Open Science compliance.

ABSTRACT

This work presents a maturity model for assessing catalogues of semantic artefacts, one of the keystones that permit semantic interoperability of systems. We defined the dimensions and related features to include in the maturity model by analysing the current literature and existing catalogues of semantic artefacts provided by experts. In addition, we assessed 26 different catalogues to demonstrate the effectiveness of the maturity model, which includes 12 different dimensions (Metadata, Openness, Quality, Availability, Statistics, PID, Governance, Community, Sustainability, Technology, Transparency, and Assessment) and 43 related features (or sub-criteria) associated with these dimensions. Such a maturity model is one of the first attempts to provide recommendations for governance and processes for preserving and maintaining semantic artefacts and helps assess/address interoperability challenges.

Motivation & Objective

• To address the lack of standardized evaluation frameworks for catalogues of semantic artefacts in scholarly and research infrastructures.
• To identify key dimensions and features that determine the quality, sustainability, and interoperability of semantic artefact catalogues.
• To support the governance and long-term preservation of semantic artefacts by providing a systematic assessment model.
• To enable benchmarking and improvement of existing catalogues across diverse domains and technical stacks.
• To contribute to the implementation of FAIR principles and the European Open Science Cloud (EOSC) interoperability goals.

Proposed method

• The maturity model was developed through a multi-phase expert review process involving 12 reviewers from diverse institutions and domains.
• Initial assessment of 2–3 catalogues per reviewer identified 63 potential features across 12 dimensions, which were iteratively refined and harmonized.
• Features were validated and refined through iterative discussions, merging, and removal of redundant or irrelevant items, resulting in 43 final features.
• The model was applied to evaluate 26 real-world catalogues, including systems like OntoPortal, OLS, and SKOSMOS, representing diverse technologies and application domains.
• A structured spreadsheet with 12 tabs (one per dimension) was used to document features, assign reviewers, and record assessments with comments.
• Data and the maturity model itself were published in multiple formats (PDF, CSV, XSLX) on Zenodo for transparency and reuse.

Experimental results

Research questions

• RQ1 What dimensions and features are most critical for assessing the maturity of semantic artefact catalogues?
• RQ2 How can a standardized maturity model improve the governance, sustainability, and interoperability of semantic artefact catalogues?
• RQ3 To what extent do existing catalogues meet the criteria defined by the maturity model across key dimensions like openness, transparency, and PID support?
• RQ4 How can expert consensus be achieved in defining a shared evaluation framework for semantic artefact catalogues across diverse technical and disciplinary contexts?
• RQ5 What role does the maturity model play in supporting the FAIR principles and the European Open Science Cloud (EOSC) interoperability framework?

Key findings

• The maturity model identifies 12 core dimensions—Metadata, Openness, Quality, Availability, Statistics, PID, Governance, Community, Sustainability, Technology, Transparency, and Assessment—each with specific, measurable features.
• Only 15% of the 26 assessed catalogues stored a copy of the semantic artefacts they described, highlighting a critical gap in long-term preservation.
• Just 4 out of 26 catalogues used open-source, generic software for deployment, indicating limited reusability and transparency in underlying technologies.
• The model’s 43 final features were derived from an initial pool of 63, after expert review and harmonization, ensuring clarity and consistency across assessments.
• The maturity model was successfully applied to diverse catalogues across domains such as digital humanities, life sciences, and social sciences, demonstrating broad applicability.
• All data, including the model and assessment results, were made publicly available on Zenodo in multiple formats, ensuring reproducibility and reuse.