[Paper Review] SemEval-2021 Task 11: NLPCONTRIBUTIONGRAPH - Structuring Scholarly NLP Contributions for a Research Knowledge Graph

Jennifer D’Souza, Sören Auer+1 more|arXiv (Cornell University)|Jan 1, 2021

Topic Modeling23 references1 citations

TL;DR

This paper introduces SemEval-2021 Task 11, NLPCONTRIBUTIONGRAPH (NCG), a novel shared task that structures scholarly NLP contributions into knowledge graph (KG) triples across three granularities: contribution sentences, scientific terms and relations, and subject-predicate-object triples. The best end-to-end system achieved 57.27% F1 on sentence classification, 46.41% F1 on phrase extraction, and 22.28% F1 on triple generation, highlighting the task's difficulty despite strong subtask performance with gold inputs.

ABSTRACT

There is currently a gap between the natural language expression of scholarly publications and their structured semantic content modeling to enable intelligent content search. With the volume of research growing exponentially every year, a search feature operating over semantically structured content is compelling. The SemEval-2021 Shared Task NLPContributionGraph (a.k.a. ‘the NCG task’) tasks participants to develop automated systems that structure contributions from NLP scholarly articles in the English language. Being the first-of-its-kind in the SemEval series, the task released structured data from NLP scholarly articles at three levels of information granularity, i.e. at sentence-level, phrase-level, and phrases organized as triples toward Knowledge Graph (KG) building. The sentence-level annotations comprised the few sentences about the article’s contribution. The phrase-level annotations were scientific term and predicate phrases from the contribution sentences. Finally, the triples constituted the research overview KG. For the Shared Task, participating systems were then expected to automatically classify contribution sentences, extract scientific terms and relations from the sentences, and organize them as KG triples. Overall, the task drew a strong participation demographic of seven teams and 27 participants. The best end-to-end task system classified contribution sentences at 57.27% F1, phrases at 46.41% F1, and triples at 22.28% F1. While the absolute performance to generate triples remains low, as conclusion to the article, the difficulty of producing such data and as a consequence of modeling it is highlighted.

Motivation & Objective

• To develop automated systems that structure unstructured scholarly NLP contributions into machine-readable, knowledge graph-ready formats.
• To address the growing challenge of information overload in scholarly literature by enabling fine-grained, semantic search through structured research content.
• To create a standardized annotation scheme—NLPCONTRIBUTIONGRAPH (NCG)—for capturing research contributions at sentence, phrase, and triple levels.
• To support the construction of a comprehensive, interconnected research knowledge graph (ORKG) by formalizing contribution extraction as an automated NLP task.
• To evaluate the feasibility and difficulty of mining structured research contributions from scholarly articles using NLP techniques.

Proposed method

• The task released a dataset of 299 NLP scholarly articles annotated at three granularities: contribution sentences, scientific terms and relations, and subject-predicate-object triples.
• An annotation scheme was developed through pilot and adjudication phases, defining 12 information units (IUs) such as RESEARCHPROBLEM, MODEL, RESULTS, and EXPERIMENTALSETUP to organize triples.
• Systems were evaluated in two phases: end-to-end pipeline using predicted sentences, and a gold-standard phase using manually labeled sentences to isolate IU classification performance.
• Participants employed diverse methods, including BERT-based models for classification and sequence labeling, dependency parsing, and hybrid approaches combining heuristics and machine learning.
• The NCG dataset was made publicly available, and a sample knowledge graph was published via the ORKG digital library.

Experimental results

Research questions

• RQ1 How effectively can NLP systems automatically identify and classify contribution sentences in scholarly NLP articles?
• RQ2 To what extent can systems extract scientific terms and relational predicates from contribution sentences with high precision and recall?
• RQ3 What is the performance of end-to-end systems in generating structured knowledge graph triples from unstructured scholarly text?
• RQ4 How does performance vary when systems are given gold-standard sentences versus predicted ones in the pipeline?
• RQ5 What are the key challenges and limitations in creating a standardized, multi-granular annotation scheme for research contributions?

Key findings

• The best end-to-end system achieved 57.27% F1 on contribution sentence classification, 46.41% F1 on phrase extraction, and 22.28% F1 on triple generation, indicating significant room for improvement in end-to-end knowledge graph construction.
• When provided with gold sentences, the best system achieved 71.72% F1 on information unit classification, demonstrating that sentence classification is a major bottleneck in the pipeline.
• A strong baseline system that predicted only the three mandatory IUs (RESEARCHPROBLEM, MODEL, RESULTS) achieved 69.01% F1, outperforming several complex models, suggesting that simple heuristics can be effective.
• The INNOVATORS team’s system showed no improvement with gold sentences, indicating it relied on a fixed heuristic not dependent on input sentence quality.
• The dataset’s non-uniform task distribution—especially in the training set, where 58.62% of tasks had fewer than five articles—likely contributed to low generalization and poor performance on low-resource tasks.