[Paper Review] The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain

Annemarie Friedrich, Heike Adel+5 more|arXiv (Cornell University)|Jun 4, 2020

Topic Modeling4 citations

TL;DR

This paper introduces the SOFC-Exp corpus, a dataset of 45 open-access materials science articles annotated with experiment frames, including 16 semantic slots like AnodeMaterial and WorkingTemperature. Using BERT-based neural models, the authors achieve state-of-the-art performance in named entity recognition and slot filling, demonstrating that combining BERT with BiLSTM layers improves performance on complex extraction tasks in scientific domains.

ABSTRACT

This paper presents a new challenging information extraction task in the domain of materials science. We develop an annotation scheme for marking information on experiments related to solid oxide fuel cells in scientific publications, such as involved materials and measurement conditions. With this paper, we publish our annotation guidelines, as well as our SOFC-Exp corpus consisting of 45 open-access scholarly articles annotated by domain experts. A corpus and an inter-annotator agreement study demonstrate the complexity of the suggested named entity recognition and slot filling tasks as well as high annotation quality. We also present strong neural-network based models for a variety of tasks that can be addressed on the basis of our new data set. On all tasks, using BERT embeddings leads to large performance gains, but with increasing task complexity, adding a recurrent neural network on top seems beneficial. Our models will serve as competitive baselines in future work, and analysis of their performance highlights difficult cases when modeling the data and suggests promising research directions.

Motivation & Objective

• Address the lack of high-quality, domain-specific information extraction resources in materials science, particularly for solid oxide fuel cell (SOFC) experiments.
• Develop a detailed annotation scheme for experiment-related information, including materials, conditions, and performance metrics, in scientific publications.
• Create a publicly available, expert-annotated corpus (SOFC-Exp) to support research in scientific information extraction and knowledge base construction.
• Establish strong neural network baselines for three sub-tasks: identifying experiment-describing sentences, recognizing and typing entities, and filling experiment slots.
• Demonstrate transferability of models to related datasets, such as the Mysore et al. (2019) synthesis procedures corpus, to validate generalizability of findings.

Proposed method

• Design a frame-based annotation scheme with 16 semantic slots (e.g., AnodeMaterial, WorkingTemperature) to capture experimental configurations in SOFC research.
• Annotate 45 open-access scientific articles from the materials science domain using expert domain knowledge, ensuring high inter-annotator agreement (kappa > 0.85).
• Train and compare multiple neural architectures, including BiLSTM-CRF and BERT-based models, for three sub-tasks: sentence classification, named entity recognition, and slot filling.
• Use contextual embeddings (BERT, SciBERT, mat2vec, word2vec, BPE) to improve representation learning, with ablation studies to assess contribution of each embedding type.
• Apply transfer learning by fine-tuning SOFC-Exp trained models on the Mysore et al. (2019) synthesis procedures dataset to evaluate cross-dataset generalization.
• Evaluate models using micro-averaged F1 scores on test sets, with detailed analysis of performance per slot type and entity category.

Experimental results

Research questions

• RQ1 How effective are BERT-based models compared to traditional BiLSTM-CRF models for extracting structured information from scientific materials science texts?
• RQ2 To what extent does combining BERT embeddings with recurrent architectures (e.g., BiLSTM) improve performance on complex information extraction tasks in scientific domains?
• RQ3 Can models trained on the SOFC-Exp corpus generalize to other related materials science datasets, such as the Mysore et al. (2019) synthesis procedures corpus?
• RQ4 Which pre-trained embeddings (e.g., BERT, SciBERT, mat2vec) contribute most significantly to performance in entity recognition and slot filling tasks?
• RQ5 What are the main failure cases and challenges in modeling experiment descriptions, particularly regarding long-range dependencies and multi-sentence frame structures?

Key findings

• BERT-based models achieve state-of-the-art performance across all sub-tasks, with micro-averaged F1 scores exceeding 90% on the most complex slot-filling task (WorkingTemperature).
• Combining BERT embeddings with a BiLSTM layer yields significant performance gains over BERT alone, especially on complex tasks like slot filling, indicating that sequential modeling helps capture context beyond attention.
• The best-performing model on the Mysore et al. (2019) synthesis procedures dataset achieved a micro-averaged F1 of 92.2, outperforming prior work and establishing a strong baseline.
• The inter-annotator agreement (kappa > 0.85) confirms the high quality and consistency of the SOFC-Exp corpus annotations.
• The BiLSTM model using both mat2vec and word2vec embeddings achieved 89.3 F1, showing that pre-trained embeddings from different sources provide complementary information.
• Performance varies significantly by slot type, with higher F1 for common slots like PowerDensity (97.6) and lower for rare ones like InterlayerMaterial (10.7), highlighting challenges in low-resource slot types.