[Paper Review] MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling

Yu Song, Santiago Miret+1 more|arXiv (Cornell University)|May 14, 2023

Topic Modeling4 citations

TL;DR

This paper introduces MatSci-NLP, a benchmark for evaluating NLP models on materials science tasks using text-to-schema prompting. It demonstrates that domain-specific pretraining (e.g., MatBERT) and a novel Task-Schema input format inspired by question-answering significantly improve performance in low-resource settings, outperforming standard fine-tuning across seven materials science NLP tasks.

ABSTRACT

We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. We construct the benchmark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classification, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for materials. We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pretraining strategies on understanding materials science text. Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the generalize across MatSci-NLP tasks. Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text. MatBERT, a model pretrained specifically on materials science journals, generally performs best for most tasks. Moreover, we propose a unified text-to-schema for multitask learning on \benchmark and compare its performance with traditional fine-tuning methods. In our analysis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods. The code and datasets are publicly available at \url{https://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23}.

Motivation & Objective

• To address the lack of comprehensive, standardized benchmarks for evaluating NLP models on materials science text, especially in low-resource settings.
• To investigate how pretraining strategies—particularly domain-specific vs. general scientific vs. general language—impact downstream performance on materials science NLP tasks.
• To design and evaluate a unified text-to-schema framework for multitask learning that improves model generalization and performance across diverse materials science NLP tasks.
• To compare the effectiveness of conventional fine-tuning with novel schema-based prompting methods, especially in scenarios with limited annotated data.

Proposed method

• Constructed MatSci-NLP as a benchmark comprising seven distinct NLP tasks, including named entity recognition, relation classification, and synthesis action retrieval, using publicly available materials science text.
• Designed multiple textual input schemas inspired by question-answering to guide model input, including task-specific and multi-task formats, to improve learning efficiency.
• Proposed a new Task-Schema input format for joint multitask training that unifies inputs across tasks using structured, question-answering-style prompts.
• Fine-tuned various BERT-based models—ranging from general BERT to domain-specific models like MatBERT and SciBERT—on the MatSci-NLP benchmark under low-data conditions.
• Evaluated model performance using standard NLP metrics (e.g., F1, accuracy) across all tasks, comparing schema-based prompting against single-task and multitask fine-tuning.
• Utilized a unified encoder-decoder architecture for the proposed text-to-schema method, enabling modular adaptation across different domains and tasks.

Figure 1: Example of different question-answering inspired textual input schemas (Task-Schema , Potential Choices, Example) applied on MatSci-NLP. The input of the language model includes the shared text (green) along with relevant task details (blue for NER and orange for event extraction). The sha

Experimental results

Research questions

• RQ1 How does in-domain pretraining of language models affect their performance on materials science NLP tasks compared to general or scientific pretraining?
• RQ2 How do different textual input schemas—especially those inspired by question-answering—impact learning efficiency and model performance in low-resource settings?
• RQ3 Can a unified text-to-schema multitask framework outperform conventional single-task and multitask fine-tuning approaches across diverse materials science NLP tasks?
• RQ4 To what extent do domain-specific pretraining and structured input schemas improve generalization and robustness in materials science NLP applications?

Key findings

• MatBERT, a BERT model pre-trained specifically on materials science journals, consistently achieved the highest performance across most tasks in the MatSci-NLP benchmark.
• Language models pre-trained on scientific corpora (e.g., SciBERT) outperformed general-domain BERT, and in some cases even outperformed other domain-specific models like MatSciBERT and BatteryBERT.
• The proposed Task-Schema input format, inspired by question-answering, consistently outperformed both single-task and standard multitask fine-tuning across all evaluated models and tasks.
• Text-to-schema prompting significantly improved model generalization in low-resource settings, demonstrating that structured input design can be as impactful as model architecture or pretraining data.
• The study found that models fine-tuned with the Task-Schema method achieved higher F1 scores on average across all tasks compared to conventional fine-tuning, indicating improved learning efficiency.
• Despite limited data, the results suggest that careful schema design can mitigate memorization risks and enhance understanding beyond simple pattern matching.

Figure 2: Unified text-to-schema method for MatSci-NLP text understanding applied across the seven tasks. The language model includes a domain specific encoder, which can be exchanged in a modular manner, as well as a general language pretrained transformer decoder.