[Paper Review] Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue

Shiwei Zhang, Mingfang Wu+1 more|arXiv (Cornell University)|Oct 17, 2023

Research Data Management Practices4 citations

TL;DR

This paper proposes using GPT-3.5 with in-context learning prompts to automate subject metadata annotation for research datasets in an Australian national data catalogue. The approach achieves promising performance in subject classification but faces limitations in discipline-specific accuracy due to insufficient contextual information, marking the first application of in-context learning for automated metadata annotation in this domain.

ABSTRACT

In support of open and reproducible research, there has been a rapidly increasing number of datasets made available for research. As the availability of datasets increases, it becomes more important to have quality metadata for discovering and reusing them. Yet, it is a common issue that datasets often lack quality metadata due to limited resources for data curation. Meanwhile, technologies such as artificial intelligence and large language models (LLMs) are progressing rapidly. Recently, systems based on these technologies, such as ChatGPT, have demonstrated promising capabilities for certain data curation tasks. This paper proposes to leverage LLMs for cost-effective annotation of subject metadata through the LLM-based in-context learning. Our method employs GPT-3.5 with prompts designed for annotating subject metadata, demonstrating promising performance in automatic metadata annotation. However, models based on in-context learning cannot acquire discipline-specific rules, resulting in lower performance in several categories. This limitation arises from the limited contextual information available for subject inference. To the best of our knowledge, we are introducing, for the first time, an in-context learning method that harnesses large language models for automated subject metadata annotation.

Motivation & Objective

• To address the growing challenge of poor-quality metadata in research datasets due to limited curation resources.
• To investigate the feasibility of leveraging large language models (LLMs) for cost-effective, automated subject metadata annotation.
• To evaluate the performance of in-context learning with LLMs in classifying research dataset subjects within a national data catalogue context.
• To identify limitations of LLM-based annotation, particularly in capturing discipline-specific rules due to insufficient context.

Proposed method

• Employing GPT-3.5 as the foundational LLM for subject metadata annotation.
• Designing task-specific prompts for in-context learning to guide the model in assigning subject categories to datasets.
• Using few-shot prompting with exemplars to improve zero-shot generalization for subject classification.
• Applying the LLM to annotate subject metadata in a real-world Australian national research data catalogue.
• Evaluating model outputs against human-annotated ground truth to measure accuracy and identify failure cases.
• Analyzing performance variations across subject categories to assess domain-specific limitations.

Experimental results

Research questions

• RQ1 Can in-context learning with a large language model effectively automate subject metadata annotation for research datasets?
• RQ2 How does the performance of the LLM-based annotation method vary across different academic disciplines in the dataset catalogue?
• RQ3 What are the primary limitations of in-context learning in capturing discipline-specific metadata rules?
• RQ4 To what extent can LLMs reduce the manual curation burden for national research data repositories?
• RQ5 How does the quality of contextual information in prompts affect the accuracy of subject classification?

Key findings

• The LLM-based method achieved promising performance in subject metadata annotation, demonstrating strong generalization across diverse research domains.
• Performance was lower in specialized or niche academic disciplines due to insufficient contextual cues in prompts.
• The model struggled to learn discipline-specific rules, indicating a key limitation of in-context learning in complex metadata tasks.
• Subject categories with clearer linguistic patterns showed higher annotation accuracy compared to those with ambiguous or overlapping terminology.
• The approach significantly reduces manual curation effort, offering a scalable solution for large-scale metadata enrichment.
• The study presents the first known application of in-context learning with LLMs for automated subject metadata annotation in a national research data infrastructure.