[Paper Review] AI on AI: Exploring the Utility of GPT as an Expert Annotator of AI Publications

Autumn Toney-Wails, Christian Schoeberl+1 more|arXiv (Cornell University)|Mar 14, 2024

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper evaluates GPT models as expert annotators for classifying AI research publications, using prompt engineering to achieve 94% accuracy—comparable to a fine-tuned SPECTER model (96%)—and demonstrates that GPT-labeled data can train a downstream classifier that outperforms the original arXiv-trained model by 9 percentage points, reaching 82% accuracy.

ABSTRACT

Identifying scientific publications that are within a dynamic field of research often requires costly annotation by subject-matter experts. Resources like widely-accepted classification criteria or field taxonomies are unavailable for a domain like artificial intelligence (AI), which spans emerging topics and technologies. We address these challenges by inferring a functional definition of AI research from existing expert labels, and then evaluating state-of-the-art chatbot models on the task of expert data annotation. Using the arXiv publication database as ground-truth, we experiment with prompt engineering for GPT chatbot models to identify an alternative, automated expert annotation pipeline that assigns AI labels with 94% accuracy. For comparison, we fine-tune SPECTER, a transformer language model pre-trained on scientific publications, that achieves 96% accuracy (only 2% higher than GPT) on classifying AI publications. Our results indicate that with effective prompt engineering, chatbots can be used as reliable data annotators even where subject-area expertise is required. To evaluate the utility of chatbot-annotated datasets on downstream classification tasks, we train a new classifier on GPT-labeled data and compare its performance to the arXiv-trained model. The classifier trained on GPT-labeled data outperforms the arXiv-trained model by nine percentage points, achieving 82% accuracy.

Motivation & Objective

• To address the challenge of identifying AI research in a rapidly evolving field lacking standardized definitions or taxonomies.
• To evaluate whether large language models like GPT can serve as reliable, automated expert annotators for scientific publication classification.
• To develop a prompt engineering framework that enhances GPT’s annotation consistency and accuracy for domain-specific tasks.
• To assess the utility of GPT-annotated data as training data for downstream classification models compared to expert-labeled baselines.
• To explore the feasibility of using crowd-sourced expert labels from arXiv as a functional, time-relevant ground truth for AI research classification.

Proposed method

• Constructed a ground-truth dataset, AI-arXiv, using author-assigned subject categories from arXiv to identify AI publications.
• Fine-tuned SPECTER, a transformer model pre-trained on scientific text, on AI-arXiv to create a high-accuracy baseline classifier (96% accuracy).
• Designed zero-shot prompting with personas (reader, researcher, subject-matter expert) to guide GPT-3.5-Turbo and GPT-4 in labeling AI publications.
• Incorporated clarity and uncertainty clauses in prompts to improve reasoning and reduce hallucination, enhancing consistency.
• Generated a new dataset, AI-GPT, using GPT-annotated labels to train a separate classifier for comparison with the AI-arXiv baseline.
• Evaluated model performance on a held-out test set of publications from the top 13 AI conferences to measure generalization and accuracy.

Experimental results

Research questions

• RQ1 Can GPT models achieve high-accuracy annotation of AI research publications without fine-tuning, using only prompt engineering?
• RQ2 How does the performance of GPT models compare to a fine-tuned SPECTER model on classifying AI publications using the same ground-truth dataset?
• RQ3 Does the choice of persona (e.g., researcher vs. subject-matter expert) in system prompts significantly affect GPT’s annotation accuracy?
• RQ4 Can GPT-labeled data be effectively used to train a downstream classifier that generalizes better than models trained on expert-labeled data?
• RQ5 To what extent do GPT models rely on memorization versus reasoning when classifying AI publications, especially in edge cases?

Key findings

• GPT-3.5-Turbo and GPT-4 achieved 94% accuracy in zero-shot classification of AI publications, closely matching the 96% accuracy of the fine-tuned SPECTER baseline.
• The AI-GPT classifier, trained on GPT-annotated data, outperformed the AI-arXiv classifier by 9 percentage points, achieving 82% accuracy on a test set of top AI conference publications.
• Prompt engineering that included clarity and uncertainty clauses significantly improved GPT-3.5-Turbo’s performance, making it comparable to GPT-4 despite lower cost.
• GPT-4 exhibited a median confidence probability of 0.95 for correctly labeling non-AI publications, suggesting potential misalignment between response probability and internal confidence.
• GPT models showed higher accuracy on open-source AI publications, indicating possible memorization of training data patterns rather than general reasoning.
• The study demonstrates that LLMs can serve as scalable, reliable annotators for scientific literature classification in domains lacking formal taxonomies.