[Paper Review] Automated Annotation with Generative AI Requires Validation

Nicholas Pangakis, Samuel Wolken+1 more|arXiv (Cornell University)|May 31, 2023

Computational and Text Analysis MethodsSocial Sciences34 citations

TL;DR

The paper argues that automated annotation with LLMs must be validated against human labels, proposes a five-step workflow, and demonstrates varied LLM performance across 27 tasks, with overall promising results but task-dependent reliability.

ABSTRACT

Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures, but their performance varies across annotation tasks due to prompt quality, text data idiosyncrasies, and conceptual difficulty. Because these challenges will persist even as LLM technology improves, we argue that any automated annotation process using an LLM must validate the LLM's performance against labels generated by humans. To this end, we outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. Using GPT-4, we validate this approach by replicating 27 annotation tasks across 11 datasets from recent social science articles in high-impact journals. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis. We make available easy-to-use software designed to implement our workflow and streamline the deployment of LLMs for automated annotation.

Motivation & Objective

• Demonstrate that LLM-based annotation requires task-specific validation against expert human labels.
• Develop a principled, efficient workflow to augment text annotation with LLMs while foregrounding human judgment.
• Evaluate LLM performance across diverse, non-public social science datasets to identify use-cases and limitations.
• Provide open-source software to implement the workflow and measure annotation consistency.

Proposed method

• Propose a five-step workflow for LLM-augmented annotation with a task-specific codebook.
• Have two subject-matter experts and an LLM annotate a shared subset of data using the same codebook.
• Measure performance (accuracy, precision, recall, F1) against human labels on held-out samples.
• Iteratively update the codebook in a human-in-the-loop manner to address misclassifications.
• Test LLM performance on remaining human-labeled samples to determine viability for automated annotation.

Figure 2: Precision and recall for each of 27 replicated classification tasks. Color reflects dataset, such that points sharing the same color are conducted on the same text data.

Experimental results

Research questions

• RQ1 Can LLMs achieve reliable text annotation across diverse social science tasks when validated against expert labels?
• RQ2 How does codebook (prompt) refinement affect LLM classification performance?
• RQ3 What are practical use cases for integrating LLMs into annotation workflows given variable task performance?
• RQ4 What metrics and concepts (consistency scores) aid in identifying edge cases and ensuring quality?

Key findings

• LLM annotation achieves a median F1 of 0.707 across 27 tasks, but performance is highly task-specific.
• Nine of 27 tasks had precision or recall below 0.5, indicating substantial edge cases and non-generalizable results.
• Consistency scores correlate with accuracy and true positive/negative rates, helping identify difficult samples.
• Codebook updates yield modest improvements; prompt engineering alone is not a strong lever on performance.
• Four use cases enable flexible integration of LLMs depending on performance and resource constraints.
• The workflow and accompanying open-source tools streamline implementing LLM-augmented annotation with human-in-the-loop validation.

Figure 3: Relationship between consistency score and accuracy, TPR, and TNR.