[Paper Review] MEGAnno+: A Human-LLM Collaborative Annotation System

Hannah Kim, Kushan Mitra+3 more|arXiv (Cornell University)|Feb 28, 2024

Natural Language Processing Techniques4 citations

TL;DR

MEGAnno+ is a human-LLM collaborative annotation system that enables efficient, high-quality data labeling by integrating LLMs as annotators with selective human verification. It supports LLM agent management, robust LLM prompting, and exploratory human review of low-confidence predictions, significantly improving labeling efficiency and reliability over human-only or LLM-only approaches.

ABSTRACT

Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.

Motivation & Objective

• To address the limitations of LLMs in understanding complex, sociocultural, or domain-specific contexts during data annotation.
• To reduce labeling costs and time while maintaining high-quality annotations by combining LLM speed with human expertise.
• To design a scalable annotation system that supports effective management of LLM agents, labels, and metadata.
• To enable selective, exploratory human verification of LLM-generated labels to correct errors and biases.
• To provide a practical framework for integrating LLMs as annotators within human-in-the-loop data labeling pipelines.

Proposed method

• The system employs a human-LLM collaborative workflow: LLMs generate labels first, followed by targeted human verification of low-confidence predictions.
• It supports customizable LLM agents with configurable models and temperature settings, enabling users to test and select optimal configurations.
• MEGAnno+ provides a unified backend for storing and managing LLM models, annotations, and metadata such as confidence scores and logprobs.
• The system includes a flexible search and recommendation interface to guide human annotators toward potentially problematic LLM outputs.
• It enables real-time progress tracking during large-scale LLM annotation jobs, improving workflow transparency.
• The platform supports prompt templating and in-context learning with data-aware prompt engineering to improve consistency and performance.

Experimental results

Research questions

• RQ1 How can LLMs be effectively integrated into a human-in-the-loop annotation pipeline to improve labeling efficiency and quality?
• RQ2 What system-level components are necessary to manage LLM agents, their outputs, and associated metadata in a collaborative annotation environment?
• RQ3 How can human verification be selectively targeted to LLM-generated labels with low confidence to maximize cost-effectiveness?
• RQ4 To what extent do prompt design and model configuration (e.g., temperature) affect the reliability of LLM annotations?
• RQ5 What are the practical challenges and risks of using commercial LLMs for labeling sensitive or domain-specific data?

Key findings

• LLMs can reduce labeling costs by up to 96% compared to human-only annotation, as demonstrated in prior work and supported by MEGAnno+’s workflow.
• LLM-generated labels can outperform human-annotated labels in certain classification tasks, especially when combined with human verification of low-confidence predictions.
• The system enables users to compare multiple LLM configurations (e.g., temperature=0 vs. temperature=0.7) and select the most reliable model for a given task.
• Human verification focused on low-confidence predictions (e.g., <95%) significantly improves label quality with minimal human effort.
• MEGAnno+ supports real-time progress tracking during large-scale LLM annotation jobs, enhancing workflow transparency and usability.
• The system’s post-processing and metadata handling are limited by the LLM’s native capabilities, such as the absence of logprobs in GPT-4, which affects uncertainty estimation.