[Paper Review] Quality and Efficiency of Manual Annotation: Pre-annotation Bias

Marie Mikulová, Milan Straka+3 more|arXiv (Cornell University)|Jun 15, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper evaluates the impact of automatic pre-annotation on the quality and efficiency of manual dependency syntax annotation using the Prague Dependency Treebank - Consolidated. It finds that pre-annotation significantly improves annotation speed and consistency without reducing quality, especially when combined with rule-based checks and access to prior annotations.

ABSTRACT

This paper presents an analysis of annotation using an automatic pre-annotation for a mid-level annotation complexity task -- dependency syntax annotation. It compares the annotation efforts made by annotators using a pre-annotated version (with a high-accuracy parser) and those made by fully manual annotation. The aim of the experiment is to judge the final annotation quality when pre-annotation is used. In addition, it evaluates the effect of automatic linguistically-based (rule-formulated) checks and another annotation on the same data available to the annotators, and their influence on annotation quality and efficiency. The experiment confirmed that the pre-annotation is an efficient tool for faster manual syntactic annotation which increases the consistency of the resulting annotation without reducing its quality.

Motivation & Objective

• To assess whether pre-annotation compromises annotation quality compared to fully manual annotation.
• To evaluate the influence of additional tools—rule-based checks and access to prior annotations—on annotation quality and efficiency.
• To identify the optimal configuration of annotation support tools for large-scale syntactic annotation projects.
• To inform the design of annotation workflows for the 2 million-token PDT-C project and similar initiatives.
• To provide evidence-based recommendations for balancing speed, consistency, and accuracy in manual syntactic annotation.

Proposed method

• Conducted a controlled experiment comparing four annotation conditions: fully manual, pre-annotated, pre-annotated with rule-based checks, and pre-annotated with access to prior annotations.
• Used a high-accuracy dependency parser to generate automatic pre-annotations for the same data across all conditions.
• Implemented real-time, linguistically motivated rule-based checks to validate manual edits during annotation.
• Provided annotators with access to existing annotations on the same data to support consistency and error detection.
• Measured performance using standard metrics: UAS (Unlabeled Attachment Score), LAS (Labeled Attachment Score), and Cohen’s Kappa for inter-annotator agreement.
• Designed the experiment to minimize confounding effects such as learning curves and annotator bias through balanced data presentation and randomized assignment.

Experimental results

Research questions

• RQ1 Does the quality of manual annotation remain acceptably high when using pre-annotation, or does it suffer due to over-reliance on automatic suggestions?
• RQ2 How do additional tools—specifically rule-based checks and access to prior annotations—affect annotation quality and efficiency?
• RQ3 Which combination of annotation support tools yields the best trade-off between speed, consistency, and accuracy?
• RQ4 To what extent does pre-annotation reduce inter-annotator disagreement compared to fully manual annotation?
• RQ5 Can pre-annotation be reliably used in large-scale syntactic annotation projects without degrading final output quality?

Key findings

• Pre-annotation significantly increases annotation efficiency by reducing the time required per sentence without compromising final quality.
• The use of pre-annotation leads to higher inter-annotator agreement, indicating improved consistency across annotators.
• Rule-based checks and access to prior annotations individually had limited positive impact on quality and efficiency.
• The combination of pre-annotation with both rule-based checks and access to prior annotations produced the highest quality results, with measurable improvements in UAS and LAS scores.
• Annotators who used pre-annotation were less likely to introduce errors, suggesting that pre-annotation acts as a cognitive scaffold that supports accurate decision-making.
• The final annotation quality achieved with pre-annotation was comparable to or better than that of fully manual annotation, especially in terms of consistency and agreement metrics.