[Paper Review] Automatic Error Detection in Part of Speech Tagging

David Elworthy|ArXiv.org|Oct 21, 1994

Natural Language Processing Techniques1 references3 citations

TL;DR

This paper presents a threshold-based method for detecting errors in part-of-speech (POS) tagging using Hidden Markov Models (HMMs). By comparing observable tagging probabilities against a dynamically chosen threshold, the system identifies low-confidence predictions, enabling error detection without retraining. The approach improves overall accuracy by selectively rejecting uncertain tags, with empirical results showing significant gains in precision at the cost of reduced tagging efficiency.

ABSTRACT

A technique for detecting errors made by Hidden Markov Model taggers is described, based on comparing observable values of the tagging process with a threshold. The resulting approach allows the accuracy of the tagger to be improved by accepting a lower efficiency, defined as the proportion of words which are tagged. Empirical observations are presented which demonstrate the validity of the technique and suggest how to choose an appropriate threshold.

Motivation & Objective

• To address the problem of undetected errors in HMM-based part-of-speech tagging systems.
• To develop a method that identifies low-confidence tagging decisions without requiring external linguistic knowledge or retraining.
• To improve overall tagging accuracy by selectively rejecting uncertain predictions through a thresholding mechanism.
• To empirically validate the effectiveness of the error detection technique on real-world text data.
• To provide guidance on selecting an optimal threshold value for balancing accuracy and tagging efficiency.

Proposed method

• The method evaluates the observable probability of each word's predicted POS tag during HMM tagging.
• It compares each tag's probability against a predefined threshold to flag low-confidence predictions as potential errors.
• The threshold is selected based on empirical analysis of tagging confidence distributions across a test corpus.
• Tags falling below the threshold are rejected, and the system reports them as error candidates.
• The approach does not modify the HMM model parameters but operates as a post-processing filter on tagging output.
• The method is evaluated by measuring precision improvement when only high-confidence tags are retained.

Experimental results

Research questions

• RQ1 Can a threshold-based method effectively detect errors in HMM-based part-of-speech tagging?
• RQ2 How does the choice of threshold affect the trade-off between tagging accuracy and efficiency?
• RQ3 What is the optimal threshold value for maximizing precision while maintaining acceptable coverage?
• RQ4 Can this error detection method be applied without retraining or external linguistic resources?
• RQ5 How do empirical confidence distributions inform threshold selection in practice?

Key findings

• The threshold-based error detection method successfully identifies a significant proportion of incorrect POS tags by focusing on low-confidence predictions.
• By rejecting tags below a carefully chosen threshold, the system achieves higher precision, even though overall tagging efficiency decreases.
• Empirical results show that the method improves accuracy when only high-confidence tags are accepted, demonstrating the value of confidence filtering.
• The optimal threshold is not fixed and depends on the distribution of tagging probabilities in the test data.
• The technique is effective without modifying the underlying HMM model, making it a lightweight, post-processing solution.
• The method provides a practical way to enhance HMM tagger reliability by leveraging internal probability estimates.