[Paper Review] Tagger Evaluation Given Hierarchical Tag Sets

I. Dan Melamed, Philip Resnik|ArXiv.org|Aug 10, 2000

Advanced Text Analysis Techniques5 references4 citations

TL;DR

This paper proposes a probabilistic evaluation framework for tagging systems that handles multiple tags, multiple correct answers, and tree-structured is-a tag hierarchies by assigning uniform probabilities to sub-tags and computing scores as the sum of probabilities over correct tags. The key contribution is a generalized scoring method compatible with standard metrics like Cohen's kappa, enabling fair and nuanced evaluation of both human and automatic taggers on hierarchical gold standards.

ABSTRACT

We present methods for evaluating human and automatic taggers that extend current practice in three ways. First, we show how to evaluate taggers that assign multiple tags to each test instance, even if they do not assign probabilities. Second, we show how to accommodate a common property of manually constructed ``gold standards'' that are typically used for objective evaluation, namely that there is often more than one correct answer. Third, we show how to measure performance when the set of possible tags is tree-structured in an IS-A hierarchy. To illustrate how our methods can be used to measure inter-annotator agreement, we show how to compute the kappa coefficient over hierarchical tag sets.

Motivation & Objective

• To address limitations in standard evaluation metrics that treat all incorrect tags as equally wrong, even when they are semantically related.
• To enable fair evaluation of taggers that assign multiple tags or are non-probabilistic, by interpreting such outputs as uniform probability distributions.
• To resolve ambiguity in gold standards where multiple correct tags may coexist, by treating them disjunctively and summing probabilities.
• To extend evaluation to tree-structured tag sets (e.g., WordNet, hector) by modeling is-a relationships as probabilistic hierarchies with uniform downward branching.
• To support inter-annotator agreement measurement on hierarchical tag sets using the kappa coefficient, adjusted for tag specificity and probability distribution.

Proposed method

• Non-probabilistic taggers are interpreted as assigning uniform probabilities to all tags in their output set, with probability 1.0 for single-tag outputs.
• Multiple correct tags are handled by summing the algorithm’s assigned probabilities across all correct tags, enabling partial credit.
• Tree-structured tag sets are evaluated by recursively distributing the probability mass of each non-leaf tag equally among its direct children, under the maximum entropy principle.
• The scoring function computes the total probability assigned to any correct tag, using the formula: Score(A) = Σ Pr(correct_tag | context) over all correct tags.
• For inter-annotator agreement, each annotator’s tag is expanded into its leaf-level probabilities using uniform downward probabilities.
• The kappa coefficient is computed using the joint probability of agreement at the leaf level, with chance agreement Pr(E) calculated as the sum of squared leaf-level probabilities.

Experimental results

Research questions

• RQ1 How can tagging systems that output multiple non-probabilistic tags be fairly evaluated when exact match fails to capture partial correctness?
• RQ2 How should evaluation be adjusted when gold standards include multiple valid tags for a single instance?
• RQ3 How can performance be measured when the tag set is a tree-structured is-a hierarchy, especially when parent and child tags co-occur?
• RQ4 Can standard inter-annotator agreement statistics like Cohen’s kappa be meaningfully applied to hierarchical tag sets without inflating the importance of general tags?
• RQ5 What assumptions are necessary to generalize probabilistic evaluation to hierarchical tag sets, and how do they affect fairness and interpretability?

Key findings

• The proposed method assigns a score of 1.0 to a tagger that outputs the correct leaf tag, and 0.5 to one that outputs its parent, due to uniform probability distribution over children.
• A tagger that outputs both A.1 and A.2 for a word correctly tagged as A.1a receives a score of 0.25, reflecting the probability mass assigned to the correct sub-sense.
• When both annotators use the same non-leaf tag (e.g., A), the agreement probability Pr(A) is computed as the sum of products of leaf-level probabilities, yielding 0.25 for A.1a and 0.25 for A.2a.
• For a gold standard with multiple correct tags (e.g., A.1a and B.2), the score is the sum of probabilities assigned to each correct tag, e.g., 0.5 + 1/3 = 0.833 when the algorithm outputs A.1 and B.2.
• The kappa coefficient computed on leaf-level probabilities correctly accounts for chance agreement and avoids overestimating agreement when non-leaf tags are used.
• The method ensures that the importance of non-leaf tags is not inflated, as their contribution is proportionally distributed to the leaves, maintaining fairness in evaluation.