[Paper Review] TnT - A Statistical Part-of-Speech Tagger

Thorsten Brants|ArXiv.org|Mar 13, 2000

Natural Language Processing TechniquesComputer Science11 references325 citations

TL;DR

TnT presents a highly accurate, efficient part-of-speech tagger based on second-order Markov models with linear interpolation smoothing and robust unknown word handling. It achieves state-of-the-art accuracy—96.7% on the Penn Treebank—surpassing or matching Maximum Entropy models, demonstrating that simple Markov models with careful implementation choices outperform more complex frameworks.

ABSTRACT

Trigrams'n'Tags (TnT) is an efficient statistical part-of-speech tagger. Contrary to claims found elsewhere in the literature, we argue that a tagger based on Markov models performs at least as well as other current approaches, including the Maximum Entropy framework. A recent comparison has even shown that TnT performs significantly better for the tested corpora. We describe the basic model of TnT, the techniques used for smoothing and for handling unknown words. Furthermore, we present evaluations on two corpora.

Motivation & Objective

• To develop a part-of-speech tagger that achieves state-of-the-art accuracy using a simple, efficient Markov model framework.
• To challenge the prevailing belief that Maximum Entropy models outperform statistical taggers based on Markov models.
• To demonstrate that careful implementation choices—especially in smoothing, unknown word handling, and boundary treatment—significantly impact performance.
• To provide a detailed, reproducible account of techniques that are often omitted in prior literature, enabling high accuracy with minimal complexity.
• To show that a tagger based on second-order Markov models can outperform or match more complex models like Maximum Entropy, especially when optimized with proper smoothing and handling strategies.

Proposed method

• Uses second-order Markov models (trigram HMMs) where transition probabilities depend on the previous two tags and emission probabilities depend only on the current tag.
• Employs linear interpolation smoothing across unigram, bigram, and trigram probabilities with fixed, context-independent weights (λ₁, λ₂, λ₃) summing to 1.
• Applies a context-independent smoothing technique where the same λ values are used across all trigram contexts, improving generalization over context-dependent variants.
• Handles unknown words by assigning them the most likely tag based on the most frequent tag of words with the same suffix or capitalization pattern, using a heuristic rule set.
• Uses start-of-sentence and end-of-sentence markers (t₋₁, t₀, tₜ₊₁) to improve tagging accuracy, especially when sentence boundaries are detected via punctuation.
• Estimates probabilities via maximum likelihood from training corpora, with zero probabilities assigned only when both numerator and denominator are zero.

Experimental results

Research questions

• RQ1 Can a second-order Markov model tagger achieve performance comparable to or better than Maximum Entropy models in part-of-speech tagging?
• RQ2 How do implementation details—such as smoothing technique, unknown word handling, and boundary marker usage—affect tagging accuracy?
• RQ3 Does linear interpolation with fixed smoothing weights outperform context-dependent smoothing in trigram models?
• RQ4 To what extent does training data size influence tagging accuracy, particularly for known versus unknown words?
• RQ5 Can probability scores from the tagger be used to distinguish reliable from unreliable tag assignments?

Key findings

• TnT achieves 96.7% accuracy on the Penn Treebank, slightly exceeding the 96.6% reported for Maximum Entropy models in the same evaluation setup.
• For known words, tagging accuracy reaches 95–96% even with only 1,000 training tokens, demonstrating fast learning on seen vocabulary.
• Accuracy for unknown words is significantly lower (89.0% on German data), but improves to 97.7% for known words, highlighting the importance of lexical exposure.
• Reliable tag assignments—those with high probability scores—achieve over 99% accuracy, while unreliable ones are much less accurate, enabling selective post-processing.
• Linear interpolation with fixed smoothing weights yields better results than context-dependent smoothing, contrary to common intuition.
• The tagger is not only accurate but also the fastest among tested systems in both training and tagging phases, making it efficient for large-scale applications.