[Paper Review] Exploring automatic word sense disambiguation with decision lists and the Web

Eneko Agirre, David Martínez|ArXiv.org|Oct 17, 2000

Natural Language Processing Techniques18 references92 citations

TL;DR

This paper evaluates decision lists for word sense disambiguation using SemCor and DSO corpora, with additional Web-derived training data. It finds that while decision lists achieve ~0.7 precision on highly polysemous words in hand-tagged corpora, automatically acquired Web data fails to improve performance, and cross-corpus training is ineffective, indicating limitations in scaling supervised WSD beyond manually curated data.

ABSTRACT

The most effective paradigm for word sense disambiguation, supervised learning, seems to be stuck because of the knowledge acquisition bottleneck. In this paper we take an in-depth study of the performance of decision lists on two publicly available corpora and an additional corpus automatically acquired from the Web, using the fine-grained highly polysemous senses in WordNet. Decision lists are shown a versatile state-of-the-art technique. The experiments reveal, among other facts, that SemCor can be an acceptable (0.7 precision for polysemous words) starting point for an all-words system. The results on the DSO corpus show that for some highly polysemous words 0.7 precision seems to be the current state-of-the-art limit. On the other hand, independently constructed hand-tagged corpora are not mutually useful, and a corpus automatically acquired from the Web is shown to fail.

Motivation & Objective

• To assess the scalability of supervised word sense disambiguation using decision lists on fine-grained WordNet senses.
• To evaluate whether hand-tagged corpora like SemCor and DSO can support high-precision WSD systems.
• To investigate the feasibility of automatically acquiring training data from the Web to overcome the knowledge acquisition bottleneck.
• To determine the limits of cross-corpus training and the robustness of decision lists to noise and data quantity.

Proposed method

• Uses decision lists trained on hand-tagged corpora (SemCor and DSO) with features weighted by log-likelihood ratio to rank disambiguation candidates.
• Applies a feature set including local collocations, part-of-speech tags, lemmas, and semantic fields from WordNet to improve sense discrimination.
• Implements a Web data acquisition pipeline based on Mihalcea & Moldovan (1999), using WordNet synonyms and glosses to generate search queries and extract candidate examples.
• Evaluates performance via cross-corpus tagging, precision/covariance analysis, and learning curves to assess data sufficiency and noise tolerance.
• Tests coarser sense distinctions derived from WordNet to assess performance improvements under less granular labeling.

Experimental results

Research questions

• RQ1 Can decision lists achieve high precision on fine-grained WordNet senses using existing hand-tagged corpora like SemCor and DSO?
• RQ2 To what extent can training data from one hand-tagged corpus be transferred to another for word sense disambiguation?
• RQ3 Can automatically acquired Web data serve as a viable alternative to hand-tagged corpora for training decision list WSD systems?
• RQ4 What is the upper performance bound for decision list-based WSD when trained on manually tagged data with fine-grained WordNet senses?

Key findings

• Decision lists achieve a precision of 0.70 on highly polysemous words in the DSO corpus, suggesting this may be the current state-of-the-art limit for such systems.
• SemCor provides sufficient data for basic disambiguation, achieving 0.68 precision on general running text, though performance varies significantly by word and part of speech.
• Cross-corpora training between SemCor and DSO yields disappointing results, indicating that independently constructed hand-tagged corpora are not mutually useful due to incompatibilities in sense annotation.
• Automatically acquired Web data fails to improve performance, with results showing near-useless quality, likely due to noise and low reliability of retrieved examples.
• Coarser word senses derived from WordNet enable decision lists to reach 80% precision, suggesting that reducing sense granularity improves robustness and performance.
• The learning curve for DSO suggests an upper bound has been reached for systems trained on fine-grained WordNet senses with hand-tagged data, indicating diminishing returns from more data.