[Paper Review] Selective Sampling for Example-based Word Sense Disambiguation

Atsushi Fujii, Kentaro Inui+2 more|ArXiv.org|Oct 23, 1999

Natural Language Processing Techniques50 references114 citations

TL;DR

This paper proposes a selective sampling method for example-based word sense disambiguation that reduces both supervision and search overhead by selecting examples with maximum training utility—defined as the informativeness of an example for future sampling. Experiments on ~1,000 sentences show the method achieves performance comparable to baseline methods while significantly reducing the number of required manually labeled examples and database search costs.

ABSTRACT

This paper proposes an efficient example sampling method for example-based word sense disambiguation systems. To construct a database of practical size, a considerable overhead for manual sense disambiguation (overhead for supervision) is required. In addition, the time complexity of searching a large-sized database poses a considerable problem (overhead for search). To counter these problems, our method selectively samples a smaller-sized effective subset from a given example set for use in word sense disambiguation. Our method is characterized by the reliance on the notion of training utility: the degree to which each example is informative for future example sampling when used for the training of the system. The system progressively collects examples by selecting those with greatest utility. The paper reports the effectiveness of our method through experiments on about one thousand sentences. Compared to experiments with other example sampling methods, our method reduced both the overhead for supervision and the overhead for search, without the degeneration of the performance of the system.

Motivation & Objective

• To address the high supervision cost and computational search overhead in large-scale example-based word sense disambiguation systems.
• To develop a method that selects a smaller, more effective subset of examples from a large corpus for training.
• To improve system efficiency by prioritizing examples that maximize future training utility.
• To evaluate the method against random, uncertainty, and committee-based sampling strategies in terms of performance and efficiency.

Proposed method

• The method introduces a formal notion of 'training utility'—the degree to which an example is informative for future sampling when used in training.
• It selects examples with the highest training utility at each iteration, based on how many unsupervised examples they can help disambiguate.
• The system alternates between word sense disambiguation (using nearest neighbor similarity based on a thesaurus) and a training phase where human experts label selected examples.
• The training utility is computed using a utility function that considers the overlap between the case fillers of the example and those of unsupervised examples.
• The process iteratively builds a compact, high-utility database by selecting examples that reduce uncertainty across the largest number of remaining examples.
• The method is applied to a verb sense disambiguation system using a corpus of Japanese sentences, with similarity measured via case filler overlap and a thesaurus.

Experimental results

Research questions

• RQ1 How can example selection be optimized to reduce the number of manually labeled examples required in example-based WSD?
• RQ2 To what extent does training utility correlate with actual performance improvement in word sense disambiguation?
• RQ3 How does the proposed method compare to random, uncertainty, and committee-based sampling in terms of efficiency and accuracy?
• RQ4 Can a single-model approach like this achieve performance comparable to multi-model approaches such as committee-based sampling?

Key findings

• The proposed training utility-based sampling method reduced both supervision and search overhead more effectively than random, uncertainty, and committee-based sampling methods.
• The method achieved performance comparable to baseline methods, maintaining high disambiguation accuracy without degradation despite using fewer labeled examples.
• Uncertainty sampling was found to consider redundancy (property b) but not the broader influence on unsupervised examples (property a), limiting its effectiveness.
• Committee-based sampling was deemed less suitable for example-based systems due to its design for statistics-based models, unlike the proposed method.
• The utility function successfully prioritized examples that were most informative for future disambiguation, leading to faster convergence with fewer labeled examples.