[Paper Review] Learning Probabilistic Models of Word Sense Disambiguation

Ted Pedersen|ArXiv.org|Jul 26, 2007

Natural Language Processing TechniquesComputer Science71 references19 citations

TL;DR

This dissertation proposes supervised and unsupervised probabilistic models for word sense disambiguation (WSD) using corpus-based learning. It introduces a forward sequential search strategy combined with Akaike’s Information Criterion (AIC) for model selection, achieving high accuracy; the Naive Mix algorithm, a novel supervised method, performs competitively with state-of-the-art techniques. In unsupervised learning, it applies EM and Gibbs sampling to infer word senses from raw text using a Naive Bayes parametric form, with feature sets based on local context and frequency, enabling sense discovery without sense-tagged data.

ABSTRACT

This dissertation presents several new methods of supervised and unsupervised learning of word sense disambiguation models. The supervised methods focus on performing model searches through a space of probabilistic models, and the unsupervised methods rely on the use of Gibbs Sampling and the Expectation Maximization (EM) algorithm. In both the supervised and unsupervised case, the Naive Bayesian model is found to perform well. An explanation for this success is presented in terms of learning rates and bias-variance decompositions.

Motivation & Objective

• To develop robust, corpus-based probabilistic models for word sense disambiguation that learn from large text corpora without requiring deep world knowledge.
• To address the scarcity of sense-tagged training data by proposing unsupervised learning methods that infer word senses from raw, untagged text.
• To improve model selection in supervised WSD by introducing a forward sequential search strategy and information criteria, particularly AIC.
• To design feature sets effective for unsupervised learning that capture local context and frequency patterns without relying on part-of-speech tags.
• To evaluate unsupervised sense clusters by mapping them to a standard sense inventory using post-processing and sense descriptions derived from cluster content.

Proposed method

• Employs a forward sequential search strategy to navigate the space of possible probabilistic models, selecting those that maximize predictive accuracy using information criteria.
• Uses Akaike’s Information Criterion (AIC) as an evaluation metric to balance model fit and complexity, avoiding overfitting in supervised learning.
• Introduces the Naive Mix algorithm, a supervised learning method that combines features using a Naive Bayes parametric form, achieving competitive performance.
• Applies the Expectation-Maximization (EM) algorithm and Gibbs Sampling to estimate missing word sense labels in unsupervised learning, treating senses as latent variables.
• Develops frequency-based, local-context feature sets (e.g., collocations, co-occurrences) that are effective for unsupervised WSD, even without part-of-speech information.
• Performs post-processing to map unsupervised sense clusters to standard sense inventories by generating descriptive summaries from cluster content, enabling comparison with human-annotated senses.

Experimental results

Research questions

• RQ1 Can a forward sequential search strategy combined with information criteria like AIC consistently select high-accuracy probabilistic models for word sense disambiguation?
• RQ2 How does the Naive Mix algorithm, a novel supervised learning method, compare in performance to leading machine learning approaches in WSD?
• RQ3 To what extent can unsupervised learning methods infer meaningful word senses from raw, untagged text using EM and Gibbs sampling with a Naive Bayes parametric form?
• RQ4 Can feature sets based on frequency and local context alone achieve competitive performance in unsupervised WSD without relying on part-of-speech tagging?
• RQ5 Is it feasible to automatically map unsupervised sense clusters to entries in a standard sense inventory using descriptive summaries derived from cluster content?

Key findings

• The combination of forward sequential search and Akaike’s Information Criterion (AIC) consistently selects highly accurate probabilistic models for word sense disambiguation.
• The Naive Mix algorithm, a supervised learning method based on the Naive Bayes parametric form, achieves performance competitive with leading machine learning methodologies in WSD.
• The Naive Bayes classifier performs surprisingly well despite its strong independence assumptions, which the paper explains through bias-variance decomposition and learning rate analysis.
• Unsupervised learning using EM and Gibbs sampling successfully infers word senses from raw text when a parametric form (e.g., Naive Bayes) is assumed, even without sense-tagged data.
• Frequency-based, local-context feature sets are effective for unsupervised WSD and perform well even when part-of-speech information is not used or is unreliable.
• Post-processing that generates descriptive summaries from sense clusters enables mapping of unsupervised clusters to standard sense inventories, allowing precise evaluation against human judgments.