[Paper Review] Comparative Experiments on Disambiguating Word Senses: An Illustration of the Role of Bias in Machine Learning

Raymond J. Mooney|ArXiv.org|Dec 9, 1996

Natural Language Processing TechniquesComputer Science47 references194 citations

TL;DR

This paper compares seven machine learning algorithms—naive Bayes, perceptron, decision trees, k-nearest neighbors, DNF/CNF rule learners, and decision lists—on word sense disambiguation for the word "line" using context from current and preceding sentences. The naive Bayes and perceptron methods significantly outperformed others, highlighting the importance of feature-weighting bias in handling distributed, overlapping evidence in NLP tasks.

ABSTRACT

This paper describes an experimental comparison of seven different learning algorithms on the problem of learning to disambiguate the meaning of a word from context. The algorithms tested include statistical, neural-network, decision-tree, rule-based, and case-based classification techniques. The specific problem tested involves disambiguating six senses of the word ``line'' using the words in the current and proceeding sentence as context. The statistical and neural-network methods perform the best on this particular problem and we discuss a potential reason for this observed difference. We also discuss the role of bias in machine learning and its importance in explaining performance differences observed on specific problems.

Motivation & Objective

• To empirically compare diverse machine learning algorithms on a specific lexical disambiguation task.
• To investigate how algorithmic bias influences performance differences across methods.
• To evaluate the relative effectiveness of statistical, neural, symbolic, and instance-based learning techniques on a real-world NLP problem.
• To provide a rigorous, statistically validated benchmark for word sense disambiguation using identical training and test sets.

Proposed method

• The study uses a corpus of 1,200 sentences with six senses of the word "line", annotated using words from the current and preceding sentence as context.
• Seven learning algorithms are trained and tested on identical training and test sets: naive Bayes, perceptron, C4.5 decision tree, k-nearest neighbors, PFOIL-based DNF and CNF rule learners, and decision list learner.
• Ten random trials are conducted per algorithm to compute average performance and enable statistical significance testing of differences.
• Performance is measured by accuracy on unseen test examples, with training and testing times recorded to assess efficiency.
• The feature representation encodes an unordered set of word stems from the context window, without positional or syntactic information.
• A qualitative analysis of algorithmic bias is performed to explain performance differences, focusing on how each method generalizes from training data.

Experimental results

Research questions

• RQ1 Which machine learning algorithm performs best on the task of disambiguating six senses of the word "line" using only word stems from adjacent sentences?
• RQ2 How do different algorithmic biases—such as linear thresholding, rule ordering, or feature independence—impact performance on this NLP task?
• RQ3 To what extent do statistical and neural network methods outperform symbolic and instance-based approaches in this context?
• RQ4 How do training and testing times vary across algorithm types, and what trade-offs do they imply for real-time applications?
• RQ5 Does the choice of input representation (e.g., unordered word stems) limit or shape the performance ranking of the algorithms?

Key findings

• The naive Bayes and perceptron methods achieved the highest accuracy, significantly outperforming all other algorithms on the word sense disambiguation task.
• The superior performance of naive Bayes and perceptron is attributed to their inherent bias for combining evidence from all features through weighted summation.
• Decision lists performed better than other symbolic methods, likely due to their rule-ordering mechanism that simplifies conflict resolution and representation.
• Symbolic methods like PFOIL-DNF and PFOIL-CNF had the slowest training times, with worst-case complexity O(n²), while testing was fastest due to minimal feature evaluation.
• The perceptron and naive Bayes methods had the most favorable trade-off between training time and accuracy, making them practical for large-scale applications.
• The results underscore that algorithmic bias is a key determinant of performance, and that no single method universally dominates across all NLP problems.