[Paper Review] Comparison of Global Algorithms in Word Sense Disambiguation

Loïc Vial, Andon Tchechmedjiev+1 more|arXiv (Cornell University)|Apr 7, 2017

Natural Language Processing Techniques7 references3 citations

TL;DR

This paper evaluates four global probabilistic optimization algorithms—Cuckoo Search (CSA), Bat Search (BS), Simulated Annealing (SA), and Genetic Algorithm (GA)—for Word Sense Disambiguation (WSD) using a perfect scorer. CSA achieves the highest F1 score (0.98) with fewer scorer calls than SA and GA, while BS fails to converge beyond 0.89 F1 due to premature convergence.

ABSTRACT

This article compares four probabilistic algorithms (global algorithms) for Word Sense Disambiguation (WSD) in terms of the number of scorer calls (local algo- rithm) and the F1 score as determined by a gold-standard scorer. Two algorithms come from the state of the art, a Simulated Annealing Algorithm (SAA) and a Genetic Algorithm (GA) as well as two algorithms that we first adapt from WSD that are state of the art probabilistic search algorithms, namely a Cuckoo search algorithm (CSA) and a Bat Search algorithm (BS). As WSD requires to evaluate exponentially many word sense combinations (with branching factors of up to 6 or more), probabilistic algorithms allow to find approximate solution in a tractable time by sampling the search space. We find that CSA, GA and SA all eventually converge to similar results (0.98 F1 score), but CSA gets there faster (in fewer scorer calls) and reaches up to 0.95 F1 before SA in fewer scorer calls. In BA a strict convergence criterion prevents it from reaching above 0.89 F1.

Motivation & Objective

• To evaluate the performance of state-of-the-art global optimization algorithms in Word Sense Disambiguation (WSD) under controlled conditions.
• To isolate the impact of global algorithms by using a perfect, gold-standard scorer, eliminating bias from heuristic scoring functions.
• To determine which algorithm achieves the highest F1 score with the fewest scorer calls, especially relevant when scoring functions are computationally expensive.
• To analyze convergence behavior and identify limitations in algorithms like Bat Search that prematurely halt improvement.
• To assess whether newer algorithms like Cuckoo Search and Bat Search outperform established methods such as Simulated Annealing and Genetic Algorithms in WSD.

Proposed method

• Adapted Cuckoo Search and Bat Search algorithms as global optimization strategies for WSD, treating sense assignments as configurations in a search space.
• Used a scoring function (fitness function) that computes the F1 score against the gold-standard sense annotations for each configuration.
• Executed experiments with increasing limits on scorer calls (200 to 16,000) to track convergence trends across all algorithms.
• Applied identical parameters and random seeds across runs to ensure fair comparison, focusing on the number of scorer calls and final F1 scores.
• Employed a perfect scorer to decouple the performance of global algorithms from heuristic scoring function variability.
• Analyzed convergence patterns and early-stage performance to assess sampling efficiency and robustness of each algorithm.

Experimental results

Research questions

• RQ1 How do Cuckoo Search, Bat Search, Simulated Annealing, and Genetic Algorithm compare in F1 score and convergence speed on WSD using a perfect scorer?
• RQ2 Does Cuckoo Search achieve higher F1 scores with fewer scorer calls compared to Simulated Annealing and Genetic Algorithm?
• RQ3 Why does Bat Search fail to reach higher F1 scores despite initial performance, and is this due to algorithmic design or parameter sensitivity?
• RQ4 At what point do Simulated Annealing and Cuckoo Search cross over in performance, and is this crossover practically relevant when scoring functions are expensive?
• RQ5 Can the performance of heuristic scoring functions be improved by aligning them more closely with the oracle scorer used in this study?

Key findings

• Cuckoo Search (CSA) achieves a final F1 score of 0.98 with significantly fewer scorer calls than Simulated Annealing (SA) and Genetic Algorithm (GA), demonstrating superior sampling efficiency.
• Simulated Annealing (SA) eventually reaches the same F1 score (0.98) as CSA and GA but only after 4,000+ scorer calls, where CSA has already achieved 0.95 F1.
• Bat Search (BS) fails to exceed 0.89 F1 due to premature convergence, caused by the algorithm’s strict acceptance rule that halts improvement even when better solutions are found.
• Genetic Algorithm (GA) performs well but is outpaced by CSA in early convergence, reaching 0.67 F1 at 500 calls compared to CSA’s 0.79.
• All algorithms except BS converge to the same optimal solution (F1 = 0.98), indicating that the search space is navigable, but CSA reaches it most efficiently.
• The study concludes that CSA is the most effective global algorithm for WSD when scorer calls are limited, especially in scenarios with expensive scoring functions.