[Paper Review] An Efficient, Probabilistically Sound Algorithm for Segmentation and Word Discovery

Michael R. Brent|ArXiv.org|May 12, 1999

Algorithms and Data Compression23 references208 citations

TL;DR

This paper presents MBDP-1, a probabilistically sound, unsupervised algorithm for word segmentation and word discovery in continuous speech-like text, using a Bayesian model that treats the entire corpus as a single probabilistic event. It outperforms prior methods on child-directed speech corpora by combining phoneme frequency, word frequency, and word-order statistics to identify high-probability segmentations without prior lexical knowledge or multiple passes.

ABSTRACT

This paper presents a model-based, unsupervised algorithm for recovering word boundaries in a natural-language text from which they have been deleted. The algorithm is derived from a probability model of the source that generated the text. The fundamental structure of the model is specified abstractly so that the detailed component models of phonology, word-order, and word frequency can be replaced in a modular fashion. The model yields a language-independent, prior probability distribution on all possible sequences of all possible words over a given alphabet, based on the assumption that the input was generated by concatenating words from a fixed but unknown lexicon. The model is unusual in that it treats the generation of a complete corpus, regardless of length, as a single event in the probability space. Accordingly, the algorithm does not estimate a probability distribution on words; instead, it attempts to calculate the prior probabilities of various word sequences that could underlie the observed text. Experiments on phonemic transcripts of spontaneous speech by parents to young children suggest that this algorithm is more effective than other proposed algorithms, at least when utterance boundaries are given and the text includes a substantial number of short utterances. Keywords: Bayesian grammar induction, probability models, minimum description length (MDL), unsupervised learning, cognitive modeling, language acquisition, segmentation

Motivation & Objective

• To develop an unsupervised, incremental algorithm for word boundary detection in continuous text, mimicking how children learn language from unsegmented input.
• To create a language-independent model that does not rely on pre-existing dictionaries or pre-segmented training data.
• To improve segmentation accuracy by modeling the joint probability of word sequences based on phoneme frequencies, word frequencies, and word-order constraints.
• To evaluate the algorithm on naturalistic, child-directed speech corpora, which differ significantly from standard language engineering datasets.
• To provide a cognitively plausible model of early language acquisition that explains how children might discover words from continuous input.

Proposed method

• The algorithm uses a Bayesian model that treats the generation of an entire corpus as a single probabilistic event, assigning prior probabilities to all possible word sequences that could produce the observed input.
• It employs a modular probability model where phonology, word order, and word frequency are treated as interchangeable components, allowing language-specific refinements.
• Word boundaries are determined by maximizing the prior probability of a segmentation, rather than estimating a posterior distribution over words.
• The method uses dynamic programming to efficiently compute the most probable segmentation, avoiding the need for multiple passes or global optimization.
• It incorporates phoneme frequency in the lexicon as a key factor in evaluating the plausibility of novel word candidates, reducing the likelihood of low-frequency phoneme sequences being treated as words.
• The model uses a prior distribution over word types based on their frequency and length, informed by established distributions such as Zipf’s law and Mandelbrot’s model.

Experimental results

Research questions

• RQ1 Can a single, unified probabilistic model based on corpus-level priors outperform existing unsupervised segmentation algorithms on child-directed speech?
• RQ2 To what extent does incorporating lexical phoneme frequency improve the accuracy of word boundary detection in continuous input?
• RQ3 Does a model that treats the entire corpus as a single event yield better segmentation than models that estimate word probabilities incrementally?
• RQ4 How well does the algorithm perform on corpora with short utterances and variable word boundaries, typical of child-directed speech?
• RQ5 Can the model explain cognitive phenomena such as the tendency to segment familiar words and avoid overlapping segments?

Key findings

• MBDP-1 outperforms other unsupervised segmentation algorithms on phonemic transcripts of spontaneous parent-child speech, particularly when utterance boundaries are provided and utterances are short.
• The algorithm achieves higher segmentation accuracy by leveraging the prior probability of word sequences, which incorporates phoneme frequency, word frequency, and word-order statistics.
• Including the frequency of phonemes within the lexicon significantly improves the model’s ability to reject implausible novel word candidates, such as those with rare initial phonemes.
• The model predicts that novel words are less likely to be formed from low-frequency phoneme sequences, which aligns with behavioral data from artificial language learning experiments.
• The algorithm successfully identifies familiar words even when they are embedded in longer, unsegmented strings, especially when they do not overlap with other known words.
• The model’s performance is consistent with the INCDROP framework, supporting the hypothesis that children minimize novel word length and maximize word frequency in segmentation decisions.