[Paper Review] The Unsupervised Acquisition of a Lexicon from Continuous Speech

Carl de Marcken|ArXiv.org|Dec 13, 1995

Natural Language Processing Techniques31 references61 citations

TL;DR

This paper presents an unsupervised algorithm that acquires a natural-language lexicon directly from raw continuous speech using a minimum description length (MDL) framework. By modeling speech through articulatory features and applying hierarchical, recursive compression, the system learns linguistically meaningful words, segmentations, and language models without prior knowledge or labeled data—achieving strong performance on TIMIT, Brown, and CHILDES datasets.

ABSTRACT

We present an unsupervised learning algorithm that acquires a natural-language lexicon from raw speech. The algorithm is based on the optimal encoding of symbol sequences in an MDL framework, and uses a hierarchical representation of language that overcomes many of the problems that have stymied previous grammar-induction procedures. The forward mapping from symbol sequences to the speech stream is modeled using features based on articulatory gestures. We present results on the acquisition of lexicons and language models from raw speech, text, and phonetic transcripts, and demonstrate that our algorithm compares very favorably to other reported results with respect to segmentation performance and statistical efficiency.

Motivation & Objective

• To develop an unsupervised learning algorithm that acquires a lexicon from raw, continuous speech without prior linguistic knowledge or segmentation.
• To overcome limitations of prior grammar-induction methods by using a hierarchical representation that encourages linguistically plausible structures.
• To demonstrate that optimal compression via MDL can serve as a principled basis for discovering words and language structure.
• To show that the algorithm can learn from diverse input types, including raw speech, text, and phonetic transcripts, with consistent performance.
• To provide a foundation for unsupervised acquisition of syntax and semantics by first establishing robust word and language model learning.

Proposed method

• Uses a minimum description length (MDL) framework to optimize the joint compression of speech and the lexicon, favoring compact, informative representations.
• Represents speech as sequences of articulatory feature bundles, linking phonetic input to symbolic linguistic structure.
• Employs a hierarchical, recursive dictionary-based coding scheme where linguistic knowledge is encoded in terms of other linguistic knowledge.
• Applies a search strategy that avoids dependency on search history, reducing local minima and enabling dynamic restructuring of learned knowledge.
• Performs segmentation and lexicon acquisition by iteratively identifying recurring patterns that minimize description length.
• Treats word boundaries and multi-word units as emergent from compression, allowing idiomatic expressions to be learned as single units.

Experimental results

Research questions

• RQ1 Can a system learn a lexicon from raw, continuous speech without any prior linguistic knowledge or segmentation cues?
• RQ2 Can optimal compression via MDL serve as a principled basis for discovering words and syntactic structure in speech?
• RQ3 How effective is a hierarchical, recursive compression approach in capturing linguistically meaningful units compared to flat or non-hierarchical models?
• RQ4 Can the same algorithm learn from text, phonetic transcripts, and raw speech with consistent performance?
• RQ5 To what extent can unsupervised learning of lexicon and language models match or exceed supervised or manually constructed alternatives in statistical efficiency?

Key findings

• The algorithm successfully acquires a lexicon and language model from raw speech, demonstrating that supervised training is not necessary for core lexical learning.
• Segmentation performance is quantitatively strong and aligns well with linguistic intuition, as validated on TIMIT, Brown, and CHILDES datasets.
• The resulting language models show high statistical efficiency, outperforming other reported results in compression and prediction tasks.
• The system learns multi-word units (e.g., 'wanna') as single lexical entries, reflecting real-world usage better than traditional dictionaries.
• The hierarchical representation supports both compositional and idiomatic expressions, making it suitable for machine translation and speech recognition.
• This is the first reported work to learn words directly from raw speech without prior knowledge, marking a significant step toward unsupervised language acquisition.