[Paper Review] Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

David Harwath, Wei-Ning Hsu+1 more|arXiv (Cornell University)|Nov 21, 2019

Multimodal Machine Learning Applications68 references70 citations

TL;DR

The paper introduces ResDAVEnet-VQ, a visually grounded speech model that uses multiple vector quantization layers to learn hierarchical discrete linguistic units, achieving strong sub-word ABX performance and discovering word-like detectors at higher layers.

ABSTRACT

In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3\\% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

Motivation & Objective

• Motivate learning discrete, hierarchical linguistic representations from speech without transcription.
• Leverage visual grounding as a discriminative self-supervised objective to shape speech representations.
• Investigate how inserting multiple vector quantization layers yields sub-word and word-level units.
• Measure robustness of learned units to noise and domain shifts.
• Demonstrate the emergence of word-like detectors and quantify linguistic interpretability.

Proposed method

• Extend the ResDAVEnet visually-grounded speech model with multiple vector quantization (VQ) layers (VQ2, VQ3).
• Train under a discriminative, multimodal grounding objective rather than a reconstruction loss.
• Use straight-through estimators and EMA updates for the VQ codebooks with a 1024-codebook size.
• Experiment with cold-start and warm-start curricula to control which layers are quantized and when.
• Train end-to-end with a triplet-based grounding loss combining randomly-sampled and semi-hard negatives.
• Evaluate using ZeroSpeech ABX and R@10 metrics, plus RLE and segment-based bitrates for sub-word unit inventories.

Experimental results

Research questions

• RQ1 Can multiple VQ layers in a visually grounded speech model capture hierarchical discrete units (sub-word and word-like) without reconstruction losses?
• RQ2 How does the training curriculum (which layers are quantized and when) affect the emergence of phonetic versus lexical units?
• RQ3 Are the learned units robust to noise and domain shifts, compared to non-quantized baselines?
• RQ4 Do higher-layer quantizers specialize in word-like detectors, and can they map to actual words?
• RQ5 What is the interpretability of learned units in terms of phonetic/word correspondences?

Key findings

• Quantization guided by visual grounding yields discrete sub-word and word-like units depending on where and how VQ layers are inserted.
• A model with VQ2 (and with appropriate curriculum) achieves a 27.3% ABX improvement over top ZeroSpeech 2019 submissions while keeping comparable bitrate.
• The two-quantizer setup can learn lower-layer phonetic detectors and higher-layer word-like detectors, with the VQ3 layer showing word-level correspondences. A model learns 279 words with F1 > 0.5.
• Models show robustness to noise, outperforming WaveNet-VQ baselines under noisy conditions on ABX discrimination tasks.
• Visual grounding enables interpretable mappings, with the VQ3 codes aligning to word-like units and detectable words, evidenced by example word-code associations and F1-based evaluations.
• Curriculum effects: cold-start training of higher quantizers can hinder ABX but reveal word detectors, while warm-start or staged training preserves phonetic information in lower layers.