[Paper Review] Semantic speech retrieval with a visually grounded model of untranscribed speech

Herman Kamper, Gregory Shakhnarovich+1 more|arXiv (Cornell University)|Oct 5, 2017

Multimodal Machine Learning Applications124 references49 citations

TL;DR

This paper proposes a visually grounded neural model that learns semantic representations from untranscribed speech paired with images, using a pre-trained image tagger to generate soft text labels for training. Without any transcribed speech, the model achieves 58.8% precision on top-10 semantic speech retrieval, outperforming a supervised transcription-based model in retrieving non-verbatim semantic matches, demonstrating the value of visual context for low-resource semantic understanding in speech.

ABSTRACT

There is growing interest in models that can learn from unlabelled speech paired with visual context. This setting is relevant for low-resource speech processing, robotics, and human language acquisition research. Here we study how a visually grounded speech model, trained on images of scenes paired with spoken captions, captures aspects of semantics. We use an external image tagger to generate soft text labels from images, which serve as targets for a neural model that maps untranscribed speech to (semantic) keyword labels. We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query. Without seeing any text, the model trained on parallel speech and images achieves a precision of almost 60% on its top ten semantic retrievals. Compared to a supervised model trained on transcriptions, our model matches human judgements better by some measures, especially in retrieving non-verbatim semantic matches. We perform an extensive analysis of the model and its resulting representations.

Motivation & Objective

• To investigate whether visual context can effectively ground semantic understanding in untranscribed speech, especially in low-resource settings.
• To develop a method that maps untranscribed speech to semantic keyword labels using image-derived soft labels.
• To evaluate the model's performance on semantic speech retrieval, where relevance is defined beyond exact keyword matching.
• To collect and release a new dataset of human-annotated semantic relevance judgements for spoken utterances.
• To compare the visually grounded model against supervised models trained on transcriptions, focusing on semantic generalization.

Proposed method

• A pre-trained image tagger generates soft text labels (e.g., 'person', 'dog') from training images, serving as weak supervision for speech modelling.
• A neural network maps untranscribed speech to predicted keyword labels by minimizing cross-entropy loss against the soft image labels.
• The model is trained end-to-end using a joint embedding space where speech and image-derived labels are aligned.
• The model is evaluated on a new semantic speech retrieval task, where a text query retrieves semantically relevant spoken utterances regardless of exact word overlap.
• Multiple baselines are compared, including a supervised model using ground-truth transcriptions and cascaded ASR + semantic models.
• t-SNE visualization is used to analyze learned speech representations and assess semantic clustering.

Experimental results

Research questions

• RQ1 Can a model trained on untranscribed speech and images learn to retrieve semantically relevant utterances without any text supervision?
• RQ2 How does the performance of a visually grounded model compare to a supervised model trained on transcriptions in retrieving non-verbatim semantic matches?
• RQ3 To what extent do the learned speech representations cluster semantically related words, such as 'man' and 'person'?
• RQ4 Can the model's predictions better match human-annotated soft relevance scores than a supervised transcription-based model?
• RQ5 How robust is the visually grounded model compared to cascaded ASR + semantic models when ASR accuracy degrades?

Key findings

• The visually grounded model achieves a precision of 58.8% on the top-10 semantic retrievals, despite never seeing any transcriptions.
• The model outperforms the supervised transcription-based model in retrieving non-verbatim semantic matches, with 25.3% correct semantic predictions compared to 22.3% for exact matches.
• The model's predictions correlate more strongly with human-annotated soft relevance scores (Spearman’s ρ = 32.4) than the supervised model (ρ = 31.6), especially on semantic matches.
• Even when ASR error rates reach 50%, the visually grounded model (VISIONSPEECHCNN) outperforms the cascaded ASR + semantic model on most metrics, including P@10 and Spearman’s ρ.
• t-SNE visualization confirms that the model learns semantically meaningful representations, grouping related words like 'bike', 'rides', and 'riding' into similar clusters.
• The model's performance is superior to automatic text-based models (e.g., TEXTPARAGRAM) when evaluated against soft human annotations, indicating that human judgements cannot be fully replaced by automatic semantic models.