[논문 리뷰] The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling

Tu Anh Nguyen, Maureen de Seyssel외 6명|arXiv (Cornell University)|2020. 11. 23.

Speech Recognition and Synthesis인용 수 42

한 줄 요약

네 가지 작업의 제로샷 벤치마크를 무감독 음성언어 모델링에 소개하고 간단한 CPC+클러스터링+LM Baseline 제시; 실행 가능성을 보이나 텍스트 topline과의 격차 여전.

ABSTRACT

We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics. We present the results and analyses of a composite baseline made of the concatenation of three unsupervised systems: self-supervised contrastive representation learning (CPC), clustering (k-means) and language modeling (LSTM or BERT). The language models learn on the basis of the pseudo-text derived from clustering the learned representations. This simple pipeline shows better than chance performance on all four metrics, demonstrating the feasibility of spoken language modeling from raw speech. It also yields worse performance compared to text-based 'topline' systems trained on the same data, delineating the space to be explored by more sophisticated end-to-end models.

연구 동기 및 목표

• 음성 모델을 음성학적, 어휘적, 통사적, 의미적 수준에서 평가하기 위한 제로리소스, 블랙박스 벤치마크 정의.
• 레이블 없이 원시 오디오로부터 학습하는 간단한 무감독 baseline 파이프라인 시연.
• 고정된 전사 granularity에 의존하지 않는 해석 가능한 지표 제공.
• 음성-텍스트 기반 언어 모델링의 다리를 놓기 위한 오픈소스 데이터셋과 베이스라인 제공.

제안 방법

• 네 가지 제로샷 지표(ABX for phonetics, sWUGGY for lexical, sBLIMP for syntax, sSIMI for semantics) using Libri-light and synthesized stimuli.
• Build a composite baseline from Contrastive Predictive Coding (CPC), a k-means discretization, and a language model (LSTM or BERT) trained on pseudo-text.
• Discretize audio into units via clustering of CPC representations and train LMs on the resulting pseudo-text.
• Compare baseline performance to text-based toplines trained on the LibriSpeech phonetic/phoneme representations and RoBERTa large.
• Utilize a simple span-masked prediction objective for BERT-style models with masking spans of tokens.
• Provide dataset construction details including LibriSpeech LibriLight data, phonetic transcriptions, and forced alignments.

실험 결과

연구 질문

• RQ1 Can unsupervised spoken language models achieve non-zero performance on linguistically motivated, zero-shot tasks across acoustic, lexical, syntactic, and semantic levels?
• RQ2 How does a simple CPC+clustering+LM pipeline perform relative to chance and to text-based toplines on the four metrics?
• RQ3 What are the limitations and gaps between speech-based models and text-based models, and where should future work focus?

주요 결과

• A simple CPC+km50+LM baseline yields better-than-chance performance on all four zero-shot metrics.
• Performance is above chance for lexical tasks and below text toplines for syntactic and semantic tasks.
• Clustering (50 clusters) is a sweet spot for ABX, and larger than 50 clusters degrades ABX in this setup.
• End-to-end or larger-scale models (e.g., wav2vec-style approaches or larger training data) show potential to close the gap to toplines.
• The benchmark and baselines are open-sourced to facilitate bridging speech and text-based systems.