[Paper Review] Google Speech Commands-Musan test set

Trinh, Viet Anh, Kavaki, Hassan Salami+1 more|arXiv (Cornell University)|Apr 9, 2018

Speech Recognition and Synthesis6 references1,072 citations

TL;DR

This paper presents the Speech Commands dataset for limited-vocabulary, on-device keyword spotting, details its collection, quality control, and a reproducible evaluation protocol including Top-One and streaming metrics with baseline results.

ABSTRACT

Describes an audio dataset of spoken words designed to help train and evaluate keyword spotting systems. Discusses why this task is an interesting challenge, and why it requires a specialized dataset that is different from conventional datasets used for automatic speech recognition of full sentences. Suggests a methodology for reproducible and comparable accuracy metrics for this task. Describes how the data was collected and verified, what it contains, previous versions and properties. Concludes by reporting baseline results of models trained on this dataset.

Motivation & Objective

• Provide a standard, openly licensed dataset to train and evaluate small keyword-spotting models.
• Describe data collection, labeling, and quality-control procedures to ensure usable, speaker-independent data.
• Propose reproducible evaluation metrics and train/test splits to enable apples-to-apples comparisons across models.
• Offer baseline results and release code/tools to facilitate benchmarking and replication.

Proposed method

• Collect English one-second utterances of 20 core words (plus added words in version 2) using web-based recording with user consent.
• Perform multi-stage quality control including file-size filtering, conversion to 16 kHz WAV, and loudest-section extraction.
• Apply manual review via crowdsourcing to confirm labels.
• Provide background noise recordings and a deterministic data-split method using hashing for train/validation/test sets.
• Release data with speaker-identifier hashing to protect privacy and ensure reproducibility across releases.

Experimental results

Research questions

• RQ1 What vocabulary and data collection approach best support on-device keyword spotting with low false positives?
• RQ2 How can a reproducible data collection and evaluation protocol be established for fair model comparisons?
• RQ3 What baseline performance do simple models achieve on the Speech Commands dataset under standardized metrics?
• RQ4 How should training, validation, and test splits be constructed to avoid leakage across dataset releases?
• RQ5 How do version 1 and version 2 datasets compare in terms of Top-One accuracy and streaming metrics?

Key findings

• The final dataset contains 105,829 utterances across 35 words from 2,618 speakers, stored as 16 kHz 16-bit mono WAV files (~3.8 GB uncompressed).
• Version 2 improves Top-One accuracy over version 1 when trained and tested on the respective data: V1 Training with V1 Test 85.4% vs V2 Training with V2 Test 88.2%; cross-version results show 89.7% and 82.7% respectively.
• Baseline CNN-based keyword-spotting models achieve 88.2% Top-One on the V2 data using provided training commands.
• A standardized streaming evaluation yields multiple metrics (matched, correctly, wrongly, false positives) with a typical tolerance of 750 ms and a one-hour streaming test file for reproducibility.
• The dataset includes ten core words plus additional commands and distractor words, a background-noise corpus, and explicit train/validation/test lists with hash-based partitioning to prevent leakage across releases.
• The release process hashes speaker IDs to anonymize identities and ensures consistent IDs across releases.