[Paper Review] Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers

Loren Lugosch, Piyush Papreja+3 more|arXiv (Cornell University)|Apr 4, 2021

Intelligent Tutoring Systems and Adaptive Learning4 citations

TL;DR

This paper introduces Timers and Such, a new open-source dataset of spoken English commands involving multi-digit numbers for offline voice assistant applications. It evaluates end-to-end SLU models and ASR-based baselines, showing that direct audio-to-intent models outperform ASR-based models (96.7% vs. 85.4% accuracy) on synthetic data, highlighting its value for robust, edge-compatible SLU research.

ABSTRACT

This paper introduces Timers and Such, a new open source dataset of spoken English commands for common voice control use cases involving numbers. We describe the gap in existing spoken language understanding datasets that Timers and Such fills, the design and creation of the dataset, and experiments with a number of ASR-based and end-to-end baseline models, the code for which has been made available as part of the SpeechBrain toolkit.

Motivation & Objective

• To address the lack of open-source SLU datasets that comprehensively cover multi-digit numbers in context, especially for offline voice assistant use cases.
• To create a practical, programmatically downloadable dataset with diverse numerical expressions (e.g., '13.57', '-21.4') in natural spoken commands.
• To provide a benchmark for evaluating end-to-end SLU models and ASR-based baselines on audio, focusing on robustness and low-resource deployment.
• To support research in audio-linguistic representation learning by offering a downstream task with realistic, numerically rich utterances.

Proposed method

• The dataset was created using a script that generated varied phrasings (e.g., 'set a timer for', 'start timer for') with randomly sampled numbers appropriate to each intent.
• Four intents were defined: SetTimer, SetAlarm, SimpleMath, and UnitConversion, each with semantic labels including intent and slot values (e.g., number1, number2, op).
• The dataset contains 2,151 real audio samples from volunteer speakers and 1,000 synthetic samples generated via TTS, with a CC0 license for broad reuse.
• Baseline models were implemented using the SpeechBrain toolkit, including both decoupled (ASR + NLU) and direct (end-to-end) approaches on raw audio.
• The direct model uses a transformer-based architecture with global attention, while the ASR-based model uses a pre-trained LibriSpeech ASR system.
• Training and evaluation were conducted across multiple random seeds, with performance measured via intent accuracy and SLU WER, and resource usage tracked for reproducibility.

Experimental results

Research questions

• RQ1 How do end-to-end SLU models perform on audio with complex numerical expressions compared to ASR-based pipelines in offline voice assistant scenarios?
• RQ2 What is the impact of training data composition (real vs. synthetic) on model generalization and performance stability?
• RQ3 Why does test accuracy show high variability across random seeds, even with large real test sets, and can this be mitigated?
• RQ4 Can direct audio-to-intent models achieve superior performance to ASR-based models on numerically rich, context-dependent commands?
• RQ5 How can the dataset be used to improve robustness in low-resource, edge-compatible SLU systems?

Key findings

• The direct end-to-end SLU model achieved 96.7% intent accuracy on synthetic test data, significantly outperforming the ASR-based baseline at 85.4%.
• On real test data, the direct model achieved 92.1% accuracy, while the ASR-based model reached 88.3%, indicating strong generalization despite high variance.
• Test accuracy showed high standard deviation (up to 5.7%) across random seeds, suggesting instability in training or decoding that requires further investigation.
• The direct model’s performance on synthetic data was much higher than on real data, indicating potential distribution shift or overfitting to synthetic patterns.
• Training a single model on Timers and Such takes only about one minute per epoch on a GPU, and all recipes run successfully on older 12GB GPUs, enabling broad accessibility.
• The dataset is hosted on Zenodo under CC0 and can be downloaded programmatically via wget or curl, lowering the barrier to entry for new researchers.