[Paper Review] Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces

Alice Coucke, Alaa Saade+10 more|arXiv (Cornell University)|May 25, 2018

Context-Aware Activity Recognition SystemsComputer Science355 citations

TL;DR

The paper describes an offline, private-by-design SLU platform that runs on IoT hardware, using a compact NN/HMM acoustic model and a dynamic, domain-adapted language model with on-device personalization.

ABSTRACT

This paper presents the machine learning architecture of the Snips Voice Platform, a software solution to perform Spoken Language Understanding on microprocessors typical of IoT devices. The embedded inference is fast and accurate while enforcing privacy by design, as no personal user data is ever collected. Focusing on Automatic Speech Recognition and Natural Language Understanding, we detail our approach to training high-performance Machine Learning models that are small enough to run in real-time on small devices. Additionally, we describe a data generation procedure that provides sufficient, high-quality training data without compromising user privacy.

Motivation & Objective

• Motivate privacy-by-design for voice interfaces that do not send user data to the cloud.
• Develop an end-to-end SLU pipeline (ASR + NLU) that runs in real-time on small devices.
• Train compact acoustic models and domain-adapted language models suitable for embedded hardware.
• Enable on-device personalization and efficient training/update of language models without leaking data.

Proposed method

• Build a compact NN/HMM acoustic model trained on general speech with data augmentation to simulate noisy, reverberant conditions.
• Train a domain-focused language model (LM) and a corresponding NLU model from a shared dataset that abstracts entities for generalization.
• Use a dynamic, lazy composition of transducers (HCLG) to form a runtime decoding graph that fits on-device memory constraints.
• Implement on-device entity injection to personalize the LM without transmitting user data.
• Incorporate a confidence-scoring mechanism based on confusion networks to detect out-of-vocabulary words and improve end-to-end performance.

Experimental results

Research questions

• RQ1 How can SLU be realized offline on low-resource IoT hardware while preserving user privacy?
• RQ2 What is the impact of domain-adapted LM/NLU integration on end-to-end speech-to-meaning accuracy for embedded assistants?
• RQ3 Can on-device personalization (entity injection) be performed efficiently without cloud data transfer?
• RQ4 How does the compact acoustic model balance accuracy, speed, and memory on devices like Raspberry Pi 3?

Key findings

• An embedded SLU pipeline running in real-time on a Raspberry Pi 3 is achievable with a compact NN/HMM acoustic model (nnet-256) and a tailored LM.
• Domain-adapted LM/NLU integration yields high end-to-end accuracy in targeted use cases, compensating for smaller acoustic models.
• Smaller models (nnet-256) offer substantially lower parameter counts and memory footprint than larger variants (nnet-768) with acceptable accuracy when paired with an adapted LM and robust NLU.
• On-device personalization via entity injection is feasible within seconds to a few dozen seconds, enabling user-specific vocabulary updates without cloud data.
• A confusion-network based confidence score helps identify OOV words, improving robustness of the end-to-end SLU system.