[Paper Review] Visual Wake Words Dataset

Aakanksha Chowdhery, Pete Warden+3 more|arXiv (Cornell University)|Jun 12, 2019

IoT and Edge/Fog ComputingComputer Science26 references84 citations

TL;DR

The paper introduces Visual Wake Words, a binary person/not-person dataset derived from COCO to benchmark tiny vision models under microcontroller memory constraints, achieving 85–90% accuracy within 250 KB memory and under 60M multiply‑adds. It analyzes memory-latency tradeoffs and benchmarks MobileNet variants for edge AI deployment.

ABSTRACT

The emergence of Internet of Things (IoT) applications requires intelligence on the edge. Microcontrollers provide a low-cost compute platform to deploy intelligent IoT applications using machine learning at scale, but have extremely limited on-chip memory and compute capability. To deploy computer vision on such devices, we need tiny vision models that fit within a few hundred kilobytes of memory footprint in terms of peak usage and model size on device storage. To facilitate the development of microcontroller friendly models, we present a new dataset, Visual Wake Words, that represents a common microcontroller vision use-case of identifying whether a person is present in the image or not, and provides a realistic benchmark for tiny vision models. Within a limited memory footprint of 250 KB, several state-of-the-art mobile models achieve accuracy of 85-90% on the Visual Wake Words dataset. We anticipate the proposed dataset will advance the research on tiny vision models that can push the pareto-optimal boundary in terms of accuracy versus memory usage for microcontroller applications.

Motivation & Objective

• Motivate the need for on-device vision on microcontrollers with extreme memory constraints.
• Propose Visual Wake Words as a realistic binary-class benchmark derived from COCO.
• Characterize memory, latency, and model-size tradeoffs for tiny CNNs on edge devices.
• Benchmark state-of-the-art mobile models under 250 KB flash/ SRAM limits and 60M MACs per inference.

Proposed method

• Define the design constraints for tiny vision models: ≤250 KB peak memory and ≤60M MACs per inference.
• Create the Visual Wake Words dataset by relabeling COCO images as person/not-person based on bounding box area (>0.5% of image).
• Train and quantize MobileNet V1/V2, MNasNet, and ShuffleNet with 8-bit weights/activations.
• Evaluate accuracy vs peak memory, parameter count, and MACs on ImageNet and Visual Wake Words datasets.
• Examine memory management techniques for MobileNet V2 and MNasNet to fit SRAM constraints.

Experimental results

Research questions

• RQ1 What accuracy can tiny vision models achieve on Visual Wake Words within 250 KB memory and 60M MACs?
• RQ2 How do model size, peak memory, and compute scale with image resolution and depth multipliers under edge constraints?
• RQ3 What memory-latency tradeoffs arise from residual and parallel paths in mobile architectures when constrained to microcontroller hardware?
• RQ4 Can 8-bit quantization enable competitive performance for person/not-person classification on microcontrollers?

Key findings

• Visual Wake Words allows 85–90% accuracy with state-of-the-art mobile models under 250 KB memory limits.
• MobileNet V1/V2, MNasNet, and ShuffleNet achieve high accuracy on Visual Wake Words while fitting within 250 KB flash storage.
• Peak memory usage is often dominated by activation maps in early layers, requiring memory-saving strategies for parallel paths.
• Reducing image resolution lowers peak memory and MACs but can constrain accuracy; tradeoffs depend on architecture and depth multiplier.
• 8-bit quantization with quantization-aware training enables competitive accuracy for the binary task.
• On ImageNet, the same models have lower top-1 accuracy, illustrating the Visual Wake Words dataset provides a distinct pareto frontier for tiny vision models.