[Paper Review] Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications

Colby Banbury, Emil Njor+8 more|arXiv (Cornell University)|May 1, 2024

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper introduces Wake Vision, a large-scale, diverse dataset of over 6 million images for TinyML person detection, 100x larger than prior benchmarks. It includes a fine-grained benchmark suite evaluating model performance across lighting, distance, and demographic factors, demonstrating that input resolution is more critical than model width for distant detection and that int8 quantization has negligible impact on robustness—enabling more reliable, privacy-preserving TinyML models.

ABSTRACT

Tiny machine learning (TinyML) for low-power devices lacks systematic methodologies for creating large, high-quality datasets suitable for production-grade systems. We present a novel automated pipeline for generating binary classification datasets that addresses this critical gap through several algorithmic innovations: intelligent multi-source label fusion, confidence-aware filtering, automated label correction, and systematic fine-grained benchmark generation. Crucially, automation is not merely convenient but necessary to cope with TinyML's diverse applications. TinyML requires bespoke datasets tailored to specific deployment constraints and use cases, making manual approaches prohibitively expensive and impractical for widespread adoption. Using our pipeline, we create Wake Vision, a large-scale binary classification dataset of almost 6 million images that demonstrates our methodology through person detection--the canonical vision task for TinyML. Wake Vision achieves up to a 6.6% accuracy improvement over existing datasets via a carefully designed two-stage training strategy and provides 100x more images. We demonstrate our broad applicability for automated large-scale TinyML dataset generation across two additional target categories, and show our label error rates are substantially lower than prior work. Our comprehensive fine-grained benchmark suite evaluates model robustness across five critical dimensions, revealing failure modes masked by aggregate metrics. To ensure continuous improvement, we establish ongoing community engagement through competitions hosted by the Edge AI Foundation. All datasets, benchmarks, and code are available under CC-BY 4.0 license, providing a systematic foundation for advancing TinyML research.

Motivation & Objective

• To address the lack of large-scale, high-quality datasets for TinyML by creating a scalable, permissively licensed alternative to existing benchmarks.
• To improve model robustness and fairness in real-world deployment by introducing fine-grained benchmarks for challenging conditions such as low light, distance, and demographic variation.
• To evaluate the impact of common TinyML optimization techniques—like quantization and grayscale input—on model performance across diverse data subsets.
• To enable more reliable and privacy-preserving TinyML systems by supporting on-device inference with high accuracy and minimal data leakage.

Proposed method

• The Wake Vision dataset is constructed from Open Images v7, filtered for quality and expanded to over 6 million images, with two training variants: 'Large' (maximize size) and 'Quality' (maximize label accuracy).
• A benchmark suite of five fine-grained test sets is created, each focusing on specific conditions: distance, lighting (dark, normal, bright), demographic attributes (perceived age and gender), and image depiction style.
• Models are trained and evaluated on the dataset using standard TinyML pipelines, including MobileNetV2 and int8 quantization, with performance measured via F1 score across all benchmarks.
• A case study evaluates the effects of input resolution, model width, and grayscale conversion on model robustness, particularly in challenging scenarios.
• The dataset and benchmarks are released under CC-BY 4.0, enabling commercial and research use with full reproducibility.

(a) An example of a person far away

Experimental results

Research questions

• RQ1 How does increasing dataset scale from ~60k to over 6 million images impact TinyML model accuracy for person detection?
• RQ2 To what extent do fine-grained benchmarks reveal performance disparities in real-world conditions such as low lighting or distant subjects?
• RQ3 How does int8 quantization affect model robustness across underrepresented demographic and environmental subsets?
• RQ4 Does grayscale input significantly degrade model performance in bright lighting conditions, and how does this vary across lighting benchmarks?

Key findings

• Wake Vision improves model accuracy by 2.41% over the prior state-of-the-art VWW benchmark, demonstrating the benefit of scale and quality.
• Input resolution has a more significant impact on detecting distant subjects than model width, highlighting a key design trade-off in TinyML.
• Int8 quantization causes negligible performance degradation (±0.004 F1) across all benchmarks, indicating robustness even in challenging conditions.
• Grayscale input increases sensitivity to bright lighting conditions, with models showing greater performance drops in the 'bright' lighting benchmark compared to RGB.
• The 'Quality' training set outperforms the 'Large' set, underscoring that data quality is more critical than sheer volume in TinyML.

(b) An example of a depicted person