[Paper Review] MeciFace: Mechanomyography and Inertial Fusion-based Glasses for Edge Real-Time Recognition of Facial and Eating Activities

Hymalai Bello, Sungho Suh+2 more|arXiv (Cornell University)|Jun 19, 2023

Color perception and designPsychology3 citations

TL;DR

MeciFace is a low-power, privacy-preserving wearable glasses system that fuses mechanomyography (MMG) and inertial sensor data for real-time, on-edge recognition of facial expressions and eating/drinking activities. Using lightweight neural networks deployed via TensorFlow Lite for Microcontrollers, it achieves ≥86% F1-score for facial expression recognition and 90% for eating/drinking detection, even on unseen users, with power consumption below 0.55W.

ABSTRACT

The increasing prevalence of stress-related eating behaviors and their impact on overall health highlights the importance of effective and ubiquitous monitoring systems. In this paper, we present MeciFace, an innovative wearable technology designed to monitor facial expressions and eating activities in real-time on-the-edge (RTE). MeciFace aims to provide a low-power, privacy-conscious, and highly accurate tool for promoting healthy eating behaviors and stress management. We employ lightweight convolutional neural networks as backbone models for facial expression and eating monitoring scenarios. The MeciFace system ensures efficient data processing with a tiny memory footprint, ranging from 11KB to 19 KB. During RTE evaluation, the system achieves an F1-score of < 86% for facial expression recognition and 94% for eating/drinking monitoring, for the RTE of unseen users (user-independent case).

Motivation & Objective

• To develop a low-power, privacy-conscious wearable system for real-time monitoring of facial expressions and eating behaviors.
• To enable on-the-edge inference on a microcontroller without reliance on external devices, reducing latency and memory usage.
• To address the challenge of detecting stress-related eating behaviors through multimodal sensor fusion in a minimally intrusive form factor.
• To demonstrate feasibility of embedded deep learning for facial and eating activity recognition with minimal hardware footprint.
• To lay the foundation for contextual monitoring of emotional eating by integrating multiple sensing modalities.

Proposed method

• Fuses mechanomyography (MMG) and inertial measurement unit (IMU) signals from sensors embedded in the temple and nose bridge of smart glasses.
• Employs a hierarchical multimodal fusion strategy: first stage uses MMG to detect non-null facial gestures or eating/drinking actions; second stage uses IMU data for fine-grained classification.
• Deploys lightweight convolutional neural networks (CNNs) as backbone models, optimized for microcontrollers using TensorFlow Lite for Microcontrollers.
• Applies dynamic resampling techniques—linear-based in the prototype, Fourier-based in training—to normalize sensor signal duration for consistent input to the neural networks.
• Uses a facial muscle activity dictionary with 6 expressions (e.g., joy, anger) and 2 gestures (eating, drinking), plus a null/neutral class.
• Integrates additional sensors (barometer, gas, microphone) in future extensions to enrich contextual data for detecting stress-triggered eating episodes.

Experimental results

Research questions

• RQ1 Can a wearable glasses system achieve real-time, on-edge recognition of facial expressions and eating activities using only MMG and inertial sensors?
• RQ2 How effective is hierarchical multimodal fusion of MMG and IMU data in improving recognition accuracy while minimizing power consumption?
• RQ3 To what extent can lightweight neural networks deployed on microcontrollers achieve high performance with minimal memory footprint (<20KB)?
• RQ4 How does the system generalize to unseen users in real-time edge inference scenarios?
• RQ5 What is the potential of extending the system with additional sensors (e.g., gas, microphone) for contextual detection of emotional eating?

Key findings

• The MeciFace system achieves an F1-score of ≥86% for facial expression recognition and 90% for eating/drinking monitoring in real-time edge inference, even for unseen users.
• The system maintains power consumption below 0.55 Watts through efficient hardware-software co-design and lightweight neural networks.
• A 10% drop in F1-score was observed between offline and real-time edge inference, primarily attributed to simplified linear-based dynamic resampling in the prototype versus Fourier-based resampling in training.
• The hierarchical fusion approach significantly reduces energy consumption and improves robustness against the null/neutral class by filtering out non-relevant signals early in the pipeline.
• The system demonstrates feasibility for embedded deployment of deep learning models with memory footprints ranging from 11KB to 19KB on microcontrollers.
• Future integration of barometer, gas sensors, and microphones is feasible and could enhance contextual awareness for detecting stress-related eating behaviors.