[Paper Review] mmFall: Fall Detection using 4D MmWave Radar and a Hybrid Variational RNN AutoEncoder

Feng Jin, Arindam Sengupta+1 more|arXiv (Cornell University)|Mar 5, 2020

Indoor and Outdoor Localization Technologies54 references4 citations

TL;DR

This paper proposes mmFall, a privacy-preserving fall detection system using 4D mmWave radar and a Hybrid Variational RNN Autoencoder (HVRAE) that detects falls by analyzing motion anomaly levels and centroid height drops. The method achieves 98% detection accuracy on 50 falls with only 2 false alarms, leveraging semi-supervised training on normal activities to overcome the scarcity of fall data.

ABSTRACT

In this paper we propose mmFall - a novel fall detection system, which comprises of (i) the emerging millimeter-wave (mmWave) radar sensor to collect the human body's point cloud along with the body centroid, and (ii) a variational recurrent autoencoder (VRAE) to compute the anomaly level of the body motion based on the acquired point cloud. A fall is claimed to have occurred when the spike in anomaly level and the drop in centroid height occur simultaneously. The mmWave radar sensor provides several advantages, such as privacycompliance and high-sensitivity to motion, over the traditional sensing modalities. However, (i) randomness in radar point cloud data and (ii) difficulties in fall collection/labeling in the traditional supervised fall detection approaches are the two main challenges. To overcome the randomness in radar data, the proposed VRAE uses variational inference, a probabilistic approach rather than the traditional deterministic approach, to infer the posterior probability of the body's latent motion state at each frame, followed by a recurrent neural network (RNN) to learn the temporal features of the motion over multiple frames. Moreover, to circumvent the difficulties in fall data collection/labeling, the VRAE is built upon an autoencoder architecture in a semi-supervised approach, and trained on only normal activities of daily living (ADL) such that in the inference stage the VRAE will generate a spike in the anomaly level once an abnormal motion, such as fall, occurs. During the experiment, we implemented the VRAE along with two other baselines, and tested on the dataset collected in an apartment. The receiver operating characteristic (ROC) curve indicates that our proposed model outperforms the other two baselines, and achieves 98% detection out of 50 falls at the expense of just 2 false alarms.

Motivation & Objective

• Address the critical need for effective, privacy-compliant fall detection in aging populations.
• Overcome the challenge of limited real-world fall data collection by using a semi-supervised approach.
• Develop a robust anomaly detection system that identifies falls without requiring labeled fall samples during training.
• Integrate mmWave radar's high motion sensitivity with temporal modeling to detect abrupt changes indicative of falls.
• Ensure practical deployment by minimizing false alarms and maintaining performance across diverse body types and movements.

Proposed method

• Utilizes a 4D mmWave radar sensor to capture dynamic point clouds (3D position + Doppler velocity) of human motion in real time.
• Employs variational inference within a variational autoencoder to model the posterior distribution of latent motion states from radar point clouds.
• Integrates a recurrent neural network (RNN) to capture temporal dependencies across sequential radar frames for motion pattern analysis.
• Designs a hybrid architecture combining variational autoencoding with RNN-based sequence modeling to compute anomaly scores per frame.
• Applies a custom loss function and data oversampling technique to stabilize training on limited normal activity data.
• Detects falls as simultaneous spikes in anomaly level and drops in body centroid height, enabling real-time alerting.

Experimental results

Research questions

• RQ1 Can a semi-supervised anomaly detection model trained solely on normal activities detect falls without requiring labeled fall data?
• RQ2 How effective is a hybrid variational RNN autoencoder in modeling complex human motion patterns from mmWave radar point clouds?
• RQ3 To what extent does mmWave radar outperform traditional vision-based sensors in fall detection while preserving privacy?
• RQ4 Can the combination of variational inference and RNNs improve detection robustness against noise and randomness in radar data?
• RQ5 What is the trade-off between detection sensitivity and false alarm rate in real-world fall detection using radar and deep learning?

Key findings

• The proposed HVRAE model achieved 98% fall detection rate across 50 real falls in a residential environment with only 2 false alarms.
• The system demonstrated high sensitivity to motion changes, detecting falls through a spike in anomaly level and concurrent drop in centroid height.
• Semi-supervised training on normal activities alone enabled effective detection of abnormal events like falls, eliminating the need for costly and ethically challenging fall data collection.
• The use of variational inference allowed robust estimation of latent motion states despite inherent randomness in mmWave radar point clouds.
• The custom data oversampling method preserved the statistical properties of the original dataset, ensuring stable model training without distribution shift.
• The ROC curve analysis confirmed that mmFall outperformed two baseline models in both detection rate and false alarm control.