[Paper Review] MiliPoint: A Point Cloud Dataset for mmWave Radar

Han Cui, Zhong Shu+4 more|arXiv (Cornell University)|Sep 23, 2023

Remote Sensing and LiDAR Applications4 citations

TL;DR

MiliPoint introduces a large-scale, open-source point cloud dataset for mmWave radar, comprising 545K frames and 49 diverse human actions, enabling benchmarking of deep learning models across three key tasks: action classification, identity recognition, and keypoint estimation. The dataset is the largest and most action-diverse to date, supporting standardized evaluation of point-based deep neural networks like DGCNN, PointNet++, and PointTransformer, with action classification identified as the most challenging task.

ABSTRACT

Millimetre-wave (mmWave) radar has emerged as an attractive and cost-effective alternative for human activity sensing compared to traditional camera-based systems. mmWave radars are also non-intrusive, providing better protection for user privacy. However, as a Radio Frequency (RF) based technology, mmWave radars rely on capturing reflected signals from objects, making them more prone to noise compared to cameras. This raises an intriguing question for the deep learning community: Can we develop more effective point set-based deep learning methods for such attractive sensors? To answer this question, our work, termed MiliPoint, delves into this idea by providing a large-scale, open dataset for the community to explore how mmWave radars can be utilised for human activity recognition. Moreover, MiliPoint stands out as it is larger in size than existing datasets, has more diverse human actions represented, and encompasses all three key tasks in human activity recognition. We have also established a range of point-based deep neural networks such as DGCNN, PointNet++ and PointTransformer, on MiliPoint, which can serve to set the ground baseline for further development.

Motivation & Objective

• To address the lack of standardized, large-scale datasets for mmWave radar-based human activity recognition.
• To enable fair comparison across deep learning models by providing a unified benchmarking framework.
• To support research in point set-based deep learning for noisy, RF-based radar data.
• To cover all three core tasks in human activity recognition: identification, action classification, and keypoint estimation.
• To establish baseline performance for state-of-the-art point-based DNNs on mmWave radar data.

Proposed method

• The dataset was collected using a single 77GHz MIMO mmWave radar with three transmitters and four receivers.
• Raw intermediate frequency (IF) signals were processed via range-Doppler and angle-FFT to generate 3D point clouds representing human motion.
• Each point in the cloud encodes range, Doppler, and azimuth angle, forming a 3D spatial representation of human movements.
• 49 distinct human actions were performed by subjects under controlled indoor conditions, ensuring consistent data collection.
• The dataset includes annotations for identity, action class, and 3D keypoint locations for each frame.
• Standardized data splits and evaluation protocols were established to support reproducible benchmarking of deep learning models.

Figure 1: An illustration of how a typical mmWave Radar work. The radar has several transmitters (TX) and Receivers (RX) for transmitting and collecting the reflected signals. These signals are then mixed and filtered to form an Intermediate Frequency (IF) signal. Subsequent to this, three Fast Four

Experimental results

Research questions

• RQ1 Can point-based deep learning models effectively recognize human actions from mmWave radar point clouds?
• RQ2 How does model performance vary across the three core tasks: identity recognition, action classification, and keypoint estimation?
• RQ3 What is the relative difficulty of action classification compared to identity recognition and keypoint estimation on mmWave radar data?
• RQ4 How does dataset size and action diversity impact the generalization and robustness of deep learning models in radar-based human activity recognition?
• RQ5 To what extent do existing point-based DNNs like DGCNN, PointNet++, and PointTransformer generalize to mmWave radar point clouds?

Key findings

• MiliPoint is the largest mmWave radar dataset to date, with 545,000 frames—3.26 times larger than the next largest dataset.
• The dataset includes 49 distinct human actions, representing a 4.08-fold increase in action diversity compared to the most action-diverse existing dataset.
• Action classification emerged as the most challenging task among the three evaluated, indicating higher difficulty in distinguishing fine-grained motion patterns.
• Point-based deep neural networks such as DGCNN, PointNet++, and PointTransformer achieved measurable performance on MiliPoint, establishing a baseline for future research.
• The dataset enables standardized evaluation across models and tasks, reducing inconsistencies in prior mmWave radar studies due to varying hardware, data collection, and task definitions.
• The study highlights the potential of mmWave radar as a privacy-preserving, cost-effective alternative to camera-based systems for human activity sensing.

(a) Front view