[Paper Review] Unsupervised Learning for Human Sensing Using Radio Signals

Tianhong Li, Lijie Fan+2 more|arXiv (Cornell University)|Jul 6, 2022

Indoor and Outdoor Localization Technologies4 citations

TL;DR

This paper proposes Trajectory-Guided Unsupervised Learning (TGUL), a novel framework that adapts predictive unsupervised learning to RF signals for human sensing, overcoming limitations of contrastive learning. By leveraging radar-based person detection and task-specific data augmentations, TGUL learns high-quality representations from unlabeled RF data, achieving state-of-the-art performance across multiple downstream tasks like 3D pose estimation and action recognition.

ABSTRACT

There is a growing literature demonstrating the feasibility of using Radio Frequency (RF) signals to enable key computer vision tasks in the presence of occlusions and poor lighting. It leverages that RF signals traverse walls and occlusions to deliver through-wall pose estimation, action recognition, scene captioning, and human re-identification. However, unlike RGB datasets which can be labeled by human workers, labeling RF signals is a daunting task because such signals are not human interpretable. Yet, it is fairly easy to collect unlabelled RF signals. It would be highly beneficial to use such unlabeled RF data to learn useful representations in an unsupervised manner. Thus, in this paper, we explore the feasibility of adapting RGB-based unsupervised representation learning to RF signals. We show that while contrastive learning has emerged as the main technique for unsupervised representation learning from images and videos, such methods produce poor performance when applied to sensing humans using RF signals. In contrast, predictive unsupervised learning methods learn high-quality representations that can be used for multiple downstream RF-based sensing tasks. Our empirical results show that this approach outperforms state-of-the-art RF-based human sensing on various tasks, opening the possibility of unsupervised representation learning from this novel modality.

Motivation & Objective

• Address the challenge of limited labeled RF data, which is difficult to collect due to privacy and interpretability constraints.
• Overcome the inapplicability of standard RGB-based unsupervised learning methods to RF signals due to signal sparsity, background noise, and lack of color or rotational invariance.
• Develop a framework that focuses unsupervised learning on human-relevant signal components by using radar-based trajectory detection to guide representation learning.
• Demonstrate that predictive self-supervised learning (e.g., inpainting) outperforms contrastive learning on RF signals, which tends to exploit shortcut signals like distance to device.
• Enable effective use of large-scale unlabeled RF datasets to improve downstream human sensing performance, even with minimal labeled data.

Proposed method

• Introduce a radar-based module to detect human trajectories in raw RF signals, enabling localization of human-relevant signal components.
• Design RF-specific data augmentations, including frame masking strategies (e.g., random segments of 5 frames) to preserve spatiotemporal structure.
• Adopt a predictive self-supervised learning objective—specifically, inpainting masked frames in short RF sequences—to learn meaningful representations.
• Use the detected human trajectory to guide data augmentation and representation learning, reducing interference from background reflections.
• Pre-train a feature encoder using the predictive task on large-scale unlabeled RF data, then fine-tune on small labeled datasets for downstream tasks.
• Apply the framework to multiple RF datasets (e.g., RF-MMD) and tasks, including 3D pose estimation, action recognition, and re-identification.

Figure 1: Illustration of RF signals and corresponding RGB images (for illustration ONLY ). The signal reflected off the person occupies a small region in the received RF signal. Further, there are many other reflections from other objects.

Experimental results

Research questions

• RQ1 Can predictive self-supervised learning methods be effectively adapted to RF signals for human sensing, given the modality's unique challenges?
• RQ2 Why does contrastive learning fail on RF signals, and what biases does it introduce compared to predictive methods?
• RQ3 How do different masking strategies and segment sizes in inpainting affect representation quality for RF-based human sensing?
• RQ4 To what extent can unsupervised pre-training on large unlabeled RF datasets improve performance on downstream tasks with limited labeled data?
• RQ5 Can trajectory-guided unsupervised learning focus representation learning on human-relevant signal components and suppress background noise?

Key findings

• Predictive unsupervised learning, particularly inpainting with random 5-frame segments, outperforms contrastive learning on RF signals, reducing 3D pose estimation error by 5.3% compared to training from scratch.
• The best masking strategy for inpainting is random segments of 5 frames, achieving a pose error of 36.2 mm on the RF-MMD dataset, outperforming center-segment and random masking.
• Contrastive learning produces poor representations that discard most human-relevant information, as visualized in feature maps, while predictive learning preserves key human features.
• Pre-training on the full RF-MMD dataset (unlabeled) and fine-tuning on a 10% labeled subset (RF-MMD-S) reduces pose error from 48.7 mm (random training) to 43.2 mm, demonstrating 6.0% improvement with unlabeled data.
• The framework improves generalization and downstream performance even with minimal labeled data, showing that unsupervised pre-training on large-scale unlabeled RF data is highly effective.
• The use of radar-based trajectory detection enables the model to focus on human-relevant signal components, mitigating interference from background reflections and improving representation quality.

Figure 2: Illustration of RF signals as a pair of horizontal and vertical heatmaps after subtracting static objects, and an RGB image recorded at the same time. Red color refers to high signal power, blue refers to low power.