[Paper Review] Person-in-WiFi: Fine-grained Person Perception using WiFi

Fei Wang, Sanping Zhou+3 more|arXiv (Cornell University)|Mar 30, 2019

Indoor and Outdoor Localization TechnologiesEngineering52 references17 citations

TL;DR

This paper presents Person-in-WiFi, the first system to achieve fine-grained person perception—body segmentation and pose estimation—using only off-the-shelf WiFi antennas as 1D sensors. By leveraging deep learning on 1D Channel State Information (CSI) signals from multiple WiFi transceiver pairs and supervised by 2D image annotations, it achieves performance comparable to camera-based methods, demonstrating that 1D EM wave signals can reconstruct detailed human body spatial structures despite severe ill-posedness.

ABSTRACT

Fine-grained person perception such as body segmentation and pose estimation has been achieved with many 2D and 3D sensors such as RGB/depth cameras, radars (e.g., RF-Pose) and LiDARs. These sensors capture 2D pixels or 3D point clouds of person bodies with high spatial resolution, such that the existing Convolutional Neural Networks can be directly applied for perception. In this paper, we take one step forward to show that fine-grained person perception is possible even with 1D sensors: WiFi antennas. To our knowledge, this is the first work to perceive persons with pervasive WiFi devices, which is cheaper and power efficient than radars and LiDARs, invariant to illumination, and has little privacy concern comparing to cameras. We used two sets of off-the-shelf WiFi antennas to acquire signals, i.e., one transmitter set and one receiver set. Each set contains three antennas lined-up as a regular household WiFi router. The WiFi signal generated by a transmitter antenna, penetrates through and reflects on human bodies, furniture and walls, and then superposes at a receiver antenna as a 1D signal sample (instead of 2D pixels or 3D point clouds). We developed a deep learning approach that uses annotations on 2D images, takes the received 1D WiFi signals as inputs, and performs body segmentation and pose estimation in an end-to-end manner. Experimental results on over 100000 frames under 16 indoor scenes demonstrate that Person-in-WiFi achieved person perception comparable to approaches using 2D images.

Motivation & Objective

• To enable fine-grained person perception—such as body segmentation and pose estimation—using only off-the-shelf WiFi antennas, which are cheaper, more power-efficient, and privacy-friendly than cameras, radars, or LiDARs.
• To address the challenge of reconstructing 2D spatial details of human bodies from 1D WiFi Channel State Information (CSI) signals, which are inherently ambiguous due to multipath propagation, body heterogeneity, and physiological motion.
• To develop a deep learning framework that maps 1D CSI signals to 2D human body masks and joint coordinates using annotations from 2D RGB videos, enabling end-to-end training.
• To evaluate the feasibility and performance of WiFi-based perception in real-world indoor environments, including generalization to untrained scenes.
• To explore data augmentation and adversarial training techniques to improve robustness to environmental variations and improve generalization in unseen settings.

Proposed method

• The system uses two sets of three off-the-shelf WiFi antennas (one transmitter, one receiver), arranged like a standard router, to capture 1D CSI signals across 30 frequencies centered at 2.4 GHz.
• Each CSI signal is a superposition of reflected waves from the human body, furniture, and walls, forming a 1D time-series signal per antenna pair.
• A deep learning model is trained end-to-end using 2D image annotations (body masks and joint coordinates from OpenPose and Mask R-CNN) as supervision, with 1D CSI signals as input.
• The model employs a U-Net-based architecture for segmentation and a heatmap regression head for keypoint estimation, jointly optimizing both tasks.
• To improve robustness to environmental shifts, the authors introduce a GAN-based adversarial training strategy where a generator network learns to produce environment-invariant CSI representations.
• The system is evaluated on over 100,000 frames across 16 indoor scenes, with performance compared to camera-based baselines using mIoU and mPCK metrics.

Experimental results

Research questions

• RQ1 Can fine-grained human body perception (segmentation and pose) be achieved using only 1D WiFi signals, despite the severe ill-posedness of reconstructing 2D spatial structure from 1D amplitude data?
• RQ2 How does the performance of WiFi-based person perception compare to camera-based state-of-the-art methods in terms of segmentation accuracy and pose estimation precision?
• RQ3 To what extent can a deep learning model trained on 2D image annotations generalize to 1D WiFi CSI signals, and what are the main failure modes?
• RQ4 Can adversarial training and data augmentation improve the generalization of the model to untrained indoor environments with different layouts and materials?
• RQ5 What are the key limitations of WiFi-based perception, particularly regarding small limbs, rare poses, and occlusions, and how can they be mitigated?

Key findings

• Person-in-WiFi achieved a mean Intersection over Union (mIoU) of 0.66 for body segmentation and a mean PCK@0.20 of 78.75% for pose estimation, demonstrating performance comparable to camera-based methods.
• The system achieved mIoU of 0.66 and mPCK@0.20 of 78.75% on a test set of 160 uniformly sampled frames, indicating strong generalization to diverse indoor scenes.
• Adversarial training for environment invariance improved mIoU from 0.12 to 0.24 and mPCK@0.20 from 19.34% to 31.06% in untrained environments, showing promise for real-world deployment.
• Failure cases were primarily due to low spatial resolution (e.g., missing small limbs due to diffraction at 12.5 cm wavelength), rare poses, and incomplete annotations from single-camera field-of-view limitations.
• The model performs better on larger body parts (e.g., torso and arms) than on small or occluded parts (e.g., head and feet), with lower PCK scores for joints in the head and feet groups.
• The results suggest that with more high-quality data and improved data augmentation, the performance gap between WiFi-based and camera-based perception can be further reduced.