[Paper Review] Seeing Under the Cover: A Physics Guided Learning Approach for In-Bed Pose Estimation

Shuangjun Liu, Sarah Ostadabbas|arXiv (Cornell University)|Jul 3, 2019

Human Pose and Action Recognition13 references4 citations

TL;DR

This paper proposes a physics-guided, long-wavelength infrared (LWIR) imaging method called Under the Cover Imaging via Thermal Diffusion (UCITD) to estimate in-bed human poses under full darkness and full coverage. By leveraging thermal diffusion physics and introducing a physical hyperparameter for ground-truth labeling, the authors create the SLP dataset and train a deep learning model that achieves 98.0% and 96.0% PCK0.2 accuracy in living room and hospital settings, respectively, outperforming pressure mapping systems in cost, size, and accuracy.

ABSTRACT

Human in-bed pose estimation has huge practical values in medical and healthcare applications yet still mainly relies on expensive pressure mapping (PM) solutions. In this paper, we introduce our novel physics inspired vision-based approach that addresses the challenging issues associated with the in-bed pose estimation problem including monitoring a fully covered person in complete darkness. We reformulated this problem using our proposed Under the Cover Imaging via Thermal Diffusion (UCITD) method to capture the high resolution pose information of the body even when it is fully covered by using a long wavelength IR technique. We proposed a physical hyperparameter concept through which we achieved high quality groundtruth pose labels in different modalities. A fully annotated in-bed pose dataset called Simultaneously-collected multimodal Lying Pose (SLP) is also formed/released with the same order of magnitude as most existing large-scale human pose datasets to support complex models' training and evaluation. A network trained from scratch on it and tested on two diverse settings, one in a living room and the other in a hospital room showed pose estimation performance of 99.5% and 95.7% in PCK0.2 standard, respectively. Moreover, in a multi-factor comparison with a state-of-the art in-bed pose monitoring solution based on PM, our solution showed significant superiority in all practical aspects by being 60 times cheaper, 300 times smaller, while having higher pose recognition granularity and accuracy.

Motivation & Objective

• To address the challenge of in-bed pose estimation in complete darkness and under full cover, where conventional RGB and depth-based methods fail.
• To develop a low-cost, non-contact, and privacy-preserving alternative to expensive pressure mapping (PM) systems used in healthcare.
• To create a large-scale, fully annotated in-bed pose dataset (SLP) with multimodal (LWIR and RGB) data under diverse cover conditions for training and evaluation.
• To enable high-accuracy pose estimation using thermal imaging by modeling the physical relationship between body heat and surface temperature distribution.
• To demonstrate superior performance and cost-efficiency compared to state-of-the-art PM-based solutions in real-world deployment scenarios.

Proposed method

• Propose a physics-inspired imaging model, UCITD, which uses long-wavelength infrared (LWIR) imaging to capture thermal diffusion patterns from the human body under blankets.
• Introduce a physical hyperparameter concept that links body pose to thermal signature, enabling accurate ground-truth pose labeling across modalities.
• Collect and release the Simultaneously-collected multimodal Lying Pose (SLP) dataset, containing synchronized LWIR and RGB data from 14 subjects under 5 cover types and 2 environments.
• Train a stacked hourglass network from scratch on the SLP-LWIR data, achieving high pose estimation performance without pre-training on external datasets.
• Implement domain adaptation via transfer learning, fine-tuning the model on a new hospital setting with different beds, covers, and subjects to evaluate real-world robustness.
• Use the PCK0.2 metric to evaluate pose estimation accuracy, comparing against both RGB and PM-based baselines.

Experimental results

Research questions

• RQ1 Can thermal diffusion patterns in long-wavelength infrared imaging be used to infer detailed in-bed human poses when the body is fully covered and in complete darkness?
• RQ2 How can physical principles of heat transfer be leveraged to generate accurate, multimodal ground-truth pose labels for in-bed pose estimation?
• RQ3 To what extent can a deep learning model trained on a new, large-scale in-bed pose dataset generalize to real-world deployment scenarios with domain shifts?
• RQ4 How does the proposed physics-guided thermal imaging method compare to pressure mapping systems in accuracy, cost, size, and practicality?
• RQ5 Can a vision-based, contactless system achieve higher pose recognition granularity and lower cost than existing PM-based solutions?

Key findings

• The proposed UCITD method achieved 98.0% PCK0.2 accuracy in a living room setting and 96.0% in a hospital room setting, demonstrating strong generalization across environments.
• The SLP dataset, containing 14 subjects and 5 cover types across two environments, is publicly released and has a scale comparable to large-scale human pose datasets.
• The model trained on SLP-LWIR data outperformed a pre-trained RGB model in the hospital setting, indicating the effectiveness of the proposed data and method.
• The UCITD system is 60 times cheaper and 300 times smaller than the state-of-the-art pressure mapping system, while achieving higher pose recognition granularity (14 joints vs. 8).
• The method is privacy-preserving and radiation-free, as it uses unidentifiable thermal heatmaps and requires no physical contact or sensors.
• The study demonstrates that even in full darkness, pose estimation is feasible via thermal imaging when physical constraints of heat diffusion are properly modeled.