[Paper Review] Advancing Smart Malnutrition Monitoring: A Multi-Modal Learning Approach for Vital Health Parameter Estimation

Ashish Marisetty, Prathistith Raj M+3 more|arXiv (Cornell University)|Jul 31, 2023

Nutrition and Health in AgingMedicine3 citations

TL;DR

This paper proposes a multi-modal learning framework that estimates height, weight, and vital health parameters (BMI, BMR, BFP) from a single full-body image using 3D reconstruction and fused 2D facial/body embeddings. It achieves state-of-the-art performance with a mean absolute error of ±4.7 cm for height and ±5.3 kg for weight, enabling smartphone-based, non-invasive malnutrition monitoring in resource-limited settings.

ABSTRACT

Malnutrition poses a significant threat to global health, resulting from an inadequate intake of essential nutrients that adversely impacts vital organs and overall bodily functioning. Periodic examinations and mass screenings, incorporating both conventional and non-invasive techniques, have been employed to combat this challenge. However, these approaches suffer from critical limitations, such as the need for additional equipment, lack of comprehensive feature representation, absence of suitable health indicators, and the unavailability of smartphone implementations for precise estimations of Body Fat Percentage (BFP), Basal Metabolic Rate (BMR), and Body Mass Index (BMI) to enable efficient smart-malnutrition monitoring. To address these constraints, this study presents a groundbreaking, scalable, and robust smart malnutrition-monitoring system that leverages a single full-body image of an individual to estimate height, weight, and other crucial health parameters within a multi-modal learning framework. Our proposed methodology involves the reconstruction of a highly precise 3D point cloud, from which 512-dimensional feature embeddings are extracted using a headless-3D classification network. Concurrently, facial and body embeddings are also extracted, and through the application of learnable parameters, these features are then utilized to estimate weight accurately. Furthermore, essential health metrics, including BMR, BFP, and BMI, are computed to conduct a comprehensive analysis of the subject's health, subsequently facilitating the provision of personalized nutrition plans. While being robust to a wide range of lighting conditions across multiple devices, our model achieves a low Mean Absolute Error (MAE) of $\pm$ 4.7 cm and $\pm$ 5.3 kg in estimating height and weight.

Motivation & Objective

• To address the limitations of conventional malnutrition screening methods, which require specialized equipment and are impractical in remote or pandemic-affected areas.
• To develop a non-invasive, smartphone-deployable system for estimating key health parameters like BMI, BMR, and BFP from a single full-body image.
• To overcome the lack of holistic feature representation and robustness to lighting variations in existing approaches.
• To enable real-time, autonomous health parameter estimation on edge devices without reliance on external sensors or infrastructure.
• To provide personalized nutrition plans through accurate, data-driven health metric inference from visual input alone.

Proposed method

• Reconstructs a high-precision 3D point cloud from a single full-body image using deep learning-based 3D reconstruction.
• Extracts 512-dimensional 3D feature embeddings using a headless 3D classification network trained on the reconstructed point cloud.
• Simultaneously extracts 2D facial and body embeddings from the same image using convolutional neural networks.
• Fuses 3D, facial, and body embeddings using learnable parameters to improve weight estimation accuracy.
• Computes derived health metrics—Body Mass Index (BMI), Basal Metabolic Rate (BMR), and Body Fat Percentage (BFP)—from estimated height and weight.
• Deploys the model on an edge device prototype to enable real-time, on-device inference without external sensors or internet dependency.

Figure 1: Conceptual Overview

Experimental results

Research questions

• RQ1 Can a single full-body image enable accurate estimation of height and weight using multi-modal feature fusion?
• RQ2 How does the integration of 3D point cloud features with 2D facial and body embeddings improve weight estimation performance compared to single-modality approaches?
• RQ3 To what extent is the proposed system robust to variations in lighting and device types in real-world deployment scenarios?
• RQ4 Can the system achieve real-time, autonomous inference on edge devices without requiring additional hardware or infrastructure?
• RQ5 How does the model’s performance in estimating BMI, BMR, and BFP compare to existing methods in non-invasive malnutrition monitoring?

Key findings

• The proposed method achieves a mean absolute error (MAE) of ±4.7 cm for height estimation and ±5.3 kg for weight estimation, outperforming prior works.
• The model demonstrates robustness to diverse lighting conditions and multiple device types, enabling reliable deployment in real-world settings.
• The use of learnable fusion parameters for multi-modal features significantly improves weight estimation accuracy, achieving the lowest reported MAE of 5.3 kg in the literature.
• The system operates autonomously on edge devices, eliminating the need for external sensors or infrastructure, which is critical for remote and low-resource environments.
• The derived health metrics—BMI, BMR, and BFP—are computed accurately from estimated height and weight, enabling comprehensive malnutrition risk assessment.
• The edge-deployed prototype enables real-time, on-device health parameter estimation, supporting scalable and privacy-preserving smart malnutrition monitoring.

Figure 2: Proposed System Overview