[Paper Review] Video Human Segmentation using Fuzzy Object Models and its Application to Body Pose Estimation of Toddlers for Behavior Studies

Thiago V. Spina, Mariano Tepper+5 more|arXiv (Cornell University)|May 29, 2013

Human Pose and Action Recognition29 references3 citations

TL;DR

This paper proposes a semi-automatic video segmentation method using an extended Cloud System Model (CSM) with a 2D stickman to jointly segment human body parts and estimate 2D pose in video, enabling accurate detection of arm asymmetry during gait—a potential behavioral marker of autism in toddlers. The approach achieves high clinical relevance by reducing manual effort to a single-frame initialization while maintaining strong correlation with expert assessments.

ABSTRACT

Video object segmentation is a challenging problem due to the presence of deformable, connected, and articulated objects, intra- and inter-object occlusions, object motion, and poor lighting. Some of these challenges call for object models that can locate a desired object and separate it from its surrounding background, even when both share similar colors and textures. In this work, we extend a fuzzy object model, named cloud system model (CSM), to handle video segmentation, and evaluate it for body pose estimation of toddlers at risk of autism. CSM has been successfully used to model the parts of the brain (cerebrum, left and right brain hemispheres, and cerebellum) in order to automatically locate and separate them from each other, the connected brain stem, and the background in 3D MR-images. In our case, the objects are articulated parts (2D projections) of the human body, which can deform, cause self-occlusions, and move along the video. The proposed CSM extension handles articulation by connecting the individual clouds, body parts, of the system using a 2D stickman model. The stickman representation naturally allows us to extract 2D body pose measures of arm asymmetry patterns during unsupported gait of toddlers, a possible behavioral marker of autism. The results show that our method can provide insightful knowledge to assist the specialist's observations during real in-clinic assessments.

Motivation & Objective

• To develop a robust method for segmenting articulated human bodies in video, particularly in challenging conditions such as occlusions and poor lighting.
• To extend the Cloud System Model (CSM) framework to handle 2D articulated bodies using a relational stickman model to maintain part connectivity.
• To enable accurate, semi-automatic 2D body pose estimation from video for clinical applications in early autism spectrum disorder (ASD) detection.
• To quantify and detect stereotypical motor behaviors, such as arm asymmetry during unsupported gait, as potential behavioral markers of ASD in toddlers.

Proposed method

• The Cloud System Model (CSM) is extended to represent each human body part (e.g., head, torso, limbs) as a fuzzy object cloud, capturing shape variations and uncertainty in boundary locations.
• A 2D stickman model is used to connect the individual body-part clouds, encoding articulation and kinematic constraints to maintain anatomical plausibility across frames.
• Multi-scale search is employed to optimize the parameters of the stickman model, aligning the CSM clouds with the true body parts in each video frame.
• Segmentation is performed by evaluating the CSM’s uncertainty regions at each search position, using shape, color, and texture cues to guide delineation.
• Pose estimation is derived from the optimized stickman configuration, with asymmetry scores computed for upper arms and forearms to detect abnormal motor patterns.
• The method requires only a single interactive initialization frame, after which segmentation and pose estimation are automatically propagated through the video sequence.

Experimental results

Research questions

• RQ1 Can a fuzzy object model like CSM be effectively extended to handle 2D articulated human bodies in video, despite deformations and occlusions?
• RQ2 To what extent can the CSM with a stickman relational model enable accurate and robust simultaneous segmentation and 2D pose estimation in real-world video sequences of toddlers?
• RQ3 How well does the computed arm asymmetry score correlate with clinical expert ratings of symmetry during unsupported gait in toddlers?
• RQ4 Can the method detect subtle, potentially stereotypical motor behaviors such as asymmetric arm movement or parallel forearm positioning, which may serve as early markers of ASD?
• RQ5 Is the semi-automatic nature of the method—requiring only one initialization frame—clinically practical for use in in-clinic assessments?

Key findings

• The proposed method achieved high accuracy in segmenting human bodies across video frames, successfully handling self-occlusions, articulation, and background clutter.
• The computed arm asymmetry scores showed strong correlation with expert visual ratings, demonstrating clinical validity and reliability.
• The system significantly reduced manual effort by requiring only a single-frame initialization, after which pose and segmentation were automatically propagated through the video.
• The method detected abnormal gait and persistent arm asymmetry in a toddler later diagnosed with autism, supporting its potential as a diagnostic aid.
• False positives were observed in some cases, such as participant #2, where forearm angle differences suggested asymmetry that was not confirmed by the final asymmetry score, indicating room for refinement.
• The approach successfully captured and quantified stereotypical motor behaviors such as parallel forearm positioning, suggesting utility for detecting a broader range of ASD-related motor patterns.