[Paper Review] BlazePose: On-device Real-time Body Pose tracking

Valentin Bazarevsky, Ivan Grishchenko+4 more|arXiv (Cornell University)|Jun 17, 2020

Human Pose and Action Recognition7 references86 citations

TL;DR

BlazePose presents a lightweight on-device pose estimator that outputs 33 keypoints for a single person and runs at over 30 FPS on a Pixel 2, using a detector-tracker pipeline with a heatmap-assisted regression architecture.

ABSTRACT

We present BlazePose, a lightweight convolutional neural network architecture for human pose estimation that is tailored for real-time inference on mobile devices. During inference, the network produces 33 body keypoints for a single person and runs at over 30 frames per second on a Pixel 2 phone. This makes it particularly suited to real-time use cases like fitness tracking and sign language recognition. Our main contributions include a novel body pose tracking solution and a lightweight body pose estimation neural network that uses both heatmaps and regression to keypoint coordinates.

Motivation & Objective

• Enable real-time 2D single-person pose estimation on mobile devices at high frame rates.
• Reduce model size and compute while maintaining pose accuracy for applications like fitness tracking and sign language recognition.
• Integrate heatmap supervision with a lightweight regression encoder to allow inference-time removal of the heatmap branch.
• Leverage a fast, on-device face/torso-based detector to initialize and guide pose alignment.
• Handle occlusions and pose variability through augmentation and a per-point visibility mechanism.

Proposed method

• Propose a detector-tracker pipeline where a lightweight detector initializes the pose region and a tracker network predicts keypoint coordinates and presence on each frame.
• Use a user-centric alignment strategy by leveraging a fast on-device face detector as a proxy for person localization in single-person scenarios.
• Adopt a 33-point joint topology sourced from BlazeFace, BlazePalm, and COCO to maintain consistency with related models.
• Train a combined heatmap-offset-regression network where heatmaps supervise a lightweight embedding, but inference uses only regression to coordinates.
• Discard heatmap outputs at inference to achieve real-time performance, while gradients from the regression encoder do not backpropagate into the heatmap features to improve coordinate accuracy.
• Augment data with pose priors and occlusion simulation, and include a per-point visibility classifier to handle occluded joints.

Experimental results

Research questions

• RQ1 Can a lightweight on-device model achieve real-time single-person pose estimation with a rich 33-point topology?
• RQ2 Does removing the heatmap head at inference significantly reduce compute without sacrificing accuracy?
• RQ3 Can a detector-tracker pipeline initialized by a fast face detector provide robust alignment for single-person pose tracking on mobile devices?
• RQ4 How do occlusions and pose variability affect on-device pose estimation, and can augmentation and visibility modeling mitigate these effects?
• RQ5 How does BlazePose compare to OpenPose in accuracy and speed on mobile hardware and representative datasets?

Key findings

• BlazePose achieves real-time inference (over 30 FPS) on a mid-range mobile CPU (Pixel 2).
• The model outputs 33 body keypoints for a single person and uses a combination of heatmaps and regression during training, with the heatmap branch discarded during inference for speed.
• The detector-tracker pipeline, guided by a fast on-device face detector, yields efficient single-person pose tracking.
• Compared to OpenPose, BlazePose offers competitive accuracy with substantially higher on-device speed, particularly on Yoga/Fitness poses.
• Two model variants were evaluated: BlazePose Full (higher capacity) and BlazePose Lite (more compact), with BlazePose Full outperforming the baseline OpenPose on fitness-related poses and both BlazePose versions offering significant speed advantages on mobile hardware.