[Paper Review] Augmenting Vision-Based Human Pose Estimation with Rotation Matrix

Milad Vazan, Fatemeh Sadat Masoumi+2 more|arXiv (Cornell University)|Oct 9, 2023

Human Pose and Action RecognitionComputer Science3 citations

TL;DR

This paper proposes a rotation matrix-based data augmentation technique to improve vision-based human pose estimation for physical activity classification. By generating 360° view variations from a single camera input using rotation matrices, the method enhances pose data diversity, enabling a 96% accuracy with SVM-SGD on five fitness activities—significantly outperforming the 64% baseline without augmentation.

ABSTRACT

Fitness applications are commonly used to monitor activities within the gym, but they often fail to automatically track indoor activities inside the gym. This study proposes a model that utilizes pose estimation combined with a novel data augmentation method, i.e., rotation matrix. We aim to enhance the classification accuracy of activity recognition based on pose estimation data. Through our experiments, we experiment with different classification algorithms along with image augmentation approaches. Our findings demonstrate that the SVM with SGD optimization, using data augmentation with the Rotation Matrix, yields the most accurate results, achieving a 96% accuracy rate in classifying five physical activities. Conversely, without implementing the data augmentation techniques, the baseline accuracy remains at a modest 64%.

Motivation & Objective

• Address the challenge of limited, single-view labeled data in vision-based human activity recognition (HAR) for fitness applications.
• Overcome ambiguity in pose estimation caused by fixed camera angles and lack of multi-perspective training samples.
• Improve classification accuracy of physical activities using pose keypoint data derived from video streams.
• Evaluate the effectiveness of rotation matrix-based data augmentation on pose estimation features for HAR.
• Identify the optimal combination of pose estimation model, data augmentation, and classifier for multi-class fitness activity recognition.

Proposed method

• Employ BlazePose for accurate 2D human keypoint detection from single-view video frames.
• Apply rotation matrix transformations to generate synthetic training samples at 180° clockwise and counterclockwise orientations from each original pose.
• Use rotation matrices to simulate diverse camera perspectives, increasing data variability without additional data collection.
• Train and compare multiple machine learning classifiers (e.g., SVM-SGD, Logistic Regression, KNN) on augmented and non-augmented pose keypoint data.
• Optimize model performance by selectively removing redundant or noisy keypoint features (e.g., right shoulder) to improve classification accuracy.
• Evaluate model performance using 5-class physical activity classification on a limited but representative dataset of gym exercises.

Experimental results

Research questions

• RQ1 Can rotation matrix-based data augmentation improve the generalization and accuracy of pose-based human activity recognition models?
• RQ2 How does the inclusion of synthetic multi-perspective views via rotation matrices affect the performance of different classification algorithms on pose keypoint data?
• RQ3 Which machine learning classifier achieves the highest accuracy in classifying five distinct physical activities using augmented pose features?
• RQ4 Does removing specific body keypoint features (e.g., right shoulder) improve classification performance by reducing noise or redundancy?
• RQ5 How does the proposed method compare to state-of-the-art HAR models in terms of accuracy and data efficiency?

Key findings

• The SVM with SGD optimization (SVM-SGD) achieved the highest classification accuracy of 96% when combined with rotation matrix data augmentation.
• Without any data augmentation, the baseline accuracy for activity classification was only 64%, highlighting the critical role of augmentation.
• Removing the right shoulder keypoint from the 12-keypoint pose representation improved SVM-SGD performance, achieving 96% accuracy with 11 keypoint features.
• The rotation matrix augmentation method generated 360° synthetic views per original sample, effectively simulating multi-perspective training data from a single camera source.
• Among all evaluated classifiers, SVM-SGD showed the most consistent and superior performance across different keypoint configurations and augmentation strategies.
• The proposed method achieved 96% accuracy on five fitness activities using a minimal number of training samples, outperforming prior work that required significantly more data.