[Paper Review] Muscle Vision: Real Time Keypoint Based Pose Classification of Physical Exercises

Alex Moran, Bart Gebka+4 more|arXiv (Cornell University)|Mar 23, 2022

Human Pose and Action Recognition4 citations

TL;DR

Muscle Vision proposes a real-time, keypoint-based pose classification system for fitness exercises using a lightweight machine learning pipeline and a web interface. It leverages 2D human keypoint detection from webcam feeds and time-series modeling to classify exercises with low latency, enabling live feedback for users during workouts.

ABSTRACT

Recent advances in machine learning technology have enabled highly portable and performant models for many common tasks, especially in image recognition. One emerging field, 3D human pose recognition extrapolated from video, has now advanced to the point of enabling real-time software applications with robust enough output to support downstream machine learning tasks. In this work we propose a new machine learning pipeline and web interface that performs human pose recognition on a live video feed to detect when common exercises are performed and classify them accordingly. We present a model interface capable of webcam input with live display of classification results. Our main contributions include a keypoint and time series based lightweight approach for classifying a selected set of fitness exercises and a web-based software application for obtaining and visualizing the results in real time.

Motivation & Objective

• To develop a real-time system that classifies physical exercises from live video using human pose keypoints.
• To create a lightweight, portable machine learning pipeline suitable for deployment on consumer-grade devices.
• To design a web-based interface that visualizes pose classification results in real time.
• To support practical applications in fitness tracking and form correction through accurate, low-latency classification.
• To demonstrate the feasibility of using 2D keypoint data for robust exercise recognition in real-world settings.

Proposed method

• Utilizes 2D human keypoint detection from video frames using a pre-trained pose estimation model (e.g., MediaPipe Pose).
• Extracts temporal sequences of keypoint coordinates to model motion patterns over time.
• Applies a time-series classification model (e.g., 1D CNN or Transformer-based architecture) to classify exercises from keypoint sequences.
• Optimizes model inference for low latency using model quantization and efficient architecture design.
• Deploys the system via a web-based interface using WebAssembly or similar for client-side inference on web browsers.
• Visualizes real-time classification results and pose landmarks directly on the webcam feed.

Experimental results

Research questions

• RQ1 Can a lightweight, real-time pose classification system be built using 2D keypoint data from consumer webcams?
• RQ2 How accurately can keypoint-based time-series modeling classify common resistance and bodyweight exercises?
• RQ3 To what extent does the system support real-time feedback for fitness applications?
• RQ4 What is the performance overhead and latency of deploying such a system in a web environment?
• RQ5 How effective is the web-based interface in visualizing pose and classification results in real time?

Key findings

• The system achieves real-time inference with low latency, suitable for interactive fitness applications.
• The keypoint and time-series approach enables accurate classification of a predefined set of common exercises.
• The web-based deployment allows for cross-platform accessibility without requiring native app installation.
• The model maintains high inference speed even on edge devices, thanks to model optimization techniques.
• The visualization interface effectively displays both pose landmarks and classification results in real time.
• The pipeline demonstrates feasibility for practical use in fitness tracking and form feedback systems.