[Paper Review] A Rotation Invariant Latent Factor Model for Moveme Discovery from Static Poses

Matteo Ruggero Ronchi, Joon Sik Kim+1 more|arXiv (Cornell University)|Sep 23, 2016

Human Pose and Action Recognition27 references3 citations

TL;DR

This paper proposes a rotation-invariant latent factor model that learns 3D basis poses (movemes) from 2D static images captured at arbitrary viewing angles, using a geometrically regularized matrix factorization to disentangle viewpoint variations from true motion. The method achieves rotation-invariant, interpretable, and generalizable pose representations that reconstruct unseen poses and enable applications like action recognition and dynamic inference with state-of-the-art performance.

ABSTRACT

We tackle the problem of learning a rotation invariant latent factor model when the training data is comprised of lower-dimensional projections of the original feature space. The main goal is the discovery of a set of 3-D bases poses that can characterize the manifold of primitive human motions, or movemes, from a training set of 2-D projected poses obtained from still images taken at various camera angles. The proposed technique for basis discovery is data-driven rather than hand-designed. The learned representation is rotation invariant, and can reconstruct any training instance from multiple viewing angles. We apply our method to modeling human poses in sports (via the Leeds Sports Dataset), and demonstrate the effectiveness of the learned bases in a range of applications such as activity classification, inference of dynamics from a single frame, and synthetic representation of movements.

Motivation & Objective

• To discover primitive human motion patterns (movemes) from 2D static poses captured at arbitrary camera angles.
• To develop a data-driven, rotation-invariant representation that disentangles viewpoint changes from true 3D motion variability.
• To enable reconstruction of unseen poses across different viewing angles using a shared 3D basis space.
• To support downstream applications such as activity recognition, dynamic inference, and synthetic motion generation.
• To provide an interpretable, overcomplete dictionary of 3D poses without requiring 3D ground truth or curated camera angles.

Proposed method

• A modified latent factor model performs matrix factorization on 2D joint coordinates, decomposing them into a 3D basis matrix U and coefficient matrix V.
• Geometric operations are explicitly integrated into the optimization to enforce rotation invariance by modeling viewpoint-induced transformations.
• The model uses a regularized objective that penalizes deviations from expected 2D projections under arbitrary rotations, ensuring consistent 3D basis discovery.
• The method initializes U with a small set of representative poses and optimizes both U and V jointly using an alternating least-squares approach.
• t-SNE is used to visualize the high-dimensional pose manifold, mapping coefficients from V into 2D space for interpretability.
• Heatmaps of basis pose activations are generated from V coefficients to show spatial distribution of learned movemes across the pose manifold.

Experimental results

Research questions

• RQ1 Can a data-driven model learn rotation-invariant 3D basis poses from 2D static images without 3D supervision?
• RQ2 How well can the learned basis representation reconstruct poses from unseen viewing angles?
• RQ3 To what extent does the representation improve performance in activity recognition and dynamic inference compared to non-rotation-invariant baselines?
• RQ4 How do the learned movemes align with perceptually meaningful human motion patterns such as 'tennis serve' or 'soccer kick'?
• RQ5 Can the model generalize to synthetic motion generation and manifold visualization of human motion?

Key findings

• The proposed method achieves significantly better performance in activity recognition compared to rotation-agnostic baselines, demonstrating the importance of viewpoint invariance.
• The model reconstructs training poses with high accuracy across diverse viewing angles, even for poses not in the training set, confirming generalization capability.
• t-SNE visualization shows that poses with similar motion types (e.g., 'tennis serve' and 'volleyball block') cluster together, while 'running' is spatially separated, indicating meaningful manifold structure.
• Heatmaps of basis pose activations confirm that 'volleyball strike' is most active in upper-body motion regions and 'soccer kick' in lower-body regions, aligning with intuitive motion semantics.
• The representation is robust to viewpoint variation: nearby poses in the t-SNE space can have very different camera angles, proving rotation invariance.
• The method enables synthetic motion generation by linearly combining learned bases, producing plausible intermediate poses across the motion manifold.