[Paper Review] ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting

Di Yang, Yaohui Wang+4 more|arXiv (Cornell University)|Aug 31, 2022

Human Pose and Action Recognition4 citations

TL;DR

ViA proposes a self-supervised framework for learning view- and subject-invariant skeleton action representations by leveraging motion retargeting as a pretext task. It disentangles motion and character features via a latent motion disentanglement module, enabling state-of-the-art transfer performance on cross-view and cross-subject action recognition across real-world 2D and lab-based 3D datasets.

ABSTRACT

Current self-supervised approaches for skeleton action representation learning often focus on constrained scenarios, where videos and skeleton data are recorded in laboratory settings. When dealing with estimated skeleton data in real-world videos, such methods perform poorly due to the large variations across subjects and camera viewpoints. To address this issue, we introduce ViA, a novel View-Invariant Autoencoder for self-supervised skeleton action representation learning. ViA leverages motion retargeting between different human performers as a pretext task, in order to disentangle the latent action-specific `Motion' features on top of the visual representation of a 2D or 3D skeleton sequence. Such `Motion' features are invariant to skeleton geometry and camera view and allow ViA to facilitate both, cross-subject and cross-view action classification tasks. We conduct a study focusing on transfer-learning for skeleton-based action recognition with self-supervised pre-training on real-world data (e.g., Posetics). Our results showcase that skeleton representations learned from ViA are generic enough to improve upon state-of-the-art action classification accuracy, not only on 3D laboratory datasets such as NTU-RGB+D 60 and NTU-RGB+D 120, but also on real-world datasets where only 2D data are accurately estimated, e.g., Toyota Smarthome, UAV-Human and Penn Action.

Motivation & Objective

• Address the poor generalization of existing self-supervised skeleton action representation methods to real-world videos with diverse subjects and viewpoints.
• Improve action recognition robustness in scenarios where 3D skeleton data are unavailable or noisy, by learning invariant representations from 2D estimated skeletons.
• Enable effective transfer learning for cross-subject and cross-view action classification using only self-supervised pre-training on real-world data.
• Decouple motion-specific features from subject- and viewpoint-specific characteristics in skeleton sequences to achieve invariance.
• Demonstrate the effectiveness of motion retargeting as a pretext task for learning generic, transferable skeleton representations.

Proposed method

• Train a view- and subject-invariant autoencoder using motion retargeting between source and driving skeleton sequences as a pretext task.
• Employ a Latent Motion Disentanglement (LMD) module to decompose visual representations into 'Motion' (action-specific) and 'Character' (viewpoint/body-size-specific) components via orthogonal decomposition.
• Use a lightweight skeleton sequence decoder to reconstruct sequences by swapping 'Motion' features between source and driving sequences.
• Optimize the model using a combination of contrastive loss, cycle reconstruction loss, triplet loss, and velocity loss to enforce view-invariance and action discrimination.
• Pre-train the visual encoder on the large-scale real-world Posetics dataset to learn generic representations before fine-tuning on downstream tasks.
• Apply both linear evaluation and fine-tuning protocols to evaluate transferability on unseen 2D real-world and 3D lab-based datasets.

Experimental results

Research questions

• RQ1 Can motion retargeting serve as an effective self-supervised pretext task for learning view- and subject-invariant skeleton representations in real-world 2D data?
• RQ2 To what extent can disentangled 'Motion' features generalize across different subjects and camera viewpoints in action recognition?
• RQ3 How does self-supervised pre-training on real-world data (e.g., Posetics) improve transfer performance compared to pre-training on synthetic data?
• RQ4 What is the contribution of each loss component (self-reconstruction, cycle, triplet, velocity) to the final representation quality and downstream accuracy?
• RQ5 Can the learned representations generalize to both 2D real-world datasets and 3D lab-based datasets without retraining?

Key findings

• ViA achieves state-of-the-art performance on cross-view and cross-subject action recognition, outperforming prior self-supervised methods on both real-world (Toyota Smarthome, UAV-Human, Penn Action) and lab-based (NTU-RGB+D 60/120) datasets.
• On the Toyota Smarthome dataset with fine-tuning, ViA achieves 65.4% top-1 accuracy, a significant improvement over the baseline (61.7%) and prior methods.
• The ablation study shows that adding cycle loss (+1.1%) and triplet loss (+2.1%) to the baseline yields the largest performance gains, highlighting their role in enhancing view-invariance.
• The addition of velocity loss provides a minor but consistent boost (0.4% accuracy gain), indicating improved temporal consistency in motion representation.
• Qualitative results confirm effective disentanglement: 'Motion' features represent a canonical, view-invariant motion regardless of input viewpoint, while 'Character' features control body size and viewpoint.
• t-SNE visualization on Mixamo confirms that 'Motion' features cluster by action type regardless of subject or viewpoint, validating the disentanglement and invariance of the learned representation.