[Paper Review] Human Motion Analysis with Deep Metric Learning

Huseyin Coskun, David Joseph Tan+3 more|arXiv (Cornell University)|Jul 30, 2018

Human Pose and Action Recognition28 references5 citations

TL;DR

This paper proposes a deep metric learning framework for human motion analysis that uses a novel MMD-NCA loss and attentive RNNs to learn semantically meaningful embeddings from variable-length motion sequences. By leveraging distribution-level supervision and attention mechanisms, it achieves state-of-the-art performance in action retrieval and person identification, reducing false positive rates by up to 20% compared to conventional metrics like DTW and L2.

ABSTRACT

Effectively measuring the similarity between two human motions is necessary for several computer vision tasks such as gait analysis, person identi- fication and action retrieval. Nevertheless, we believe that traditional approaches such as L2 distance or Dynamic Time Warping based on hand-crafted local pose metrics fail to appropriately capture the semantic relationship across motions and, as such, are not suitable for being employed as metrics within these tasks. This work addresses this limitation by means of a triplet-based deep metric learning specifically tailored to deal with human motion data, in particular with the prob- lem of varying input size and computationally expensive hard negative mining due to motion pair alignment. Specifically, we propose (1) a novel metric learn- ing objective based on a triplet architecture and Maximum Mean Discrepancy; as well as, (2) a novel deep architecture based on attentive recurrent neural networks. One benefit of our objective function is that it enforces a better separation within the learned embedding space of the different motion categories by means of the associated distribution moments. At the same time, our attentive recurrent neural network allows processing varying input sizes to a fixed size of embedding while learning to focus on those motion parts that are semantically distinctive. Our ex- periments on two different datasets demonstrate significant improvements over conventional human motion metrics.

Motivation & Objective

• To address the limitations of traditional motion similarity metrics like L2 and DTW, which fail to capture semantic and contextual relationships in human motion sequences.
• To design a deep metric learning framework that maps semantically similar motions close in embedding space without requiring explicit sequence alignment.
• To enable effective learning from variable-length motion sequences using an attention-augmented recurrent architecture.
• To reduce computational cost by eliminating the need for hard negative mining through distribution-based loss supervision.
• To improve performance in downstream tasks such as action retrieval and motion-based person identification.

Proposed method

• Introduces a novel MMD-NCA loss function based on Maximum Mean Discrepancy to enforce separation of motion categories by their distribution moments, avoiding hard negative mining.
• Employs an attentive recurrent neural network (RNN) to process variable-length motion sequences and generate fixed-size embeddings by focusing on semantically distinctive pose patterns.
• Uses a triplet-based architecture with a kernelized MMD objective to compare entire motion distributions rather than individual samples.
• Applies layer normalization and self-attention mechanisms to stabilize training and improve feature representation learning.
• Utilizes different kernel functions (linear, polynomial, RBF) in the MMD-NCA loss to explore the impact of higher-order statistical moments on embedding quality.
• Trains the model end-to-end using contrastive learning on triplets of motion sequences: anchor, positive (same class), and negative (different class).

Experimental results

Research questions

• RQ1 Can a deep metric learning approach based on distribution-level supervision outperform conventional motion similarity metrics like L2 and DTW in capturing semantic relationships?
• RQ2 How effective is an attention-augmented RNN architecture in learning fixed-size embeddings from variable-length human motion sequences?
• RQ3 Does eliminating hard negative mining through MMD-based loss improve training efficiency and performance in motion similarity tasks?
• RQ4 To what extent does the choice of kernel function in the MMD-NCA loss affect the quality of learned motion embeddings?
• RQ5 Can the proposed method achieve state-of-the-art performance in action retrieval and motion-based person identification?

Key findings

• The proposed MMD-NCA loss with attentive RNNs achieves a 20% reduction in false positive rate (FPR) on the CMU Mocap dataset compared to DTW, MDDTW, CTW, and GDTW.
• The method outperforms state-of-the-art deep learning baselines by 2% in FPR-70 on the CMU Mocap dataset, demonstrating superior person identification accuracy.
• Removal of the self-attention mechanism results in the largest performance drop, reducing NMI and F1 scores significantly, highlighting its critical role in capturing discriminative motion parts.
• Layer normalization and self-attention each contribute approximately 7% and 10% improvements in FPR, respectively, showing their importance in model stability and representation quality.
• The polynomial and RBF kernels in the MMD-NCA loss yield better results than linear kernels, indicating that higher-order moments of motion distributions enhance embedding quality.
• Attention visualization confirms that the model successfully focuses on semantically distinctive motion phases—such as ball-throwing in basketball or bending in gait—while ignoring less informative frames.