[Paper Review] Learning Mid-level Words on Riemannian Manifold for Action Recognition

Mengyi Liu, Ruiping Wang+2 more|arXiv (Cornell University)|Nov 16, 2015

Human Pose and Action Recognition7 references3 citations

TL;DR

This paper proposes a novel action recognition framework that constructs mid-level visual words from densely extracted low-level features using global alignment and models them on Riemannian manifolds (via subspace, covariance, and Gaussian statistics). It extends Euclidean-based encoding methods (BoVW, VLAD, FV) to Riemannian manifolds using intrinsic codebooks, achieving state-of-the-art performance on YouTube, UCF50, HMDB51, and ASLAN datasets with competitive accuracy and faster computation than prior mid-level methods.

ABSTRACT

Human action recognition remains a challenging task due to the various sources of video data and large intra-class variations. It thus becomes one of the key issues in recent research to explore effective and robust representation to handle such challenges. In this paper, we propose a novel representation approach by constructing mid-level words in videos and encoding them on Riemannian manifold. Specifically, we first conduct a global alignment on the densely extracted low-level features to build a bank of corresponding feature groups, each of which can be statistically modeled as a mid-level word lying on some specific Riemannian manifold. Based on these mid-level words, we construct intrinsic Riemannian codebooks by employing K-Karcher-means clustering and Riemannian Gaussian Mixture Model, and consequently extend the Riemannian manifold version of three well studied encoding methods in Euclidean space, i.e. Bag of Visual Words (BoVW), Vector of Locally Aggregated Descriptors (VLAD), and Fisher Vector (FV), to obtain the final action video representations. Our method is evaluated in two tasks on four popular realistic datasets: action recognition on YouTube, UCF50, HMDB51 databases, and action similarity labeling on ASLAN database. In all cases, the reported results achieve very competitive performance with those most recent state-of-the-art works.

Motivation & Objective

• To address large intra-class variations and high-dimensional video data in realistic action recognition scenarios.
• To improve representation robustness and discriminative power by introducing mid-level visual words with geometric structure.
• To model mid-level features on Riemannian manifolds to better capture intrinsic statistical diversity.
• To extend standard Euclidean encoding methods (BoVW, VLAD, FV) to Riemannian spaces for improved video representation.
• To achieve state-of-the-art performance with lower computational cost than existing mid-level representation methods.

Proposed method

• Global alignment of densely extracted low-level features (HOG, HOF, MBH) using a universal GMM to establish semantic correspondence across videos.
• Statistical modeling of aligned feature groups as mid-level words on three distinct Riemannian manifolds: linear subspaces (Grassmann), covariance matrices (SPD), and Gaussian distributions.
• Construction of intrinsic Riemannian codebooks via K-Karcher-means clustering and Riemannian Gaussian Mixture Models for manifold-aware clustering.
• Extension of three standard encoding methods—Bag of Visual Words (BoVW), Vector of Locally Aggregated Descriptors (VLAD), and Fisher Vector (FV)—to Riemannian manifolds using geodesic distances and Riemannian gradients.
• Fusion of multiple mid-level word representations (subspace, covariance, Gaussian) and descriptors (HOG, HOF, MBH) for improved robustness.
• Final video representation fed into a linear SVM for classification, with late fusion used to boost performance.

Experimental results

Research questions

• RQ1 Can global alignment of low-level features improve mid-level word construction and reduce noise from real-world video variations?
• RQ2 How do different Riemannian manifold models (Grassmann, SPD, Gaussian) enhance mid-level feature representation compared to Euclidean methods?
• RQ3 To what extent can Riemannian extensions of BoVW, VLAD, and FV improve action recognition accuracy on realistic datasets?
• RQ4 Does combining multiple mid-level word modeling strategies yield better performance than individual approaches?
• RQ5 Can the proposed method achieve state-of-the-art results with lower computational cost than existing mid-level representation techniques?

Key findings

• The proposed method achieves 90.3% accuracy on the YouTube dataset, surpassing previous state-of-the-art methods.
• On UCF50, the method reaches 90.7% accuracy, outperforming prior works including Action-Gons and Actons.
• On HMDB51, the method achieves 56.4% accuracy, significantly improving upon earlier mid-level methods like Action-parts and Motionlets.
• For action similarity labeling on ASLAN, the method achieves 65.17% accuracy and 70.29% AUC, outperforming all prior unsupervised baselines.
• Computation time is reduced to 43.9s per video on average, significantly faster than Motionlets (70s), Action-parts (227s), and Action Bank (1156s).
• Late fusion of descriptors and mid-level word models yields the best performance across all datasets, confirming the benefit of multi-modal and multi-structure representation.