[Paper Review] Multiple Riemannian Manifold-valued Descriptors based Image Set Classification with Multi-Kernel Metric Learning

Rui Wang, Xiao‐Jun Wu+1 more|View|Aug 6, 2019

Face and Expression Recognition4 citations

TL;DR

This paper proposes a multi-geometric image set classification method that jointly models image sets using covariance matrices, linear subspaces, and Gaussian distributions on Riemannian manifolds. By leveraging Riemannian kernels to map these heterogeneous features into Hilbert spaces and employing a multi-kernel metric learning framework to embed them into a lower-dimensional discriminative subspace, the approach achieves state-of-the-art performance on four video-based classification tasks, demonstrating the complementary value of multi-manifold descriptors.

ABSTRACT

The importance of wild video based image set recognition is becoming monotonically increasing. However, the contents of these collected videos are often complicated, and how to efficiently perform set modeling and feature extraction is a big challenge for set-based classification algorithms. In recent years, some proposed image set classification methods have made a considerable advance by modeling the original image set with covariance matrix, linear subspace, or Gaussian distribution. As a matter of fact, most of them just adopt a single geometric model to describe each given image set, which may lose some other useful information for classification. To tackle this problem, we propose a novel algorithm to model each image set from a multi-geometric perspective. Specifically, the covariance matrix, linear subspace, and Gaussian distribution are applied for set representation simultaneously. In order to fuse these multiple heterogeneous Riemannian manifoldvalued features, the well-equipped Riemannian kernel functions are first utilized to map them into high dimensional Hilbert spaces. Then, a multi-kernel metric learning framework is devised to embed the learned hybrid kernels into a lower dimensional common subspace for classification. We conduct experiments on four widely used datasets corresponding to four different classification tasks: video-based face recognition, set-based object categorization, video-based emotion recognition, and dynamic scene classification, to evaluate the classification performance of the proposed algorithm. Extensive experimental results justify its superiority over the state-of-the-art.

Motivation & Objective

• To address the limitation of single-geometry modeling in image set classification, which may overlook complementary information from diverse manifold structures.
• To improve robustness and discriminative power in video-based recognition by fusing multiple Riemannian manifold-valued features derived from covariance matrices, subspaces, and Gaussian distributions.
• To develop a unified framework that effectively combines heterogeneous Riemannian features through kernel mapping and metric learning for enhanced classification performance.
• To validate the superiority of multi-geometric representation over single-geometry approaches across diverse video-based classification benchmarks.

Proposed method

• Models each image set using three distinct Riemannian manifold-valued descriptors: symmetric positive definite (SPD) matrices (covariance), Grassmannian subspaces (linear subspaces), and Gaussian distributions (Gaussian-embedded manifold).
• Applies well-established Riemannian kernel functions to map the heterogeneous manifold-valued features into high-dimensional Hilbert spaces, preserving intrinsic geometric structure.
• Designs a multi-kernel metric learning framework that learns a shared, lower-dimensional Euclidean subspace by combining multiple Riemannian kernels, enabling effective classification.
• Uses an iterative optimization strategy to jointly learn the transformation matrix, kernel weights, and regularization parameters, with convergence empirically validated on AFEW and YTC datasets.
• Employs trace ratio optimization for subspace learning and gradient ascent for kernel weight updates, ensuring stable convergence of the objective function.
• Configures the final feature dimension $d_w$ through ablation, identifying optimal values (e.g., 25 on MDSD, 70 on YTC) to balance discriminative power and redundancy.

Experimental results

Research questions

• RQ1 Can combining multiple Riemannian manifold-valued descriptors (SPD, Grassmann, Gaussian) improve image set classification performance compared to using a single geometry?
• RQ2 How effectively can Riemannian kernels map heterogeneous manifold features into a common Hilbert space for subsequent metric learning?
• RQ3 Does multi-kernel metric learning in a shared subspace yield better generalization and discriminative power than single-kernel or Euclidean-based methods?
• RQ4 What is the optimal dimensionality $d_w$ of the embedded subspace for balancing feature richness and redundancy in the final representation?
• RQ5 How does the proposed method compare to state-of-the-art approaches across diverse video-based classification tasks?

Key findings

• The proposed method achieves state-of-the-art performance on four benchmark datasets: AFEW (video-based face recognition), YTC (set-based object categorization), MDSD (video-based emotion recognition), and ETH-80 (dynamic scene classification).
• The fusion of multiple Riemannian descriptors (SPD, Grassmann, Gaussian) significantly improves classification accuracy over any single descriptor, confirming their complementary nature.
• On the MDSD dataset, the optimal $d_w = 25$ yields the highest recognition rate, while higher dimensions (e.g., 91) lead to a drop in performance to 5.13%, indicating redundancy at high dimensions.
• On the YTC dataset, $d_w = 70$ achieves the best performance, and increasing it to 141 results in a drop to 72.22%, further confirming the trade-off between dimensionality and discriminative quality.
• The objective function converges stably after multiple iterations, with values stabilizing at 0.8398 (AFEW) and 0.9207 (YTC) after 40 iterations, indicating robust optimization behavior.
• Empirical results confirm that the multi-kernel metric learning framework effectively integrates heterogeneous Riemannian features, outperforming conventional Euclidean and single-geometry approaches.