[Paper Review] Deep manifold-to-manifold transforming network for action recognition

Tong Zhang, Wenming Zheng+2 more|arXiv (Cornell University)|May 30, 2017

Human Pose and Action Recognition2 references3 citations

TL;DR

This paper proposes Deep Manifold-to-Manifold Transforming Network (DMT-Net), an end-to-end deep learning framework that maps symmetric positive definite (SPD) matrices from one Riemannian manifold to a more discriminative one for action recognition. It introduces novel layers—local SPD convolutional, non-linear SPD activation, and Riemannian-preserved recursive—while preserving SPD structure without singular value decomposition (SVD), achieving state-of-the-art performance on action recognition benchmarks.

ABSTRACT

Symmetric positive definite (SPD) matrices (e.g., covariances, graph Laplacians, etc.) are widely used to model the relationship of spatial or temporal domain. Nevertheless, SPD matrices are theoretically embedded on Riemannian manifolds. In this paper, we propose an end-to-end deep manifold-to-manifold transforming network (DMT-Net) which can make SPD matrices flow from one Riemannian manifold to another more discriminative one. To learn discriminative SPD features characterizing both spatial and temporal dependencies, we specifically develop three novel layers on manifolds: (i) the local SPD convolutional layer, (ii) the non-linear SPD activation layer, and (iii) the Riemannian-preserved recursive layer. The SPD property is preserved through all layers without any requirement of singular value decomposition (SVD), which is often used in the existing methods with expensive computation cost. Furthermore, a diagonalizing SPD layer is designed to efficiently calculate the final metric for the classification task. To evaluate our proposed method, we conduct extensive experiments on the task of action recognition, where input signals are popularly modeled as SPD matrices. The experimental results demonstrate that our DMT-Net is much more competitive over state-of-the-art.

Motivation & Objective

• Address the challenge of learning discriminative features from SPD matrices embedded on Riemannian manifolds for action recognition.
• Overcome the high computational cost of SVD-based operations in existing SPD deep learning methods.
• Enable effective modeling of temporal dynamics in action sequences using recursive layers on Riemannian manifolds.
• Design efficient metric computation for classification without relying on expensive matrix logarithms or SVD.
• Develop a unified, end-to-end trainable network that preserves SPD structure across all layers for improved representation learning.

Proposed method

• Proposes a local SPD convolutional layer that constrains filters to be SPD matrices, ensuring manifold preservation through theoretical proof.
• Introduces a non-linear SPD activation layer using element-wise operations (e.g., exp, sinh, cosh), which avoids SVD and maintains SPD structure via the Schur product theorem.
• Designs a Riemannian-preserved recursive layer inspired by GRU, where all operations (bilinear projection, activation, Hadamard product) are proven to preserve SPD properties.
• Incorporates a diagonalizing SPD layer that transforms SPD matrices into diagonal form to enable efficient Log-Euclidean metric computation without SVD.
• Employs a manifold-to-manifold transformation strategy, allowing SPD features to flow across multiple Riemannian manifolds for enhanced discriminative power.
• Integrates all components into an end-to-end deep network trained jointly for action recognition, with theoretical guarantees on SPD preservation at each stage.

Experimental results

Research questions

• RQ1 How can local convolutional filtering be performed on SPD matrices while preserving their Riemannian manifold structure?
• RQ2 Can non-linear activation functions be designed for SPD matrices without requiring SVD or matrix logarithms?
• RQ3 How can recursive learning be generalized to Riemannian manifolds to model temporal dependencies in action sequences?
• RQ4 Can efficient metric computation be achieved for SPD matrices without relying on computationally expensive SVD or matrix logarithm operations?
• RQ5 To what extent does a manifold-to-manifold deep network improve action recognition performance compared to existing SOTA methods?

Key findings

• DMT-Net achieves state-of-the-art performance on multiple action recognition benchmarks, outperforming existing SOTA methods.
• The proposed local SPD convolutional layer preserves the Riemannian manifold structure and enables effective local feature learning without SVD.
• The non-linear SPD activation layer avoids SVD by using element-wise operations and maintains SPD properties through the Schur product theorem.
• The Riemannian-preserved recursive layer successfully models long-range temporal dependencies in action sequences while maintaining SPD structure across time steps.
• The diagonalizing SPD layer enables efficient Log-Euclidean metric computation, reducing computational cost while preserving classification accuracy.
• Extensive experiments confirm that DMT-Net consistently outperforms prior methods across various datasets, demonstrating its robustness and generalization capability.