[Paper Review] Temporal Transformer Networks: Joint Learning of Invariant and Discriminative Time Warping

Suhas Lohit, Qiao Wang+1 more|arXiv (Cornell University)|Jun 13, 2019

Human Pose and Action Recognition42 references4 citations

TL;DR

This paper proposes the Temporal Transformer Network (TTN), a differentiable, interpretable module that jointly learns invariant and discriminative time warping for time-series classification. By applying input-dependent, non-parametric diffeomorphisms, TTN reduces intra-class variance and increases inter-class separation, achieving state-of-the-art performance on 3D action recognition benchmarks, especially with limited training data.

ABSTRACT

Many time-series classification problems involve developing metrics that are invariant to temporal misalignment. In human activity analysis, temporal misalignment arises due to various reasons including differing initial phase, sensor sampling rates, and elastic time-warps due to subject-specific biomechanics. Past work in this area has only looked at reducing intra-class variability by elastic temporal alignment. In this paper, we propose a hybrid model-based and data-driven approach to learn warping functions that not just reduce intra-class variability, but also increase inter-class separation. We call this a temporal transformer network (TTN). TTN is an interpretable differentiable module, which can be easily integrated at the front end of a classification network. The module is capable of reducing intra-class variance by generating input-dependent warping functions which lead to rate-robust representations. At the same time, it increases inter-class variance by learning warping functions that are more discriminative. We show improvements over strong baselines in 3D action recognition on challenging datasets using the proposed framework. The improvements are especially pronounced when training sets are smaller.

Motivation & Objective

• To address temporal misalignment in time-series classification, particularly in human activity recognition due to variable motion rates and biomechanical differences.
• To develop a differentiable, interpretable module that learns warping functions to reduce intra-class variability while enhancing inter-class discriminability.
• To integrate seamlessly into existing time-series classifiers like TCNs and LSTMs without architectural overhaul.
• To improve classification robustness under rate variations, especially in low-data regimes.
• To provide a geometrically grounded, model-based approach that bridges the gap between analytical invariance and data-driven deep learning.

Proposed method

• TTN is a trainable, differentiable module inserted at the front of a classification network to perform input-specific temporal warping.
• It generates non-parametric, order-preserving diffeomorphisms using a learned velocity field in square-root velocity representation, ensuring smooth and invertible warping.
• The warping function is derived from a learned vector field via integration, enabling expressive, continuous temporal transformations.
• The module is optimized end-to-end with the classifier to jointly minimize classification loss and promote intra-class alignment and inter-class separation.
• TTN exploits the non-uniqueness of optimal alignments to learn discriminative warps that push dissimilar classes apart.
• It is compatible with various architectures, requiring only a single-line integration into models like TCNs and LSTMs.

Experimental results

Research questions

• RQ1 Can a deep learning module jointly learn invariant and discriminative time warping for time-series classification?
• RQ2 How can model-based geometric constraints be integrated into data-driven networks to improve robustness to temporal rate variations?
• RQ3 To what extent can a differentiable, interpretable warping module improve classification accuracy on 3D action recognition datasets with limited training data?
• RQ4 Does the TTN enhance feature space clustering and inter-class separation in representation learning?
• RQ5 Can the TTN be effectively applied across different modalities, such as Kinect and mocap-based 3D skeletons?

Key findings

• On the ICL action dataset with induced rate variations, TTN significantly improved clustering quality and classification accuracy by aligning misaligned waveforms.
• On the NTU RGB-D dataset, TCN+TTN achieved 77.55% accuracy in cross-subject evaluation, outperforming the baseline TCN (76.54%) by 1.01 percentage points.
• Using two parallel TTNs and concatenating their outputs further improved performance to 77.80% on the cross-subject split.
• The TTN improved clustering metrics: normalized cluster purity (CP) increased from 0.466 to 0.493, and normalized entropy (H) from 0.575 to 0.596.
• t-SNE visualizations confirmed that TTN outputs led to better-separated, more compact clusters in the feature space.
• The ablation study showed that TTN consistently improved both intra-class compactness and inter-class separation, especially in low-data regimes.