[Paper Review] A Variational Time Series Feature Extractor for Action Prediction

Maxime Chaveroche, Adrien Malaisé+3 more|arXiv (Cornell University)|Jul 6, 2018

Human Pose and Action Recognition20 references3 citations

TL;DR

This paper proposes a Variational Time Series Feature Extractor (VTSFE), a deep generative model based on variational autoencoders that improves feature extraction for human action prediction and classification from wearable motion capture data. By enhancing noise modeling, simplifying the latent transition dynamics, and tightening the variational lower bound, VTSFE achieves superior encoding of temporal dynamics—especially for high-amplitude movements—outperforming VAE and VAE-DMP in feature quality and computational efficiency, though reconstruction capability remains limited.

ABSTRACT

We propose a Variational Time Series Feature Extractor (VTSFE), inspired by the VAE-DMP model of Chen et al., to be used for action recognition and prediction. Our method is based on variational autoencoders. It improves VAE-DMP in that it has a better noise inference model, a simpler transition model constraining the acceleration in the trajectories of the latent space, and a tighter lower bound for the variational inference. We apply the method for classification and prediction of whole-body movements on a dataset with 7 tasks and 10 demonstrations per task, recorded with a wearable motion capture suit. The comparison with VAE and VAE-DMP suggests the better performance of our method for feature extraction. An open-source software implementation of each method with TensorFlow is also provided. In addition, a more detailed version of this work can be found in the indicated code repository. Although it was meant to, the VTSFE hasn't been tested for action prediction, due to a lack of time in the context of Maxime Chaveroche's Master thesis at INRIA.

Motivation & Objective

• To develop a robust feature extractor for whole-body motion sequences from wearable sensor data to enable accurate action recognition and prediction.
• To address limitations in existing models—such as poor dynamics encoding and high computational cost—by refining variational inference and latent space modeling.
• To improve the representation of temporal dynamics in latent space, ensuring smoother, more consistent trajectories for similar actions (e.g., 'bent forward' vs. 'bent forward strongly').
• To enable online inference by training the model offline, making it suitable for real-time robotic interaction and assistive systems.
• To provide an open-source, modular implementation in TensorFlow for reproducibility and further development.

Proposed method

• The VTSFE model uses a variational autoencoder (VAE) framework with a structured latent space to encode time-series motion data from wearable sensors.
• It introduces a simplified transition model that constrains acceleration in the latent space, improving temporal smoothness and plausibility of generated trajectories.
• A tighter variational lower bound is derived to improve the quality of posterior approximation and training stability.
• The noise inference model is enhanced to better capture uncertainty in motion data, leading to more robust latent representations.
• The encoder maps input motion sequences into a low-dimensional latent space, while the decoder reconstructs the original sequence from the latent code.
• The model is trained using Monte Carlo sampling for approximate posterior inference, with a loss function combining reconstruction and KL divergence terms.

Experimental results

Research questions

• RQ1 Can a variational autoencoder with improved latent dynamics modeling extract more informative features for action recognition and prediction from wearable motion data?
• RQ2 How does VTSFE compare to VAE and VAE-DMP in terms of latent space quality, especially for high-amplitude movements like kicking?
• RQ3 Does the tighter variational bound and refined noise model lead to more consistent and smoother latent trajectories for similar actions?
• RQ4 Can VTSFE achieve faster training and inference times while maintaining or improving feature representation compared to existing models?
• RQ5 To what extent does the latent space of VTSFE support downstream tasks like action prediction and classification?

Key findings

• VTSFE achieves significantly better encoding of motion dynamics than VAE and VAE-DMP, particularly for high-amplitude movements such as kicking, as evidenced by lower reconstruction error in these cases.
• The latent space of VTSFE produces smoother, more consistent trajectories, with similar actions (e.g., 'bent forward' and 'bent forward strongly') mapped closer together in the latent space.
• Training with VTSFE is computationally faster than VAE-DMP: VTSFE light 2D trained in 9m30s compared to 16m30s for VAE-DMP 2D on the same hardware.
• Despite better feature encoding, VTSFE’s reconstruction capability is not superior to VAE-DMP or VAE, suggesting the decoder is not the primary strength of the model.
• The model is computationally intensive during training due to Monte Carlo sampling, limiting it to offline pre-training, but inference is efficient and suitable for online applications.
• The authors note that a second training phase—freezing the encoder and retraining the decoder with a standard autoencoder loss—could significantly improve reconstruction, though this was not pursued in the current work.