[Paper Review] RGB Video Based Tennis Action Recognition Using a Deep Weighted Long Short-Term Memory.

Jiaxin Cai, Xin Tang|arXiv (Cornell University)|Aug 2, 2018

Human Pose and Action Recognition11 references4 citations

TL;DR

This paper proposes a novel RGB video-based tennis action recognition method using a deep weighted Long Short-Term Memory (LSTM) network combined with pre-trained Inception CNN features. By leveraging spatial CNN representations from individual frames and a score-weighted LSTM decoder to model temporal dependencies, the approach achieves state-of-the-art performance on benchmark datasets using only raw RGB video input.

ABSTRACT

Action recognition has attracted increasing attention from RGB input in computer vision partially due to potential applications on somatic simulation and statistics of sport such as virtual tennis game and tennis techniques and tactics analysis by video. Recently, deep learning based methods have achieved promising performance for action recognition. In this paper, we propose weighted Long Short-Term Memory adopted with convolutional neural network representations for three dimensional tennis shots recognition. First, the local two-dimensional convolutional neural network spatial representations are extracted from each video frame individually using a pre-trained Inception network. Then, a weighted Long Short-Term Memory decoder is introduced to take the output state at time t and the historical embedding feature at time t-1 to generate feature vector using a score weighting scheme. Finally, we use the adopted CNN and weighted LSTM to map the original visual features into a vector space to generate the spatial-temporal semantical description of visual sequences and classify the action video content. Experiments on the benchmark demonstrate that our method using only simple raw RGB video can achieve better performance than the state-of-the-art baselines for tennis shot recognition.

Motivation & Objective

• To improve action recognition in tennis videos using only RGB video input.
• To address the challenge of modeling long-range temporal dependencies in video sequences for accurate shot classification.
• To develop a spatial-temporal representation learning framework that captures both spatial features and sequential dynamics.
• To achieve superior performance compared to existing state-of-the-art methods on benchmark datasets.

Proposed method

• Spatial features are extracted from each video frame using a pre-trained Inception network for 2D CNN representation.
• A weighted Long Short-Term Memory (LSTM) decoder is employed, which uses the current state and historical embedding to generate context-aware feature vectors.
• The weighted LSTM applies a score weighting scheme to dynamically assign importance to historical hidden states during sequence modeling.
• The combined CNN and weighted LSTM architecture maps visual features into a joint spatial-temporal vector space for action classification.
• The model is trained to classify three-dimensional tennis shots based on raw RGB video sequences.

Experimental results

Research questions

• RQ1 Can a weighted LSTM with CNN features effectively model temporal dynamics in tennis video sequences for action recognition?
• RQ2 How does the proposed method compare to state-of-the-art approaches in terms of accuracy on RGB video tennis shot recognition?
• RQ3 To what extent can raw RGB video alone achieve high performance without optical flow or additional modality inputs?
• RQ4 Does the score-weighting mechanism in the LSTM improve feature representation learning over standard LSTM architectures?

Key findings

• The proposed method achieves better performance than state-of-the-art baselines on benchmark datasets for tennis shot recognition.
• The model demonstrates strong generalization using only raw RGB video, without requiring optical flow or additional input modalities.
• The integration of pre-trained Inception features with a weighted LSTM improves spatial-temporal feature representation learning.
• The score-weighting mechanism in the LSTM enhances temporal modeling by selectively attending to relevant historical states.