[Paper Review] ConvGRU in Fine-grained Pitching Action Recognition for Action Outcome Prediction

Tianqi Ma, Lin Zhang+2 more|arXiv (Cornell University)|Aug 18, 2020

Human Pose and Action Recognition42 references4 citations

TL;DR

This paper proposes a ConvGRU-based model for fine-grained recognition of baseball pitching actions to predict outcome (e.g., strike or ball), leveraging sequential RGB video frames. The method achieves 79.17% accuracy, outperforming prior state-of-the-art on a fine-grained pitching action dataset, and analyzes the impact of sampling strategies, fusion methods, and pre-training on performance.

ABSTRACT

Prediction of the action outcome is a new challenge for a robot collaboratively working with humans. With the impressive progress in video action recognition in recent years, fine-grained action recognition from video data turns into a new concern. Fine-grained action recognition detects subtle differences of actions in more specific granularity and is significant in many fields such as human-robot interaction, intelligent traffic management, sports training, health caring. Considering that the different outcomes are closely connected to the subtle differences in actions, fine-grained action recognition is a practical method for action outcome prediction. In this paper, we explore the performance of convolutional gate recurrent unit (ConvGRU) method on a fine-grained action recognition tasks: predicting outcomes of ball-pitching. Based on sequences of RGB images of human actions, the proposed approach achieved the performance of 79.17% accuracy, which exceeds the current state-of-the-art result. We also compared different network implementations and showed the influence of different image sampling methods, different fusion methods and pre-training, etc. Finally, we discussed the advantages and limitations of ConvGRU in such action outcome prediction and fine-grained action recognition tasks.

Motivation & Objective

• To address the challenge of predicting action outcomes in human-robot collaboration by leveraging fine-grained action recognition.
• To explore the effectiveness of ConvGRU in capturing spatiotemporal features from video sequences for subtle action differences in pitching.
• To compare various network architectures, image sampling methods, fusion strategies, and pre-training techniques in the context of outcome prediction.
• To evaluate the advantages and limitations of ConvGRU for fine-grained action recognition and outcome prediction tasks.

Proposed method

• The model uses ConvGRU to process sequential RGB video frames, capturing spatial and temporal dependencies through convolutional gates.
• It processes video clips as input sequences, applying ConvGRU layers to extract hierarchical spatiotemporal representations.
• Different image sampling strategies (e.g., uniform, dense) are evaluated to determine optimal frame selection for input.
• Feature fusion methods (e.g., early, late, early-fusion with attention) are compared to integrate spatial and temporal features effectively.
• Pre-training on large-scale action recognition datasets is applied to improve generalization and performance.
• The final prediction head classifies the action outcome (e.g., strike or ball) based on the final hidden state of the ConvGRU.

Experimental results

Research questions

• RQ1 How effective is the ConvGRU architecture in recognizing fine-grained differences in pitching actions for outcome prediction?
• RQ2 What is the impact of different image sampling methods on the performance of action outcome prediction?
• RQ3 How do various feature fusion strategies affect the accuracy of outcome prediction in sequential video data?
• RQ4 To what extent does pre-training improve the performance of the ConvGRU model on the fine-grained pitching action recognition task?
• RQ5 What are the key advantages and limitations of using ConvGRU for fine-grained action recognition in real-world human-robot interaction scenarios?

Key findings

• The proposed ConvGRU-based model achieves a test accuracy of 79.17%, surpassing the current state-of-the-art performance on the fine-grained pitching action recognition benchmark.
• The use of dense sampling and late fusion strategies leads to improved performance compared to uniform sampling and early fusion.
• Pre-training on large-scale datasets significantly enhances model generalization and final accuracy.
• ConvGRU demonstrates strong capability in modeling long-range temporal dependencies in video sequences, crucial for capturing subtle differences in pitching mechanics.
• Despite strong performance, the model shows sensitivity to input sequence quality and sampling strategy, indicating room for improvement in data preprocessing.