[Paper Review] Fine-Grained Action Detection with RGB and Pose Information using Two Stream Convolutional Networks

Leonard Hacker, Finn Bartels+1 more|arXiv (Cornell University)|Feb 6, 2023

Stroke Rehabilitation and RecoveryMedicine3 citations

TL;DR

This paper proposes a two-stream 3D CNN framework that fuses RGB video frames with human pose information extracted via MMPose for fine-grained table tennis stroke detection. By combining raw RGB and pose-over-RGB (PRGB) streams with late feature fusion, the model achieves 87.3% classification accuracy—slightly improving over the baseline—though detection performance remains limited, with an IoU of 0.349 and mAP of 0.110.

ABSTRACT

As participants of the MediaEval 2022 Sport Task, we propose a two-stream network approach for the classification and detection of table tennis strokes. Each stream is a succession of 3D Convolutional Neural Network (CNN) blocks using attention mechanisms. Each stream processes different 4D inputs. Our method utilizes raw RGB data and pose information computed from MMPose toolbox. The pose information is treated as an image by applying the pose either on a black background or on the original RGB frame it has been computed from. Best performance is obtained by feeding raw RGB data to one stream, Pose + RGB (PRGB) information to the other stream and applying late fusion on the features. The approaches were evaluated on the provided TTStroke-21 data sets. We can report an improvement in stroke classification, reaching 87.3% of accuracy, while the detection does not outperform the baseline but still reaches an IoU of 0.349 and mAP of 0.110.

Motivation & Objective

• To improve fine-grained action detection in sports videos by leveraging both visual and human pose information.
• To investigate whether pose representations can enhance action recognition and detection beyond traditional RGB-based models.
• To evaluate the effectiveness of different pose fusion strategies (e.g., PRGB vs. pose-only) in a two-stream 3D CNN architecture.
• To determine if pose data can compensate for low inter-class variability in actions like table tennis strokes.
• To benchmark performance against the MediaEval 2022 baseline on the TTStroke-21 dataset for both classification and detection tasks.

Proposed method

• Utilizes a two-stream 3D CNN architecture, with one stream processing raw RGB video clips and the other processing pose-enhanced frames.
• Pose information is extracted using the MMPose toolbox, applying top-down keypoint estimation on detected persons in each frame.
• Two pose-based inputs are evaluated: pose on black background (Pose) and pose superimposed on original RGB frames (PRGB).
• Late feature fusion is applied before the final classification layer, with summed fusion showing the best performance.
• The model employs 3D Convolutional Neural Network (3D-CNN) blocks with attention mechanisms to capture spatiotemporal features.
• Training is conducted on Tesla V100 GPUs over 2000 epochs with a learning rate of 0.0001 and momentum of 0.5.

(a) RGB

Experimental results

Research questions

• RQ1 Can combining RGB and pose information improve fine-grained action classification in table tennis videos compared to single-stream RGB models?
• RQ2 Does pose information enhance detection performance for low-variability actions like table tennis strokes?
• RQ3 Which fusion strategy—early, late, or intermediate—yields the best performance in a two-stream 3D CNN for action detection?
• RQ4 How does the quality of pose estimation impact the model’s ability to distinguish between similar table tennis strokes?
• RQ5 Can pose data compensate for the lack of object-level cues (e.g., racket, ball) in detection tasks?

Key findings

• The two-stream model achieved 87.3% test accuracy in stroke classification, representing a 0.9% improvement over the baseline.
• The best detection performance was achieved with the RGB and PRGB two-stream configuration, yielding an IoU of 0.349 and mAP of 0.110.
• The Pose-only stream underperformed significantly in detection, with mAP dropping to 0.046, suggesting pose data alone lacks discriminative cues.
• Late fusion with summed feature combination outperformed weighted and concatenated fusion, indicating optimal fusion timing at the final layer.
• Despite improved classification, detection performance did not surpass the baseline, likely due to missing ball and racket cues in pose data.
• The model showed strong generalization, with validation accuracy reaching 0.848 for the RGB and PRGB stream, indicating robust learning on the TTStroke-21 dataset.

(b) Pose