[Paper Review] FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding

Dian Shao, Yue Zhao+2 more|arXiv (Cornell University)|Apr 14, 2020

Human Pose and Action Recognition51 references31 citations

TL;DR

FineGym introduces a large-scale gymnastics video dataset with a three-level semantic hierarchy (event, set, element) and two-level temporal annotations (action, sub-action), enabling fine-grained action understanding and empirical study of related methods.

ABSTRACT

On public benchmarks, current action recognition techniques have achieved great success. However, when used in real-world applications, e.g. sport analysis, which requires the capability of parsing an activity into phases and differentiating between subtly different actions, their performances remain far from being satisfactory. To take action recognition to a new level, we develop FineGym, a new dataset built on top of gymnastic videos. Compared to existing action recognition datasets, FineGym is distinguished in richness, quality, and diversity. In particular, it provides temporal annotations at both action and sub-action levels with a three-level semantic hierarchy. For example, a "balance beam" event will be annotated as a sequence of elementary sub-actions derived from five sets: "leap-jump-hop", "beam-turns", "flight-salto", "flight-handspring", and "dismount", where the sub-action in each set will be further annotated with finely defined class labels. This new level of granularity presents significant challenges for action recognition, e.g. how to parse the temporal structures from a coherent action, and how to distinguish between subtly different action classes. We systematically investigate representative methods on this dataset and obtain a number of interesting findings. We hope this dataset could advance research towards action understanding.

Motivation & Objective

• Create a high-quality, fine-grained gymnastics video dataset with multi-level semantic labels (event, set, element).
• Provide temporal annotations at both action and sub-action levels to enable temporal fine-grained understanding.
• Ensure data quality through expert-built categories and rigorous annotation quality control.
• Benchmark representative action recognition methods to reveal challenges and guide future research in fine-grained action understanding.

Proposed method

• Assemble high-resolution videos from official gymnastics competitions to ensure data quality and relevance.
• Annotate actions with a three-level semantic hierarchy (event, set, element) and temporally identify actions and sub-actions.
• Use decision-tree based reasoning to derive element labels within a given set.
• Implement multi-stage annotation quality control including annotator training, pretests, and cross-validation.
• Evaluate representative action recognition models across coarse to fine granularities (event/set/element) using RGB, Flow, 2Stream, and skeleton-based approaches.
• Demonstrate temporal action localization with the SSN framework to study event vs. sub-action localization.

Experimental results

Research questions

• RQ1 How well do current action recognition models perform onFineGym's fine-grained (element-level) actions?
• RQ2 What is the impact of temporal modeling and frame sampling on fine-grained action understanding?
• RQ3 Does pre-training on coarse-grained datasets help for fine-grained gymnastics actions?
• RQ4 What challenges emerge for appearance- versus motion-based features in fine-grained, fast motions?
• RQ5 How feasible is temporal action localization for sub-actions compared to coarse actions in FineGym?

Key findings

• Sparse frame sampling is insufficient for element-level recognition; many frames are needed to achieve robust performance.
• Motion information plays a significantly important role, and temporal modeling is crucial for fine-grained actions.
• Temporal dynamics modeling (TRN/TSM) yields substantial gains over static-frame baselines, and mismatches between training/testing frame counts hurt performance.
• Pre-training on large coarse-grained datasets is not always beneficial for FineGym due to temporal pattern gaps.
• Skeleton-based methods (ST-GCN) struggle on gymnastics data due to challenging pose estimation in diverse actions.
• FineGym presents a challenging benchmark with high-quality, action-centric data and a rich hierarchy, guiding future research in fine-grained action understanding.