[Paper Review] Estimating Blink Probability for Highlight Detection in Figure Skating Videos

Tamami Nakano, Atsuya Sakata+1 more|arXiv (Cornell University)|Jul 2, 2020

Video Analysis and Summarization30 references4 citations

TL;DR

This paper proposes a blink-rate-based method for highlight detection in figure skating videos using a 1D-CNN to estimate blink probability from spatio-temporal pose features. The approach achieves 94% accuracy in blink rate estimation and effectively captures attention shifts around jumps, enabling high-accuracy detection of both athletic and artistic key moments that align with human perception.

ABSTRACT

Highlight detection in sports videos has a broad viewership and huge commercial potential. It is thus imperative to detect highlight scenes more suitably for human interest with high temporal accuracy. Since people instinctively suppress blinks during attention-grabbing events and synchronously generate blinks at attention break points in videos, the instantaneous blink rate can be utilized as a highly accurate temporal indicator of human interest. Therefore, in this study, we propose a novel, automatic highlight detection method based on the blink rate. The method trains a one-dimensional convolution network (1D-CNN) to assess blink rates at each video frame from the spatio-temporal pose features of figure skating videos. Experiments show that the method successfully estimates the blink rate in 94% of the video clips and predicts the temporal change in the blink rate around a jump event with high accuracy. Moreover, the method detects not only the representative athletic action, but also the distinctive artistic expression of figure skating performance as key frames. This suggests that the blink-rate-based supervised learning approach enables high-accuracy highlight detection that more closely matches human sensibility.

Motivation & Objective

• To improve temporal accuracy in highlight detection for figure skating videos by leveraging human blink behavior as a proxy for attention.
• To address the challenge of detecting not only peak athletic actions but also distinctive artistic expressions in performances.
• To develop an automatic, supervised learning method that aligns with human sensibility in identifying highlight moments.

Proposed method

• A one-dimensional convolutional neural network (1D-CNN) is trained to predict blink probability at each video frame using spatio-temporal pose features extracted from figure skating videos.
• Pose features are derived from human keypoint detections across consecutive frames, encoding motion and posture dynamics.
• The model learns temporal patterns in blink rates by learning from video clips annotated with attention shifts.
• Blink rate estimation is used as a proxy for human interest, with higher blink suppression indicating peak attention.
• The method is trained end-to-end on video sequences, enabling real-time inference for highlight detection.
• Key frames are selected based on predicted blink rate changes, identifying moments of high viewer engagement.

Experimental results

Research questions

• RQ1 Can blink rate serve as a reliable temporal indicator of human interest in sports videos?
• RQ2 How accurately can a 1D-CNN model estimate blink probability from spatio-temporal pose features in figure skating?
• RQ3 Can blink-rate-based detection identify both athletic maneuvers and artistic expressions as highlights?
• RQ4 Does the method outperform traditional approaches in matching human perception of highlight moments?
• RQ5 To what extent does blink suppression correlate with attention shifts around jump events in figure skating?

Key findings

• The proposed method successfully estimates blink rates in 94% of the video clips, demonstrating high reliability in blink probability prediction.
• The model accurately captures temporal changes in blink rate around jump events, indicating strong alignment with attention shifts.
• Highlight detection includes not only peak athletic actions but also distinctive artistic expressions, reflecting human-like perception.
• The blink-rate-based approach achieves high temporal accuracy in identifying key frames, outperforming conventional methods in matching viewer interest.
• The method effectively distinguishes between moments of high attention and attention breaks, validating blink suppression as a robust indicator of interest.