[Paper Review] SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation

Jiaben Chen, Huaizu Jiang|arXiv (Cornell University)|Aug 31, 2023

Advanced Image Processing TechniquesComputer Science3 citations

TL;DR

This paper introduces SportsSloMo, a new benchmark of 130K high-resolution (≥720p) sports video clips for human-centric video frame interpolation, featuring complex human motion and frequent occlusions. To improve interpolation accuracy, it proposes model-agnostic human-aware loss terms using panoptic human segmentation and keypoint detection as auxiliary supervision, which consistently boost performance across seven state-of-the-art VFI methods, with PSNR gains up to 0.69 dB and SSIM improvements of 0.006.

ABSTRACT

Human-centric video frame interpolation has great potential for improving people's entertainment experiences and finding commercial applications in the sports analysis industry, e.g., synthesizing slow-motion videos. Although there are multiple benchmark datasets available in the community, none of them is dedicated for human-centric scenarios. To bridge this gap, we introduce SportsSloMo, a benchmark consisting of more than 130K video clips and 1M video frames of high-resolution ($\geq$720p) slow-motion sports videos crawled from YouTube. We re-train several state-of-the-art methods on our benchmark, and the results show a decrease in their accuracy compared to other datasets. It highlights the difficulty of our benchmark and suggests that it poses significant challenges even for the best-performing methods, as human bodies are highly deformable and occlusions are frequent in sports videos. To improve the accuracy, we introduce two loss terms considering the human-aware priors, where we add auxiliary supervision to panoptic segmentation and human keypoints detection, respectively. The loss terms are model agnostic and can be easily plugged into any video frame interpolation approaches. Experimental results validate the effectiveness of our proposed loss terms, leading to consistent performance improvement over 5 existing models, which establish strong baseline models on our benchmark. The dataset and code can be found at: https://neu-vi.github.io/SportsSlomo/.

Motivation & Objective

• To address the lack of dedicated benchmarks for human-centric video frame interpolation, especially in dynamic sports scenarios with complex human motion and occlusions.
• To evaluate the performance drop of existing SOTA VFI methods on human-centric content, revealing significant challenges in real-world applications.
• To propose model-agnostic human-aware loss terms—based on panoptic segmentation and keypoint detection—to improve interpolation accuracy at motion boundaries.
• To release a large-scale, curated dataset of 130K high-resolution sports clips to enable future research in human-centric video generation and analysis.

Proposed method

• The authors curate 130K high-resolution (≥720p) slow-motion sports video clips from YouTube under the Creative Commons license, removing non-slow-motion, ad, and transition content.
• Each video is split into 9-frame clips with the first and last frames as inputs and the middle seven as ground truth for training and evaluation.
• They introduce two auxiliary loss terms: one for panoptic human segmentation to improve boundary accuracy and reduce ghosting, and another for human keypoint detection to enforce coherent motion trajectories.
• These losses are applied during training and are model-agnostic, allowing integration into any VFI framework without architectural changes.
• The losses are applied by comparing predicted intermediate frames with ground truth using pre-trained panoptic segmentation and keypoint detection models as supervision.
• The method is evaluated on seven SOTA VFI models, including SuperSloMo and EBME, showing consistent performance gains across metrics like PSNR, SSIM, and IE.

Figure 1: Human-centric video frame interpolation results. We propose human-aware auxiliary losses to improve interpolation accuracy at motion boundaries. From left to right: (a): Overlayed inputs, (b): Ground truth, (c): Interpolation results, (d): Interpolation results with our proposed human-awar

Experimental results

Research questions

• RQ1 How does the performance of existing SOTA video frame interpolation models degrade when applied to human-centric sports videos with complex motion and occlusions?
• RQ2 Can auxiliary supervision from human segmentation and keypoint detection improve interpolation accuracy in highly deformable human motion scenarios?
• RQ3 To what extent do model-agnostic human-aware losses enhance performance across diverse VFI architectures?
• RQ4 What are the limitations of current optical flow estimation in human-centric scenarios, and can knowledge distillation from pre-trained models improve flow accuracy for VFI?
• RQ5 Can 3D human body reconstruction be leveraged to further improve VFI in occluded or complex motion settings?

Key findings

• The SportsSloMo benchmark significantly degrades the performance of existing SOTA VFI models, with EBME dropping from 36.64 PSNR on SNU-FILM to 30.70 on SportsSloMo, highlighting its difficulty.
• The proposed human-aware loss terms consistently improve PSNR by up to 0.69 dB and SSIM by up to 0.006 across seven SOTA models, with the largest gains seen when combining both losses.
• For SuperSloMo, the combination of segmentation and keypoint losses increased PSNR from 29.77 to 30.24 and SSIM from 0.910 to 0.917.
• For EBME, the combined losses improved PSNR from 30.15 to 30.48 and SSIM from 0.941 to 0.944, with a 5.6% reduction in interpolation error (IE).
• Qualitative results show improved interpolation around highly deformable limbs (e.g., elbows and hands) and reduced ghosting under occlusions.
• Knowledge distillation using GMFlow for optical flow supervision yielded mixed results, improving SuperSloMo but degrading EBME, indicating domain shift and model-specific sensitivity.

Figure 2: Visualization of human keypoints [ 78 ] and panoptic segmentation masks [ 10 ] .