[Paper Review] Fine-grained Video Classification and Captioning

Farzaneh Mahdisoltani, Guillaume Berger+3 more|arXiv (Cornell University)|Apr 24, 2018

Human Pose and Action Recognition28 citations

TL;DR

This paper proposes an end-to-end deep neural network for joint video classification and captioning with shared features across multiple levels of granularity, demonstrating that training on finer-grained actions improves feature quality for transfer learning. On the Something-Something dataset, the model achieves state-of-the-art performance in fine-grained classification and establishes a strong baseline for video captioning.

ABSTRACT

We describe a DNN for video classification and captioning, trained end-to-end, with shared features, to solve tasks at different levels of granularity, exploring the link between granularity in a source task and the quality of learned features for transfer learning. For solving the new task domain in transfer learning, we freeze the trained encoder and fine-tune a neural net on the target domain. We train on the Something-Something dataset with over 220, 000 videos, and multiple levels of target granularity, including 50 action groups, 174 fine-grained action categories and captions. Classification and captioning with Something-Something are challenging because of the subtle differences between actions, applied to thousands of different object classes, and the diversity of captions penned by crowd actors. Our model performs better than existing classification baselines for SomethingSomething, with impressive fine-grained results. And it yields a strong baseline on the new Something-Something captioning task. Experiments reveal that training with more fine-grained tasks tends to produce better features for transfer learning.

Motivation & Objective

• To investigate how granularity in source tasks affects the quality of learned features for transfer learning.
• To develop a unified deep learning model that performs video classification and captioning simultaneously with shared features.
• To evaluate the model's performance on the challenging Something-Something dataset, which features subtle action differences and diverse captions.
• To establish a strong baseline for the new video captioning task on Something-Something.

Proposed method

• The model uses a shared encoder to extract features from video clips, shared across both classification and captioning heads.
• It is trained end-to-end on multiple levels of granularity: 50 coarse action groups, 174 fine-grained action categories, and full captions.
• The encoder is frozen during transfer learning, and a new classifier or captioning head is fine-tuned on the target domain.
• The approach leverages temporal modeling through 3D convolutions and attention mechanisms to capture fine-grained action dynamics.
• The model is trained on over 220,000 videos from the Something-Something dataset, including diverse object classes and human-annotated captions.
• Transfer learning is performed by freezing the encoder and fine-tuning a new head on downstream tasks.

Experimental results

Research questions

• RQ1 How does the granularity of the source task influence the quality of features learned for transfer learning?
• RQ2 Can a single shared encoder effectively support both video classification and captioning at multiple levels of granularity?
• RQ3 Does training on fine-grained actions lead to better performance on downstream classification tasks?
• RQ4 How does the model perform on the newly introduced video captioning task for Something-Something?
• RQ5 What is the impact of multi-level supervision (coarse, fine-grained, and captioning) on feature representation quality?

Key findings

• The model outperforms existing classification baselines on the Something-Something dataset, particularly in fine-grained action recognition.
• Training with more fine-grained tasks leads to better feature representations for transfer learning, as evidenced by improved downstream performance.
• The model establishes a strong baseline for the new video captioning task on Something-Something, demonstrating effective generation of diverse and accurate captions.
• Shared features across tasks improve generalization, especially when the source data includes fine-grained action distinctions.
• The end-to-end training with multi-level supervision enhances feature learning, resulting in robust representations for both classification and captioning.