[Paper Review] Spatio-Temporal Graph for Video Captioning with Knowledge Distillation

Boxiao Pan, Haoye Cai+5 more|arXiv (Cornell University)|Mar 31, 2020

Multimodal Machine Learning Applications52 references30 citations

TL;DR

This paper proposes a spatio-temporal object graph for video captioning with an object-aware knowledge distillation mechanism that distills object-level information into a scene branch, achieving competitive results with interpretable grounding.

ABSTRACT

Video captioning is a challenging task that requires a deep understanding of visual scenes. State-of-the-art methods generate captions using either scene-level or object-level information but without explicitly modeling object interactions. Thus, they often fail to make visually grounded predictions, and are sensitive to spurious correlations. In this paper, we propose a novel spatio-temporal graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid unstable performance caused by the variable number of objects, we further propose an object-aware knowledge distillation mechanism, in which local object information is used to regularize global scene features. We demonstrate the efficacy of our approach through extensive experiments on two benchmarks, showing our approach yields competitive performance with interpretable predictions.

Motivation & Objective

• Motivate video captioning to explicitly model object interactions beyond global frame features.
• Propose a spatio-temporal graph to capture space-time object relationships.
• Introduce an object-aware knowledge distillation mechanism to regularize global scene features with object information.
• Develop a two-branch architecture where the object branch informs the scene branch during training, enabling efficient testing with scene features only.
• Demonstrate improved performance and grounding on standard benchmarks.

Proposed method

• Represent video as a spatio-temporal graph with per-frame spatial graphs and inter-frame temporal graphs.
• Compute spatial adjacency using normalized IoU between object proposals; compute temporal adjacency using cosine similarity between object features across adjacent frames.
• Apply graph convolution on the unified spatio-temporal graph to update object features.
• Introduce a scene branch that processes global frame features and fuses with the distilled object information via a soft, logit-level distillation loss.
• Use two separate Transformer-based language decoders for object and scene branches and train with language losses plus distillation loss.
• During testing, deploy only the scene branch for caption generation, leveraging distilled object knowledge.

Experimental results

Research questions

• RQ1 Can explicit spatio-temporal modeling of object interactions improve video captioning quality and grounding?
• RQ2 Does object-aware knowledge distillation effectively regularize noisy object-graph representations when integrating with global scene features?
• RQ3 How does the proposed two-branch framework compare to single-branch baselines and other graph-based approaches on standard benchmarks?
• RQ4 What is the impact of including both spatial and temporal graphs versus using only one or a dense graph?
• RQ5 Does the approach provide interpretable visual grounding of captions?

Key findings

• On MSVD, the full model achieves state-of-the-art or competitive results across BLEU@4, METEOR, ROUGE-L, and CIDEr.
• On MSR-VTT, the full model shows competitive performance with strong baselines, with scene-only variants performing worse, highlighting the benefit of object-informed distillation.
• Ablation shows that both spatial and temporal graphs contribute meaningfully, and that prior-knowledge-based graphs outperform dense, unstructured graphs.
• Object-aware knowledge distillation improves grounding as evidenced by qualitative saliency maps and more accurate object-related captioning details.
• The two-branch architecture with logit-level distillation yields better performance than concatenation or L2-based fusion, while avoiding test-time object detection.