[Paper Review] Object Relational Graph with Teacher-Recommended Learning for Video Captioning

Ziqi Zhang, Yaya Shi+5 more|arXiv (Cornell University)|Feb 26, 2020

Multimodal Machine Learning ApplicationsComputer Science44 references39 citations

TL;DR

The paper introduces an object-relational graph encoder with GCN-based relational reasoning and a teacher-recommended learning strategy that leverages an external language model to improve video captioning performance.

ABSTRACT

Taking full advantage of the information from both vision and language is critical for the video captioning task. Existing models lack adequate visual representation due to the neglect of interaction between object, and sufficient training for content-related words due to long-tailed problems. In this paper, we propose a complete video captioning system including both a novel model and an effective training strategy. Specifically, we propose an object relational graph (ORG) based encoder, which captures more detailed interaction features to enrich visual representation. Meanwhile, we design a teacher-recommended learning (TRL) method to make full use of the successful external language model (ELM) to integrate the abundant linguistic knowledge into the caption model. The ELM generates more semantically similar word proposals which extend the ground-truth words used for training to deal with the long-tailed problem. Experimental evaluations on three benchmarks: MSVD, MSR-VTT and VATEX show the proposed ORG-TRL system achieves state-of-the-art performance. Extensive ablation studies and visualizations illustrate the effectiveness of our system.

Motivation & Objective

• Motivate improved video captioning by enriching visual representation through object interactions across frames.
• Address the long-tailed word distribution by incorporating linguistic knowledge from external language models during training.
• Develop a training strategy that combines visual relational reasoning with teacher-guided language learning to enhance generalization.

Proposed method

• Construct a learnable Object Relational Graph (ORG) to model spatial-temporal interactions among objects using GCNs.
• Implement two graph variants: Partial ORG (P-ORG) within a frame and Complete ORG (C-ORG) across the video with top-k connections.
• Introduce Teacher-Recommended Learning (TRL) that uses an External Language Model (ELM) to generate soft targets and enrich training with linguistically diverse word proposals.
• Train the captioning model with a joint loss combining cross-entropy on hard targets and KL divergence with soft targets from the ELM (L = lambda * L_KL + (1-lambda) * L_CE).
• Describe a hierarchical decoder with temporal-spatial attention for word generation, incorporating global and local context features.

Experimental results

Research questions

• RQ1 How can object-level relational reasoning improve visual representations for video captioning?
• RQ2 Can external linguistic knowledge be effectively integrated into a captioning model to mitigate long-tailed word distributions?
• RQ3 What is the impact of combining ORG-based relational encoding with TRL on standard video captioning benchmarks?

Key findings

• The ORG encoder improves object representations by modeling interactions via GCNs (P-ORG and C-ORG).
• TRL leveraging an off-line ELM (e.g., BERT) provides soft targets that alleviate long-tailed word issues and boosts linguistic diversity in captions.
• The combined ORG-TRL system achieves state-of-the-art performance on MSVD, MSR-VTT, and VATEX benchmarks.
• Ablation studies show that C-ORG with a top-k setting (k=5) yields the best results, and TRL consistently benefits performance.
• Qualitative results indicate richer, more detailed captions that capture object relations and actions.