[Paper Review] Not All Words are Equal: Video-specific Information Loss for Video Captioning

Jiarong Dong, Ke Gao+4 more|arXiv (Cornell University)|Jan 1, 2019

Multimodal Machine Learning Applications26 references3 citations

TL;DR

This paper proposes Information Loss, a novel learning strategy that dynamically resizes cross-entropy loss based on word importance—balancing information relevance and content—to reduce bias toward common words in video captioning. Combined with a hierarchical attention mechanism and multi-level visual features, the method achieves state-of-the-art performance, improving CIDEr by 18.2% on MSVD (87.5) and 47.7 on MSR-VTT.

ABSTRACT

An ideal description for a given video should fix its gaze on salient and representative content, which is capable of distinguishing this video from others. However, the distribution of different words is unbalanced in video captioning datasets, where distinctive words for describing video-specific salient objects are far less than common words such as 'a' 'the' and 'person'. The dataset bias often results in recognition error or detail deficiency of salient but unusual objects. To address this issue, we propose a novel learning strategy called Information Loss, which focuses on the relationship between the video-specific visual content and corresponding representative words. Moreover, a framework with hierarchical visual representations and an optimized hierarchical attention mechanism is established to capture the most salient spatial-temporal visual information, which fully exploits the potential strength of the proposed learning strategy. Extensive experiments demonstrate that the ingenious guidance strategy together with the optimized architecture outperforms state-of-the-art video captioning methods on MSVD with CIDEr score 87.5, and achieves superior CIDEr score 47.7 on MSR-VTT. We also show that our Information Loss is generic which improves various models by significant margins.

Motivation & Objective

• Address the imbalance in word frequency distribution in video captioning datasets, where common words dominate training loss.
• Reduce recognition errors and detail deficiencies for distinctive, video-specific objects by focusing learning on informative, rare words.
• Develop a generic training strategy that enhances performance across diverse models without architectural changes.
• Improve video captioning quality by emphasizing salient visual content through a refined loss function and attention mechanism.

Proposed method

• Propose Information Loss, a dynamic loss function that reweights words based on their importance, defined as the product of information relevance and information content.
• Calculate information relevance as the cosine similarity between a word's embedding and the video's overall semantic representation.
• Define information content as the inverse of word frequency, giving rare words higher weight to emphasize their discriminative value.
• Integrate Information Loss into a hierarchical attention mechanism that processes visual features at both frame-level and video-level for improved spatial-temporal representation.
• Use a two-layer LSTM architecture with hierarchical attention to extract and aggregate salient visual features across time and space.
• Optimize the loss with two hyperparameters: γ (controls balance between relevance and content) and λ (controls loss weighting intensity).

Experimental results

Research questions

• RQ1 How does word frequency imbalance in video captioning datasets affect model performance on rare, salient visual concepts?
• RQ2 Can a dynamic loss function that prioritizes distinctive, informative words improve captioning quality and reduce ambiguity?
• RQ3 To what extent can Information Loss enhance performance across different video captioning models without architectural modifications?
• RQ4 How does hierarchical visual representation and attention improve the model’s ability to attend to salient video content?
• RQ5 What hyperparameter settings (γ and λ) yield optimal performance in balancing relevance and content in the loss function?

Key findings

• The proposed HA_IL model achieves a CIDEr score of 87.5 on MSVD, representing an 18.2% relative improvement over prior state-of-the-art methods.
• On MSR-VTT, the model achieves a CIDEr score of 47.7, demonstrating strong generalization despite using fewer features than competing models.
• The method significantly reduces description ambiguity, as shown in qualitative examples where the model correctly identifies rare actions like 'slam dunk' and 'cracking'.
• Information Loss is generic: it improves performance across multiple base models, including those using different attention mechanisms and feature types.
• Ablation studies confirm that γ=2 and λ=0.5 yield optimal performance, balancing relevance and content in word importance calculation.
• Visualization shows the model effectively focuses on salient visual regions across frames, even when objects are small or distant.