[Paper Review] SACT: Self-Aware Multi-Space Feature Composition Transformer for Multinomial Attention for Video Captioning

Chiranjib Sur|arXiv (Cornell University)|Jun 25, 2020

Multimodal Machine Learning Applications38 references4 citations

TL;DR

This paper proposes SACT, a Self-Aware Multi-Space Feature Composition Transformer that enhances video captioning by introducing Multinomial Attention (MultAtt) to selectively compose relevant video frames, improving attention quality over traditional multi-head attention. The model outperforms prior methods on ActivityNet and YouCookII, achieving state-of-the-art results through refined feature composition and awareness of salient content.

ABSTRACT

Video captioning works on the two fundamental concepts, feature detection and feature composition. While modern day transformers are beneficial in composing features, they lack the fundamental problems of selecting and understanding of the contents. As the feature length increases, it becomes increasingly important to include provisions for improved capturing of the pertinent contents. In this work, we have introduced a new concept of Self-Aware Composition Transformer (SACT) that is capable of generating Multinomial Attention (MultAtt) which is a way of generating distributions of various combinations of frames. Also, multi-head attention transformer works on the principle of combining all possible contents for attention, which is good for natural language classification, but has limitations for video captioning. Video contents have repetitions and require parsing of important contents for better content composition. In this work, we have introduced SACT for more selective attention and combined them for different attention heads for better capturing of the usable contents for any applications. To address the problem of diversification and encourage selective utilization, we propose the Self-Aware Composition Transformer model for dense video captioning and apply the technique on two benchmark datasets like ActivityNet and YouCookII.

Motivation & Objective

• To address the limitation of standard multi-head attention in video captioning, which treats all features uniformly without selective awareness of salient content.
• To improve video captioning performance by enabling the model to identify and prioritize the most relevant frame combinations through self-aware feature composition.
• To develop a novel architecture that enhances attention quality by modeling distributions of frame combinations rather than aggregating all features equally.
• To achieve better caption generation by integrating awareness of content relevance at the frame level within a multi-layered transformer framework.
• To surpass existing methods on benchmark datasets like ActivityNet and YouCookII through improved attention composition and representation learning.

Proposed method

• Introduces Self-Aware Composition Transformer (SACT), a multi-layered transformer architecture that applies self-aware filtering to select and compose salient frame features before attention computation.
• Proposes Multinomial Attention (MultAtt), which models the distribution of useful frame combinations using weighted sums: $ f_{M_uA} = W_{M_uA} abla eta_i extbf{I}_i $, where $ m << n $, focusing on high-impact frames.
• Employs a separate awareness mechanism applied at the end of the transformer stack to refine multi-head attention outputs, enhancing sensitivity to frame-level relevance.
• Uses a masking framework during training to prevent overfitting and reduce dependency on previous word embeddings, emphasizing multi-head attention for language generation.
• Adopts an end-to-end training strategy with 30 epochs, learning rate of 0.0002, and joint optimization of event proposal and caption generation.
• Applies the architecture to two benchmarks—ActivityNet and YouCookII—using image and optical flow features with a total attention dimension of 1048.

Experimental results

Research questions

• RQ1 How can attention mechanisms in video captioning be made more selective to improve relevance and reduce noise from irrelevant frames?
• RQ2 Can modeling frame combinations through a multinomial distribution of importance improve caption quality compared to standard multi-head attention?
• RQ3 Does introducing a self-aware filtering mechanism at the transformer's final layers enhance feature composition and representation learning for video captioning?
• RQ4 To what extent does the SACT architecture outperform existing models in terms of both automatic metrics and semantic coherence on dense video captioning benchmarks?
• RQ5 How does the separation of awareness from joint attention improve performance compared to joint attention strategies?

Key findings

• SACT achieved state-of-the-art performance on the ActivityNet Captions dataset, outperforming previous methods in BLEU-4 and other standard metrics.
• On the YouCookII dataset, the SA s CT variant (separate awareness) outperformed the SA j CT variant (joint awareness), demonstrating the benefit of decoupled awareness for attention modeling.
• The model achieved higher BLEU-4 scores than prior methods, indicating improved syntactic and lexical alignment in generated captions.
• The use of Multinomial Attention (MultAtt) enabled better selection of salient frame combinations, reducing noise and improving attention quality.
• The self-aware mechanism significantly enhanced representation learning by focusing on relevant features, especially in long and complex video sequences.
• The masking framework helped prevent overfitting and improved generalization by reducing reliance on autoregressive word dependencies.