[Paper Review] Deep Multimodal Feature Encoding for Video Ordering

Vivek Sharma, Makarand Tapaswi+1 more|arXiv (Cornell University)|Apr 5, 2020

Human Pose and Action Recognition65 references6 citations

TL;DR

This paper proposes Temporal Compact Bilinear Pooling (TCBP), a self-supervised method that learns compact multimodal video representations by fusing visual, audio, and textual features across temporal segments. Trained via a novel proxy task—ordering unordered video clips—TCBP achieves state-of-the-art performance on action recognition, outperforming methods like C3D and two-stream networks on UCF101 and HMDB51 by leveraging joint temporal and multimodal interactions.

ABSTRACT

True understanding of videos comes from a joint analysis of all its modalities: the video frames, the audio track, and any accompanying text such as closed captions. We present a way to learn a compact multimodal feature representation that encodes all these modalities. Our model parameters are learned through a proxy task of inferring the temporal ordering of a set of unordered videos in a timeline. To this end, we create a new multimodal dataset for temporal ordering that consists of approximately 30K scenes (2-6 clips per scene) based on the "Large Scale Movie Description Challenge". We analyze and evaluate the individual and joint modalities on three challenging tasks: (i) inferring the temporal ordering of a set of videos; and (ii) action recognition. We demonstrate empirically that multimodal representations are indeed complementary, and can play a key role in improving the performance of many applications.

Motivation & Objective

• To develop a compact, joint multimodal representation that fuses visual, audio, and textual features for video clips.
• To introduce a self-supervised proxy task—temporal ordering of unordered video clips—for learning clip-level representations without expensive annotations.
• To create a new multimodal dataset of ~30K ordered movie scenes (2–6 clips per scene) for training and evaluation.
• To evaluate the effectiveness of multimodal fusion and temporal modeling in video understanding tasks.
• To demonstrate that TCBP enhances performance on downstream tasks like action recognition.

Proposed method

• Proposes Temporal Compact Bilinear Pooling (TCBP), an extension of Tensor Sketch-based compact bilinear pooling to incorporate temporal dynamics across video segments.
• Uses deep pre-trained models (e.g., C3D, ResNet) to extract visual and audio features, and CLIP for text embeddings.
• Applies TCBP to encode pairwise interactions between modalities across multiple temporal segments (typically 2–5 seconds), producing a compact feature vector of size 8192.
• Trains the model using a contrastive loss with positive and negative clip pairs: $\mathcal{L} = \sum_{(V_i,V_j)\in Pos} L(V_i,V_j) + \sum_{(V_i,V_j^\prime)\in Neg} \max(0, \alpha - L(V_i,V_j^\prime))$, with margin $\alpha = 0.2$.
• Employs a Siamese-like twin network architecture to compare clip pairs and predict their correct temporal order.
• Uses the TSN architecture with 3 segments, each processed through C3D, followed by element-wise multiplication and TCBP for final encoding.

Experimental results

Research questions

• RQ1 Can a self-supervised proxy task based on video clip ordering effectively learn robust multimodal representations?
• RQ2 How do individual modalities (visual, audio, text) and their joint encoding contribute to temporal ordering performance?
• RQ3 Does TCBP, as a compact multimodal encoding method, improve performance on downstream video understanding tasks like action recognition?
• RQ4 How does TCBP compare to existing pooling methods such as bilinear pooling or NetVLAD in capturing temporal and multimodal interactions?
• RQ5 To what extent do negative examples improve the learning of temporal ordering representations?

Key findings

• TCBP achieves 88.03% accuracy on UCF101 and 61.58% on HMDB51, outperforming C3D (82.3% and 56.8%), two-stream networks (88.0% and 59.4%), and iDT+FV (85.9% and 57.2%).
• The inclusion of negative examples during training improves performance over positive-only training, indicating better generalization.
• Joint multimodal representations (visual, audio, text) are more effective than unimodal representations for temporal ordering, demonstrating complementarity.
• TCBP significantly outperforms bilinear pooling (CBP) and iDT+FV, suggesting its effectiveness in modeling temporal and cross-modal interactions.
• The model achieves state-of-the-art performance on action recognition without requiring additional data or fine-tuning, highlighting the robustness of the learned features.
• The proposed dataset of ~30K ordered movie scenes provides a valuable benchmark for future research in multimodal video understanding.