[Paper Review] FusedLSTM: Fusing frame-level and video-level features for Content-based Video Relevance Prediction

Yash Bhalgat|arXiv (Cornell University)|Sep 28, 2018

Video Analysis and Summarization4 citations

TL;DR

This paper proposes FusedLSTM, a deep learning model that fuses frame-level Inception-pool3 and video-level C3D-pool5 features using an LSTM and dense layers to predict video relevance. It achieves state-of-the-art performance on the ACMMM-2018 CBVRP challenge by minimizing triplet loss and using online kernel similarity learning, outperforming the baseline and OMKS methods on both movie and TV show tracks.

ABSTRACT

This paper describes two of my best performing approaches on the Content-based Video Relevance Prediction challenge. In the FusedLSTM based approach, the inception-pool3 and the C3D-pool5 features are combined using an LSTM and a dense layer to form embeddings with the objective to minimize the triplet loss function. In the second approach, an Online Kernel Similarity Learning method is proposed to learn a non-linear similarity measure to adhere the relevance training data. The last section gives a complete comparison of all the approaches implemented during this challenge, including the one presented in the baseline paper.

Motivation & Objective

• To address the cold-start problem in video recommendation by enabling content-based relevance prediction without relying on user behavior data.
• To improve video relevance prediction by fusing multi-level features (frame-level and video-level) using deep neural networks.
• To learn robust video embeddings that minimize triplet loss, ensuring relevant videos are closer in embedding space than irrelevant ones.
• To explore non-linear similarity learning via online kernel methods to enhance generalization and avoid overfitting.
• To evaluate and compare multiple architectures, including FusedLSTM and kernel-based methods, on real-world video retrieval benchmarks.

Proposed method

• FusedLSTM combines Inception-pool3 (frame-level) and C3D-pool5 (video-level) features through an LSTM followed by a dense layer to generate joint embeddings.
• The model is trained using triplet loss to enforce that the similarity between anchor and positive videos exceeds that with negative videos by a margin.
• Three similarity kernels—cosine, RBF, and softmax—are used to compute pairwise similarities between embeddings in the triplet loss objective.
• To prevent embedding norm explosion, L2 regularization is applied to the triplet loss function.
• Anchor point mirroring is used to increase training data diversity by reordering triplets, reducing overfitting from repeated anchor usage.
• An online kernel similarity learning method (OMKS) is also proposed to learn non-linear similarity functions from video features.

Experimental results

Research questions

• RQ1 Can fusing frame-level and video-level features through an LSTM improve video relevance prediction performance?
• RQ2 Does using triplet loss with multiple similarity kernels lead to better generalization in video retrieval?
• RQ3 How does the FusedLSTM architecture compare to baseline and OMKS methods in terms of hit@k and recall@k metrics?
• RQ4 To what extent does anchor mirroring improve model generalization and reduce overfitting in triplet training?
• RQ5 Can online kernel similarity learning effectively model non-linear relationships in video feature space for relevance prediction?

Key findings

• FusedLSTM achieved the best performance on both the movie and TV show tracks, outperforming the CBVRP baseline and OMKS methods on the test set.
• On the movie test set, FusedLSTM with the Softmax kernel achieved a hit@50 of 0.545 and recall@50 of 0.397, significantly exceeding the baseline's 0.484 and 0.319.
• On the TV show test set, FusedLSTM with the Softmax kernel achieved a hit@50 of 0.484 and recall@50 of 0.261, surpassing the baseline's 0.463 and 0.237.
• The FusedLSTM model with RBF kernel achieved a hit@50 of 0.552 on the movie test set, indicating strong generalization on high-precision retrieval.
• OMKS with delta features performed competitively, achieving a hit@50 of 0.501 on movies and 0.493 on shows, showing the effectiveness of online kernel learning.
• The ablation study confirmed that FusedLSTM consistently outperformed simpler models like 2-layer neural networks and baseline methods across all metrics.