[Paper Review] Multimodal Feature Fusion for Video Advertisements Tagging Via Stacking Ensemble

Qingsong Zhou, Hai Liang+2 more|arXiv (Cornell University)|Aug 2, 2021

Video Analysis and Summarization34 references4 citations

TL;DR

This paper proposes a stacking ensemble-based multimodal feature fusion framework for video advertisement tagging, combining visual, textual, and audio features to improve accuracy and robustness. The method achieves state-of-the-art performance, ranking 1st in the 2021 Tencent Advertising Algorithm Competition with a Global Average Precision (GAP) of 82.63%.

ABSTRACT

Automated tagging of video advertisements has been a critical yet challenging problem, and it has drawn increasing interests in last years as its applications seem to be evident in many fields. Despite sustainable efforts have been made, the tagging task is still suffered from several challenges, such as, efficiently feature fusion approach is desirable, but under-explored in previous studies. In this paper, we present our approach for Multimodal Video Ads Tagging in the 2021 Tencent Advertising Algorithm Competition. Specifically, we propose a novel multi-modal feature fusion framework, with the goal to combine complementary information from multiple modalities. This framework introduces stacking-based ensembling approach to reduce the influence of varying levels of noise and conflicts between different modalities. Thus, our framework can boost the performance of the tagging task, compared to previous methods. To empirically investigate the effectiveness and robustness of the proposed framework, we conduct extensive experiments on the challenge datasets. The obtained results suggest that our framework can significantly outperform related approaches and our method ranks as the 1st place on the final leaderboard, with a Global Average Precision (GAP) of 82.63%. To better promote the research in this field, we will release our code in the final version.

Motivation & Objective

• Address the challenge of noisy and conflicting multimodal features in video advertisement tagging.
• Improve tagging performance by effectively fusing complementary information from visual, textual, and audio modalities.
• Develop a robust feature fusion strategy that reduces the impact of modality-specific noise and inconsistencies.
• Achieve top performance in the 2021 ACM Multimedia Multi-modal Ads Video Understanding Challenge.
• Demonstrate the effectiveness and generalization of the proposed framework through extensive ablation and comparative experiments.

Proposed method

• Employ a stacking ensemble approach to combine predictions from multiple base models trained on individual modalities (visual, text, audio).
• Use deep neural networks (DNNs) to extract and classify features from each modality: visual features from CNNs, text from OCR and ASR, and audio from speech recognition.
• Apply feature-level fusion strategies including concatenation, sum-pooling, max-pooling, and attention mechanisms for ablation and comparison.
• Utilize a meta-learner in the stacking ensemble to integrate predictions from base models, learning optimal weighting and reducing noise from conflicting modality outputs.
• Apply dropout regularization to prevent overfitting during training of the ensemble model.
• Optimize the model using Global Average Precision (GAP) as the primary evaluation metric, with confidence scores used for ranking predictions.

Experimental results

Research questions

• RQ1 How effectively can a stacking ensemble framework fuse multimodal features (visual, text, audio) to improve video advertisement tagging performance?
• RQ2 What is the relative contribution of each modality (visual, text, audio) to the final tagging accuracy and GAP?
• RQ3 How does the proposed stacking-based fusion strategy compare to traditional fusion methods (e.g., concatenation, attention, pooling) in terms of robustness and performance?
• RQ4 To what extent does the inclusion of additional features (e.g., TF-IDF, extra embeddings) improve the tagging results?
• RQ5 Can the framework generalize well across diverse video ads with varying levels of noise and modality conflicts?

Key findings

• The proposed stacking ensemble framework achieved a Global Average Precision (GAP) of 82.63%, ranking 1st in the final leaderboard of the 2021 ACM Multimedia challenge.
• Visual features alone achieved the highest individual GAP of 79.636%, outperforming text (75.612%) and audio (70.701%) modalities.
• Combining visual, text, and audio features with TF-IDF and extra embeddings yielded the best performance: 93.34% accuracy and 82.641% GAP.
• The stacking-based fusion method outperformed all baseline fusion strategies, including feature concatenation (GAP: 80.971%), sum-pooling (80.197%), max-pooling (80.511%), and attention (81.295%).
• Ablation studies confirmed that adding more modalities consistently improved performance, with the full feature set (visual+text+audio+TF-IDF+extra) yielding the highest accuracy and GAP.
• Visualization of predictions showed that true positive tags were consistently ranked within the top 9 predictions, confirming the model’s reliability and precision.