[Paper Review] Use What You Have: Video Retrieval Using Representations From Collaborative Experts

Yang Liu, Samuel Albanie+2 more|arXiv (Cornell University)|Jul 31, 2019

Multimodal Machine Learning ApplicationsComputer Science67 references206 citations

TL;DR

The paper presents a Collaborative Experts (CE) framework that aggregates multiple pre-trained, single-modality video experts (e.g., objects, actions, speech, OCR) into a single, compact video-text embedding for retrieval, using a dynamic collaborative gating mechanism. It evaluates on five benchmarks and shows improved retrieval performance by leveraging both generic and specific cues.

ABSTRACT

The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge. Human-generated queries for video datasets `in the wild' vary a lot in terms of degree of specificity, with some queries describing specific details such as the names of famous identities, content from speech, or text available on the screen. Our goal is to condense the multi-modal, extremely high dimensional information from videos into a single, compact video representation for the task of video retrieval using free-form text queries, where the degree of specificity is open-ended. For this we exploit existing knowledge in the form of pre-trained semantic embeddings which include 'general' features such as motion, appearance, and scene features from visual content. We also explore the use of more 'specific' cues from ASR and OCR which are intermittently available for videos and find that these signals remain challenging to use effectively for retrieval. We propose a collaborative experts model to aggregate information from these different pre-trained experts and assess our approach empirically on five retrieval benchmarks: MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet. Code and data can be found at www.robots.ox.ac.uk/~vgg/research/collaborative-experts/. This paper contains a correction to results reported in the previous version.

Motivation & Objective

• Reduce annotation cost for video-text retrieval by leveraging pre-trained, single-modality experts (e.g., objects, actions, speech, OCR).
• Develop a robust, fixed-length video representation that can be queried with free-form text queries.
• Demonstrate that collaborative gating enables effective integration of diverse, partially available cues.
• Evaluate the approach across multiple datasets to establish generalizability and relative gains over prior methods.

Proposed method

• Construct a joint video-text embedding by combining multiple pre-trained single-modality experts into task-specific embeddings.
• Apply a collaborative gating module that computes pairwise expert relationships to produce attention vectors and modulate expert representations.
• Project expert embeddings to a common dimension and use a Gated Embedding Module (GEM) to obtain normalized embeddings.
• Encode text queries with contextual word embeddings and aggregate with NetVLAD, then project to subspaces per expert and form a joint text embedding.
• Train the video and text encoders jointly with a bidirectional max-margin ranking loss.
• Handle missing experts by zero-padding and renormalizing mixture weights to maintain consistent similarity scoring.

Experimental results

Research questions

• RQ1 Can a joint embedding of video and text be effectively learned by aggregating diverse pre-trained expert representations?
• RQ2 To what extent do general (motion, appearance, scene) versus specific (ASR, OCR, text on screen) cues contribute to retrieval performance?
• RQ3 Does a collaborative gating mechanism improve integration of information across experts for video-text retrieval?
• RQ4 How does the proposed CE framework perform across standard video retrieval benchmarks compared to prior methods?

Key findings

• Collaborative Experts consistently improve retrieval performance over baselines across multiple benchmarks (MSR-VTT, LSMDC, MSVD, DiDeMo, ActivityNet).
• Strong generic cues (appearance, motion, scene) provide solid baselines, while specific cues (ASR, OCR, on-screen text) are valuable but challenging to utilize when intermittently available.
• The collaborative gating mechanism (pairwise relations among experts) yields improved fusion and a more compact parameterization than prior methods.
• Increasing the number of training captions per video provides performance gains comparable to adding more experts, with combined use of multiple captions and multiple experts giving the strongest results.
• CE achieves competitive or state-of-the-art results on several benchmarks, including MSR-VTT, LSMDC, MSVD, DiDeMo and ActivityNet, under video-to-text and text-to-video retrieval settings.