[Paper Review] Zelda: Video Analytics using Vision-Language Models

Francisco Romero, Caleb Winston+3 more|arXiv (Cornell University)|May 5, 2023

Advanced Neural Network Applications4 citations

TL;DR

Zelda is a video analytics system that leverages Vision-Language Models (VLMs) to enable natural language querying of large video datasets with high expressivity, relevance, and diversity. It automatically enhances prompt quality, filters low-quality frames, and uses semantic embeddings to prune redundant results, achieving up to 1.15× higher mean average precision and 7.5× faster retrieval than state-of-the-art systems.

ABSTRACT

Advances in ML have motivated the design of video analytics systems that allow for structured queries over video datasets. However, existing systems limit query expressivity, require users to specify an ML model per predicate, rely on complex optimizations that trade off accuracy for performance, and return large amounts of redundant and low-quality results. This paper focuses on the recently developed Vision-Language Models (VLMs) that allow users to query images using natural language like "cars during daytime at traffic intersections." Through an in-depth analysis, we show VLMs address three limitations of current video analytics systems: general expressivity, a single general purpose model to query many predicates, and are both simple and fast. However, VLMs still return large numbers of redundant and low-quality results that can overwhelm and burden users. In addition, VLMs often require manual prompt engineering to improve result relevance. We present Zelda: a video analytics system that uses VLMs to return both relevant and semantically diverse results for top-K queries on large video datasets. Zelda prompts the VLM with the user's query in natural language. Zelda then automatically adds discriminator and synonym terms to boost accuracy, and terms to identify low-quality frames. To improve result diversity, Zelda uses semantic-rich VLM embeddings in an algorithm that prunes similar frames while considering their relevance to the query and the number of top-K results requested. We evaluate Zelda across five datasets and 19 queries and quantitatively show it achieves higher mean average precision (up to 1.15x) and improves average pairwise similarity (up to 1.16x) compared to using VLMs out-of-the-box. We also compare Zelda to a state-of-the-art video analytics engine and show that Zelda retrieves results 7.5x (up to 10.4x) faster for the same accuracy and frame diversity.

Motivation & Objective

• Address the limited expressivity of current video analytics systems, which restrict queries to predefined model classes.
• Overcome the need for users to manually select and train multiple ML models per query predicate.
• Reduce system complexity and performance/accuracy trade-offs inherent in existing video analytics pipelines.
• Mitigate the problem of redundant and low-quality results in top-K video retrieval by improving result diversity and quality.
• Enable fast, accurate, and user-friendly video analytics using a single general-purpose VLM without query-specific training or annotations.

Proposed method

• Use a pre-trained Vision-Language Model (VLM) to encode user queries in natural language and compute semantic similarity between the query and video frames.
• Automatically augment the user query with discriminator terms (e.g., 'not', 'except') and synonym terms to improve retrieval accuracy and robustness.
• Identify and filter low-quality frames using VLM-based embeddings that detect blur, noise, or irrelevant content.
• Apply a semantic diversity-aware pruning algorithm that uses VLM-generated embeddings to rank and select top-K frames with high relevance and low pairwise similarity.
• Leverage the zero-shot capability of VLMs to support complex, multi-predicate queries without retraining or model specialization.
• Optimize for efficiency by avoiding query-specific model training and relying on fast, shared VLM inference across all queries.

Figure 1 . Current systems (a) limit predicate expressivity, require user expertise for model selection, and are complex to use. Our system Zelda (b) allows expressive querying in natural language using a single fast model while returning relevant and diverse results.

Experimental results

Research questions

• RQ1 Can Vision-Language Models (VLMs) serve as a general-purpose, zero-shot alternative to specialized ML models in video analytics systems?
• RQ2 To what extent can automatic prompt engineering with synonyms and discriminators improve VLM-based video retrieval accuracy?
• RQ3 How effective is semantic embedding-based diversity pruning in reducing redundant results while maintaining relevance?
• RQ4 Can a VLM-based system achieve higher mean average precision (MAP) and lower result similarity compared to out-of-the-box VLM inference?
• RQ5 How does the end-to-end performance of Zelda compare to state-of-the-art video analytics systems in terms of speed, accuracy, and result diversity?

Key findings

• Zelda improves mean average precision (MAP) by up to 1.15× compared to using VLMs out-of-the-box, demonstrating significant gains in retrieval accuracy.
• Zelda reduces average pairwise similarity between top-K results by up to 1.16×, indicating improved semantic diversity in the retrieved frames.
• Zelda achieves up to 10.4× faster retrieval than a state-of-the-art video analytics system (BlazeIt) while maintaining equivalent accuracy.
• On average, Zelda is 7.5× faster than the best-performing baseline system, showing strong performance scalability across diverse datasets.
• The system effectively removes low-quality frames without requiring manual annotations or additional training, relying solely on VLM-based quality scoring.
• Semantic similarity-based diversity pruning outperforms visual similarity methods, improving MAP by 1.06× while requiring less hyperparameter tuning.

Figure 2 . General architecture of vision-language models.