[Paper Review] VideoStory Embeddings Recognize Events when Examples are Scarce

Amirhossein Habibian, Thomas Mensink+1 more|arXiv (Cornell University)|Nov 8, 2015

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes VideoStory, a video representation learning method that embeds video features into a shared semantic space with textual descriptions using a joint objective balancing descriptiveness and multimodal predictability. By leveraging web-scale video-text pairs and optimizing for correlated terms, VideoStory achieves state-of-the-art performance in both few- and zero-example event recognition, outperforming prior methods on TRECVID and Columbia Consumer Videos datasets.

ABSTRACT

This paper aims for event recognition when video examples are scarce or even completely absent. The key in such a challenging setting is a semantic video representation. Rather than building the representation from individual attribute detectors and their annotations, we propose to learn the entire representation from freely available web videos and their descriptions using an embedding between video features and term vectors. In our proposed embedding, which we call VideoStory, the correlations between the terms are utilized to learn a more effective representation by optimizing a joint objective balancing descriptiveness and predictability.We show how learning the VideoStory using a multimodal predictability loss, including appearance, motion and audio features, results in a better predictable representation. We also propose a variant of VideoStory to recognize an event in video from just the important terms in a text query by introducing a term sensitive descriptiveness loss. Our experiments on three challenging collections of web videos from the NIST TRECVID Multimedia Event Detection and Columbia Consumer Videos datasets demonstrate: i) the advantages of VideoStory over representations using attributes or alternative embeddings, ii) the benefit of fusing video modalities by an embedding over common strategies, iii) the complementarity of term sensitive descriptiveness and multimodal predictability for event recognition without examples. By it abilities to improve predictability upon any underlying video feature while at the same time maximizing semantic descriptiveness, VideoStory leads to state-of-the-art accuracy for both few- and zero-example recognition of events in video.

Motivation & Objective

• To address event recognition in video when labeled examples are scarce or absent.
• To develop a video representation that is both semantically descriptive and highly predictable from video features.
• To improve generalization in few- and zero-example settings by learning from freely available web videos and their textual descriptions.
• To integrate multiple modalities—appearance, motion, and audio—into a unified semantic embedding for better predictive power.
• To enable accurate event recognition from only a textual query, without any video examples.

Proposed method

• Learning a joint embedding between video features and term vectors using a multimodal predictability loss over appearance, motion, and audio features.
• Optimizing a joint objective that balances descriptiveness (how well terms describe the video) and predictability (how well video features predict terms).
• Introducing VideoStoryF, a variant that fuses multiple video modalities in a shared semantic space to enhance predictive performance.
• Proposing VideoStory0, a variant with a term-sensitive descriptiveness loss to improve accuracy for zero-example recognition from textual queries.
• Using web-harvested video-text pairs to pre-train the embedding without relying on manually annotated attributes.
• Applying the VideoStory embedding to any underlying video feature (e.g., CNN, improved dense trajectories) to boost its predictive power.

Experimental results

Research questions

• RQ1 Can a semantic video representation be learned effectively from web videos and their descriptions without relying on manually annotated attributes?
• RQ2 Does joint optimization of descriptiveness and multimodal predictability improve event recognition in few- and zero-example settings?
• RQ3 Can fusing appearance, motion, and audio features through a shared embedding outperform traditional feature fusion strategies?
• RQ4 How effective is the term-sensitive descriptiveness loss in improving zero-example event recognition from textual queries?
• RQ5 Can VideoStory achieve state-of-the-art performance across multiple benchmarks in both few- and zero-example event detection?

Key findings

• VideoStory achieves a mean average precision (mAP) of 37.1% on the TRECVID 2013 few-example test set, setting a new state-of-the-art.
• On the zero-example setting, VideoStory achieves 20.0% mAP, surpassing the previous best result of 18.3%.
• The VideoStoryF variant, which fuses appearance, motion, and audio features, improves predictability and leads to higher accuracy than traditional fusion methods.
• The VideoStory0 variant, using a term-sensitive descriptiveness loss, enables accurate recognition from textual queries alone, demonstrating strong zero-shot generalization.
• The combination of term-sensitive descriptiveness and multimodal predictability losses is complementary and significantly boosts performance in zero-example recognition.
• VideoStory improves predictability across all underlying video features (e.g., CNN, I3D, audio) while maintaining high semantic descriptiveness, enabling state-of-the-art results.