[Paper Review] The Open World of Micro-Videos

Phuc Xuan Nguyen, Grégory Rogez+2 more|arXiv (Cornell University)|Mar 31, 2016

Human Pose and Action Recognition30 references19 citations

TL;DR

This paper introduces MV-58k, a large-scale open-world dataset of 264,327 micro-videos with 58,243 dynamic hashtags, enabling research in video understanding under open-world, viewpoint-diverse, and temporally evolving conditions. It proposes viewpoint-specific and temporally adaptive models using deep visual and motion features, showing that curated data improves accuracy and that temporal dynamics significantly affect tag prediction performance.

ABSTRACT

Micro-videos are six-second videos popular on social media networks with several unique properties. Firstly, because of the authoring process, they contain significantly more diversity and narrative structure than existing collections of video "snippets". Secondly, because they are often captured by hand-held mobile cameras, they contain specialized viewpoints including third-person, egocentric, and self-facing views seldom seen in traditional produced video. Thirdly, due to to their continuous production and publication on social networks, aggregate micro-video content contains interesting open-world dynamics that reflects the temporal evolution of tag topics. These aspects make micro-videos an appealing well of visual data for developing large-scale models for video understanding. We analyze a novel dataset of micro-videos labeled with 58 thousand tags. To analyze this data, we introduce viewpoint-specific and temporally-evolving models for video understanding, defined over state-of-the-art motion and deep visual features. We conclude that our dataset opens up new research opportunities for large-scale video analysis, novel viewpoints, and open-world dynamics.

Motivation & Objective

• To address the lack of large-scale, diverse, and dynamically evolving video datasets suitable for open-world video understanding.
• To study the impact of camera viewpoint—especially egocentric and self-facing views—on video recognition tasks.
• To model temporal dynamics in video content and tag popularity, reflecting real-world open-world evolution.
• To evaluate the effectiveness of curated vs. raw data in training robust video understanding systems.
• To explore lifelong learning and open-vocabulary recognition in streaming, real-world video data.

Proposed method

• The authors construct MV-58k, a dataset of 264,327 micro-videos collected from social media, annotated with 58,243 dynamic hashtags.
• They extract state-of-the-art deep visual features (e.g., I3D) and motion features (e.g., flow) for video representation.
• They design viewpoint-specific models to handle third-person, egocentric, and self-facing camera perspectives.
• They introduce temporally adaptive models that retrain on recent data to account for shifting tag popularity and semantic drift.
• They apply Platt calibration to causal models using a sliding window of past data to improve temporal prediction accuracy.
• They compare performance across curated (MV-40) and raw (MV-58k) data to evaluate data quality vs. quantity trade-offs.

Experimental results

Research questions

• RQ1 How does the diversity and narrative structure of micro-videos compare to traditional video snippet collections?
• RQ2 To what extent do egocentric and self-facing viewpoints in micro-videos challenge existing video understanding models?
• RQ3 How do tag popularity and semantic meaning evolve over time, and how can models adapt to these open-world dynamics?
• RQ4 What is the relative impact of data curation versus raw data volume on video tag prediction accuracy?
• RQ5 Can adaptive, causal models effectively predict future tag popularity in streaming video data?

Key findings

• Curated data (MV-40) achieves higher accuracy than a 2X larger raw dataset (MV-58k), indicating that data quality significantly outweighs quantity in tag prediction.
• A modest increase in raw training data (less than 2X) rivals the performance of a manually curated dataset, suggesting scalability of open-world data.
• Performance on tag prediction varies by tag: 'easy' tags (e.g., #cavs, #warriors) are learnable with little data, while 'unlearnable' tags (e.g., #revine, #lol) remain near-zero AP even with large training sets.
• Temporal models that use adaptive training on recent data outperform fixed training sets, with performance improving to 23.5% mAP when calibrated on a 3-week window.
• Non-causal models (using future data) achieve 28% mAP when calibrated on the ideal future distribution, indicating significant room for improvement in causal temporal modeling.
• The dataset exhibits strong long-tail statistics and dynamic tag evolution, validating its use as a benchmark for open-world, streaming video understanding.