[Paper Review] Complex Event Recognition from Images with Few Training Examples

Unaiza Ahsan, Chen Sun+2 more|arXiv (Cornell University)|Jan 17, 2017

Domain Adaptation and Few-Shot Learning48 references4 citations

TL;DR

This paper proposes a web-driven concept discovery framework that leverages Wikipedia and Flickr tags to learn discriminative, event-specific visual concepts from web images, using pretrained CNN features to classify complex social events with only one training example. It achieves state-of-the-art performance on few-shot event recognition, outperforming direct CNN fine-tuning on multiple benchmarks including a rare events dataset with unseen categories.

ABSTRACT

We propose to leverage concept-level representations for complex event recognition in photographs given limited training examples. We introduce a novel framework to discover event concept attributes from the web and use that to extract semantic features from images and classify them into social event categories with few training examples. Discovered concepts include a variety of objects, scenes, actions and event sub-types, leading to a discriminative and compact representation for event images. Web images are obtained for each discovered event concept and we use (pretrained) CNN features to train concept classifiers. Extensive experiments on challenging event datasets demonstrate that our proposed method outperforms several baselines using deep CNN features directly in classifying images into events with limited training examples. We also demonstrate that our method achieves the best overall accuracy on a dataset with unseen event categories using a single training example.

Motivation & Objective

• To address the challenge of recognizing complex social and news events in images when only a few labeled examples are available.
• To develop a scalable, web-based method for discovering relevant visual concepts (objects, scenes, actions, attributes) without manual annotation.
• To create a compact, discriminative feature representation using web-derived concepts that generalizes to unseen event categories.
• To evaluate the method on rare and diverse real-world events where traditional CNNs struggle due to limited supervision.

Proposed method

• Use Wikipedia to define event categories and extract related keywords as initial concept seeds.
• Apply a tweet segmentation algorithm to noisy Flickr tags to extract event-centric phrases.
• Project phrases into word2vec embeddings and retrieve nearest neighbors to expand the concept pool.
• Collect web images via Bing image search for each discovered concept and extract deep CNN features from a pretrained network.
• Train concept-specific classifiers using web images and their CNN features to generate concept scores per test image.
• Use the resulting concept scores as a high-level, compact representation for event classification.

Figure 1: Event concepts as an intermediate feature representation for recognizing social events in photographs.

Experimental results

Research questions

• RQ1 Can web-derived visual concepts improve few-shot recognition of complex social events in still images?
• RQ2 Does a concept-based representation generalize to event categories not seen during concept discovery?
• RQ3 How does web-supervised concept learning compare to direct fine-tuning of deep CNNs on limited training data?
• RQ4 Can the method achieve strong performance on rare or previously unseen real-world events with only one training example?

Key findings

• The proposed method achieves 82.09% average accuracy on the WIDER dataset using only one training example per class, outperforming AlexNet-fc7 (59.79%) and WEBLY-fc7 (55.39%).
• On the Rare Events Dataset (RED), the method outperforms baselines in 10 out of 21 classes under one-shot learning, demonstrating generalization to unseen categories.
• Even with full training data, the method achieves 96.68% accuracy on UIUC Sports and 78.59% on WIDER, surpassing state-of-the-art baselines.
• The concept-based representation is more discriminative than direct CNN features for complex event recognition under low-shot conditions.
• The method generalizes well to diverse real-world events such as 'Boston Bombing' and 'Nepal Earthquake', where traditional models fail due to data scarcity.

Figure 2: Generated segments from Flickr tags for event label ‘protest.’