[Paper Review] Action recognition in still images by latent superpixel classification

Shaukat Abidi, Massimo Piccardi+1 more|arXiv (Cornell University)|Jul 30, 2015

Human Pose and Action Recognition14 references3 citations

TL;DR

This paper proposes a novel action recognition approach in still images by leveraging superpixel segmentation and latent superpixel classification via a latent structural SVM. It first detects object-like regions (superpixels) and applies pre-trained detectors to extract features, then jointly infers superpixel classes and action labels using a structured graphical model, achieving 74.06% average accuracy on the Stanford 40 Actions dataset.

ABSTRACT

Action recognition from still images is an important task of computer vision applications such as image annotation, robotic navigation, video surveillance and several others. Existing approaches mainly rely on either bag-of-feature representations or articulated body-part models. However, the relationship between the action and the image segments is still substantially unexplored. For this reason, in this paper we propose to approach action recognition by leveraging an intermediate layer of "superpixels" whose latent classes can act as attributes of the action. In the proposed approach, the action class is predicted by a structural model(learnt by Latent Structural SVM) based on measurements from the image superpixels and their latent classes. Experimental results over the challenging Stanford 40 Actions dataset report a significant average accuracy of 74.06% for the positive class and 88.50% for the negative class, giving evidence to the performance of the proposed approach.

Motivation & Objective

• To explore the relationship between image segments (superpixels) and action classes in still images, which remains underexplored in existing methods.
• To improve action recognition performance by modeling superpixel classes as intermediate attributes that inform the final action prediction.
• To address the challenge of recognizing actions from single frames, especially in scenarios with limited temporal information such as surveillance or robotics.
• To develop a joint inference framework that combines superpixel classification and action recognition using a structured prediction model.
• To evaluate the method on a highly challenging benchmark dataset, Stanford 40 Actions, with a focus on balanced positive and negative class performance.

Proposed method

• The image is over-segmented into superpixels using a superpixel algorithm to create spatially coherent, homogeneous regions.
• A set of 23 pre-trained object detectors (e.g., for 'face', 'bike', 'phone') is applied to each superpixel to extract detector score vectors as measurements.
• A latent structural SVM framework is used to jointly predict superpixel classes and the global action class, modeling dependencies between superpixels via a fully-connected graphical model.
• The model uses a loss-augmented inference step with a 0-1 loss function to optimize the prediction of the correct action class during training.
• The graphical model includes three layers: input measurements (detector scores), hidden states (superpixel classes), and output (action class), with interdependencies across all nodes.
• Approximate inference is performed via a greedy algorithm that iteratively maximizes over each superpixel's class label while maintaining consistency with the action prediction.

Experimental results

Research questions

• RQ1 Can superpixel-level latent classes serve as effective intermediate representations for recognizing actions in still images?
• RQ2 How does modeling joint superpixel and action classification improve performance compared to independent or bag-of-features approaches?
• RQ3 To what extent can pre-trained object detectors, trained on unrelated datasets, generalize to action recognition in still images?
• RQ4 Does the use of a structured prediction model with latent variables enhance recognition accuracy on a challenging, fine-grained action dataset?
• RQ5 What is the performance gap between positive and negative class recognition, and how does it reflect model balance and generalization?

Key findings

• The proposed method achieves an average accuracy of 74.06% on the positive class and 88.50% on the negative class across all 40 action classes in the Stanford 40 Actions dataset.
• The model significantly outperforms prior work, such as the 55.93% accuracy reported in a recent study, demonstrating its effectiveness on a highly challenging benchmark.
• Some actions like 'rowing a boat' achieve over 93% accuracy on positive samples, indicating strong performance for clearly identifiable actions.
• Actions like 'running' show lower positive accuracy (48.99%), highlighting the difficulty of recognizing dynamic actions from static frames.
• The method maintains good balance between positive and negative predictions, as evidenced by the high negative class accuracy, suggesting robustness to class imbalance.
• The use of pre-trained detectors from unrelated datasets (e.g., microscopes, telescopes) without retraining still yields strong performance, indicating generalization capability.