[Paper Review] Visual Semantic Role Labeling

Saurabh Gupta, Jitendra Malik|arXiv (Cornell University)|May 17, 2015

Human Pose and Action RecognitionComputer Science4 references330 citations

TL;DR

This paper introduces Visual Semantic Role Labeling (VSRL), a novel task that extends action recognition by localizing agents and their associated objects in semantic roles (e.g., agent, instrument, object) for fine-grained actions. The authors present a new dataset of 16K person instances across 10K COCO images with detailed annotations for 26 action classes and baseline models using CNN-based detectors, revealing key error modes and future research directions.

ABSTRACT

In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.

Motivation & Objective

• To enable fine-grained visual understanding of actions beyond coarse activity classification by localizing semantic roles (agent, instrument, object) in images.
• To address the limitation of existing datasets that label only actions or objects, not both with role associations.
• To create a benchmark dataset with detailed annotations linking people, actions, and objects in semantic roles for complex, cluttered scenes.
• To develop and evaluate baseline algorithms for joint detection of agents, action classification, and object role localization.
• To analyze failure modes and identify key challenges for future research in visual-semantic grounding.

Proposed method

• Annotate 10,000 COCO images with 16,000 person instances, each labeled with one of 26 action classes and associated with objects in semantic roles (e.g., agent, instrument, object).
• Use a two-stage detection pipeline: first detect people and classify their actions, then localize objects in specific semantic roles using region proposals and CNN-based detectors.
• Design a full model (C) that explicitly models deformation between agent and object, improving localization accuracy over a baseline model (C₀) without deformation modeling.
• Apply IoU (Intersection over Union) thresholds to classify detection errors into 8 distinct error modes (e.g., incorrect label, mislocalization, hallucination, mispairing).
• Train and evaluate four baseline models using region proposals and CNN features, comparing performance across action classes and error types.
• Use the COCO dataset as a testbed, extending it with detailed action and role annotations to support future research in visual-semantic grounding.

Experimental results

Research questions

• RQ1 How can we move beyond coarse action classification to localize agents and their associated objects in semantic roles within images?
• RQ2 What are the key failure modes in detecting agents and their semantic roles, and how can they be quantitatively analyzed?
• RQ3 To what extent does modeling deformation between agent and object improve localization accuracy in visual semantic role labeling?
• RQ4 How do object size, pose variation, and background clutter affect performance in role localization?
• RQ5 Can existing object detectors be adapted to jointly detect agents, classify actions, and assign objects to semantic roles?

Key findings

• The most dominant error mode in baseline models is incorrect action classification, indicating a major challenge in accurate action recognition.
• Object mislocalization is prevalent for actions like 'ski', 'surf', 'skateboard', and 'snowboard', especially when objects are small or partially occluded.
• The 'person misloc' error is significant for actions like 'lay' due to unusual or non-standard agent poses.
• The 'mis pairing' error decreases when deformation between agent and object is modeled, showing the benefit of spatial context.
• Object hallucinations on the background are a major issue for actions like 'cut' and 'hit-obj', particularly in cluttered scenes.
• The full model (C) that incorporates deformation modeling shows improved performance over the baseline (C₀), especially in reducing mislocalization and mispairing errors.