[Paper Review] Video In Sentences Out

Andrei Barbu, Alexander Bridge+16 more|arXiv (Cornell University)|Apr 12, 2012

Human Pose and Action Recognition25 references15 citations

TL;DR

This paper presents a video-to-sentence generation system that produces rich, natural language descriptions of video clips by recognizing actions, tracking participants, assigning roles (agent/patient), and modeling spatial, temporal, and attribute-based relations. The system uses detection-based tracking and a body-posture codebook to infer verb, noun, adjective, prepositional, and adverbial components, achieving 25.5% accuracy in human evaluation for the top-ranked sentence per video.

ABSTRACT

We present a system that produces sentential descriptions of video: who did what to whom, and where and how they did it. Action class is rendered as a verb, participant objects as noun phrases, properties of those objects as adjectival modifiers in those noun phrases,spatial relations between those participants as prepositional phrases, and characteristics of the event as prepositional-phrase adjuncts and adverbial modifiers. Extracting the information needed to render these linguistic entities requires an approach to event recognition that recovers object tracks, the track-to-role assignments, and changing body posture.

Motivation & Objective

• To generate detailed, natural language sentences describing video content, including who did what to whom, where, and how.
• To overcome limitations of traditional video action recognition methods that fail to identify object roles, spatial relations, and object properties.
• To model linguistic structure—verbs, noun phrases, adjectives, prepositional phrases, and adverbs—based on visual and motion cues.
• To ensure referential clarity by generating adjectives only when necessary to disambiguate participants, following Gricean principles.
• To integrate motion tracking, object detection, and posture analysis to produce semantically accurate and contextually grounded descriptions.

Proposed method

• Uses detection-based tracking to maintain consistent object identities across video frames and extract motion trajectories.
• Applies a two-track Hidden Markov Model (HMM) to model event dynamics, including motion transitions and role assignments (agent, patient).
• Employs a body-posture codebook to infer person-pose adjectives (e.g., crouched, upright) from normalized part displacements and posture indices.
• Generates noun phrases with adjectival modifiers (color, size, shape) only when necessary to prevent coreference, following Gricean Maxim of Quantity.
• Constructs prepositional phrases for spatial relations (e.g., 'to the left of') based on 2D spatial positioning from the viewer’s perspective.
• Generates adverbial modifiers (e.g., slowly, leftward) from motion speed and direction derived from tracked trajectories.

Experimental results

Research questions

• RQ1 Can a system generate semantically rich, natural language sentences describing video events that include agents, patients, spatial relations, and motion characteristics?
• RQ2 How can object roles (agent vs. patient) be reliably inferred from visual motion and tracking data?
• RQ3 What visual cues are sufficient to generate linguistically appropriate adjectives (e.g., color, pose) without over-description?
• RQ4 To what extent can motion trajectories and spatial positioning be used to generate accurate prepositional phrases and adverbials?
• RQ5 How does the integration of tracking, posture analysis, and role assignment improve sentence accuracy compared to standard action recognition methods?

Key findings

• The system achieved a 25.5% human evaluation rate for truthfulness when generating the single most-likely sentence per video, indicating reliable semantic alignment with video content.
• For 49.4% of videos, at least one of the three generated sentences was judged true, demonstrating broad coverage of salient events.
• The system produced 18.4% salient descriptions (138 out of 749 videos), indicating effective identification of visually prominent events.
• The use of a body-posture codebook enabled accurate inference of person-pose adjectives such as 'crouched' or 'upright' from motion and pose data.
• The system avoided over-description by applying the Gricean Maxim of Quantity, generating adjectives only when necessary to prevent referential ambiguity.
• The integration of tracking, role assignment, and motion analysis enabled the generation of complex linguistic structures, such as 'the person to the left of the bicycle went away leftward'.