[Paper Review] Object Level Visual Reasoning in Videos

Fabien Baradel, Nathalia Neverova+3 more|arXiv (Cornell University)|Jun 16, 2018

Human Pose and Action RecognitionComputer Science37 references89 citations

TL;DR

This paper introduces an Object Relation Network (ORN) for reasoning about semantically meaningful object interactions in videos, combining Mask-RCNN based object detection with relational reasoning to achieve state-of-the-art results on SS, VLOG, and EPIC Kitchens.

ABSTRACT

Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object classes present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of understanding that pushes beyond this and call for models with capabilities for fine distinction and detailed comprehension of interactions between actors and objects in a scene. We propose a model capable of learning to reason about semantically meaningful spatiotemporal interactions in videos. The key to our approach is a choice of performing this reasoning at the object level through the integration of state of the art object detection networks. This allows the model to learn detailed spatial interactions that exist at a semantic, object-interaction relevant level. We evaluate our method on three standard datasets (Twenty-BN Something-Something, VLOG and EPIC Kitchens) and achieve state of the art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their interactions corresponding to different activity classes.

Motivation & Objective

• Motivate fine-grained understanding of human-object interactions beyond global motion and scene cues.
• Leverage explicit object detections to perform spatio-temporal reasoning about object relations in videos.
• Develop an end-to-end trainable architecture that reasons over object instances across time.
• Demonstrate that object-level reasoning yields improvements over activity-only baselines on challenging datasets.

Proposed method

• Introduce Object Relation Network (ORN) to reason between detected object instances across space and time.
• Use Mask-RCNN to obtain object masks and class predictions, with ROI-Pooling to extract per-object features.
• Model pairwise inter-frame object relations with a function h_theta and aggregate via a global function g, then propagate with a recurrent f_phi (GRU) to capture long-range dependencies.
• Combine object-reasoning representations with a separate activity head that captures global motion context.
• Train with a joint loss: activity classification loss plus an auxiliary object-class consistency loss to align object features with semantic classes.

Experimental results

Research questions

• RQ1 Can object-level, semantically grounded relational reasoning improve activity recognition in videos beyond traditional global-motion models?
• RQ2 Does explicit cross-frame object interaction reasoning (ORN) with recurrence lead to better performance on fine-grained video understanding tasks?
• RQ3 How does using semantically defined object instances compare to pixel-level relational reasoning for video activity recognition?
• RQ4 What is the impact of jointly training object head and activity head versus training them separately?

Key findings

• On VLOG, the proposed method achieves 44.7% mAP, outperforming the previous best (40.5%).
• On Something-Something, the method improves by 2.3 points over the state of the art.
• On EPIC Kitchens, the method attains 40.89% accuracy, outperforming baselines by roughly 6.4–7.9 points depending on the baselines.
• Ablation shows that adding object-level reasoning provides notable gains (0.8–2.5+ points) over an activity-head baseline across datasets.
• Using semantically defined objects further improves EPIC by about 2 points and VLOG by about 2.3 points compared to pixel-level reasoning.