[Paper Review] Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoning

Stuart James, Alessio Del Bue|arXiv (Cornell University)|Jul 16, 2018

Multimodal Machine Learning Applications29 references3 citations

TL;DR

This paper proposes Visual Graphs from Motion (VGfM), a method that leverages multi-view geometry from video sequences to infer 3D object relationships and construct accurate scene graphs. By fusing geometric and visual features through an RNN-based framework, the approach improves scene understanding in complex spatial arrangements, achieving state-of-the-art results on a newly created multi-view 3D scene graph dataset.

ABSTRACT

Recent approaches on visual scene understanding attempt to build a scene graph -- a computational representation of objects and their pairwise relationships. Such rich semantic representation is very appealing, yet difficult to obtain from a single image, especially when considering complex spatial arrangements in the scene. Differently, an image sequence conveys useful information using the multi-view geometric relations arising from camera motion. Indeed, in such cases, object relationships are naturally related to the 3D scene structure. To this end, this paper proposes a system that first computes the geometrical location of objects in a generic scene and then efficiently constructs scene graphs from video by embedding such geometrical reasoning. Such compelling representation is obtained using a new model where geometric and visual features are merged using an RNN framework. We report results on a dataset we created for the task of 3D scene graph generation in multiple views.

Motivation & Objective

• To address the challenge of constructing accurate, semantically rich scene graphs from single images, especially in complex spatial arrangements.
• To exploit multi-view geometric relations from video sequences to infer 3D scene structure and object relationships.
• To develop a unified framework that fuses geometric reasoning with visual features for improved scene understanding.
• To create and release a new benchmark dataset for multi-view 3D scene graph generation to support future research.

Proposed method

• The system first estimates the 3D geometric locations of objects in a scene using multi-view geometry from video input.
• It employs an RNN-based architecture to jointly model visual features and geometric constraints across frames.
• Geometric relationships between objects are derived from camera motion and 3D reconstruction, enabling spatial reasoning.
• Visual features are extracted from frames and fused with geometric embeddings to enrich scene graph construction.
• The model generates scene graphs by predicting object identities, attributes, and pairwise relationships grounded in 3D spatial structure.
• A novel dataset is collected and annotated to evaluate 3D scene graph generation across multiple views.

Experimental results

Research questions

• RQ1 Can multi-view video sequences improve the accuracy of scene graph construction compared to single-image methods?
• RQ2 How effectively can geometric reasoning from camera motion enhance the modeling of spatial relationships between objects?
• RQ3 To what extent does fusing geometric and visual features via an RNN improve scene graph quality in complex scenes?
• RQ4 How does the proposed method generalize across diverse 3D scene configurations and object arrangements?

Key findings

• The proposed VGfM framework achieves state-of-the-art performance on the newly created multi-view 3D scene graph dataset.
• Fusing geometric and visual features through an RNN leads to more accurate and consistent scene graph predictions than visual features alone.
• The use of multi-view geometry significantly improves the modeling of spatial relationships, especially for occluded or ambiguously positioned objects.
• The method demonstrates robustness in complex scenes with dense object arrangements, outperforming baseline single-view approaches.