[Paper Review] Recurrent Space-time Graph Neural Networks

Andrei Liviu Nicolicioiu, Iulia Duta+1 more|arXiv (Cornell University)|Apr 11, 2019

Human Pose and Action Recognition62 references19 citations

TL;DR

This paper proposes Recurrent Space-time Graph (RSTG) neural networks, a novel graph-based model that factorizes and processes space and time separately yet jointly through iterative message passing across nodes and recurrent memory states. The model achieves state-of-the-art performance on the Something-Something-v1 dataset, outperforming strong baselines by over 8% in Top-1 accuracy with a 2D ConvNet backbone and by 3.1% with a 3D-ResNet-50 backbone.

ABSTRACT

Learning in the space-time domain remains a very challenging problem in machine learning and computer vision. Current computational models for understanding spatio-temporal visual data are heavily rooted in the classical single-image based paradigm. It is not yet well understood how to integrate information in space and time into a single, general model. We propose a neural graph model, recurrent in space and time, suitable for capturing both the local appearance and the complex higher-level interactions of different entities and objects within the changing world scene. Nodes and edges in our graph have dedicated neural networks for processing information. Nodes operate over features extracted from local parts in space and time and previous memory states. Edges process messages between connected nodes at different locations and spatial scales or between past and present time. Messages are passed iteratively in order to transmit information globally and establish long range interactions. Our model is general and could learn to recognize a variety of high level spatio-temporal concepts and be applied to different learning tasks. We demonstrate, through extensive experiments and ablation studies, that our model outperforms strong baselines and top published methods on recognizing complex activities in video. Moreover, we obtain state-of-the-art performance on the challenging Something-Something human-object interaction dataset.

Motivation & Objective

• To address the challenge of learning complex spatio-temporal interactions in video data, where current models are often limited by fixed architectures or poor long-range modeling.
• To develop a general-purpose, end-to-end trainable neural network that can effectively capture both local appearance and global, higher-order spatial-temporal relationships.
• To factorize space and time processing within a unified graph framework, enabling efficient and flexible modeling of dynamic visual scenes.
• To demonstrate the model's effectiveness on real-world, complex human-object interaction datasets, particularly Something-Something-v1.
• To provide ablation studies justifying the design choices, including the role of recurrent memory and separate space-time processing.

Proposed method

• The model uses a backbone network (e.g., I3D) to extract spatio-temporal features, which are then fed into a recurrent space-time graph with nodes representing spatial regions and edges encoding messages between nodes across space and time.
• Each node processes local features, messages from neighboring spatial nodes, and its own recurrent memory state using dedicated neural networks, enabling dynamic state updates.
• The graph alternates between spatial and temporal processing stages within each iteration, allowing for iterative message passing that establishes long-range dependencies across space and time.
• Two variants are introduced: RSTG-to-vec, which produces a global video-level embedding, and RSTG-to-map, which outputs a 3D feature map at each time step, preserving spatial resolution.
• Residual connections are used to stabilize training by adding the graph output to the backbone features, ensuring gradient flow and improved optimization.
• The model is trained end-to-end using standard cross-entropy loss on video classification tasks, with data augmentation via random cropping and frame sampling.

Experimental results

Research questions

• RQ1 Can a unified graph neural network architecture effectively model both spatial and temporal dependencies in video data through separate, recurrent processing stages?
• RQ2 How does the factorization of space and time processing affect performance compared to joint or sequential modeling?
• RQ3 What is the impact of placing the graph at different depths within a 3D CNN backbone on downstream recognition accuracy?
• RQ4 Can the proposed model achieve state-of-the-art performance on complex, real-world human-object interaction datasets like Something-Something-v1?
• RQ5 How does the computational complexity of the RSTG model compare to existing methods such as Non-Local networks, especially under similar accuracy conditions?

Key findings

• The RSTG-to-map model placed after the res3 and res4 blocks of the I3D backbone achieved the highest performance, with 49.2% Top-1 accuracy on the Something-Something-v1 dataset.
• With a 2D ConvNet backbone, the RSTG model improved Top-1 accuracy by over 8% compared to all other methods using the same backbone architecture.
• When using the 3D-ResNet-50 backbone, the RSTG model achieved a 3.1% improvement over the best published method in the Top-1 accuracy setting.
• The RSTG-to-vec variant achieved better inference speed than the Non-Local network while maintaining higher accuracy, demonstrating favorable efficiency-accuracy trade-offs.
• Ablation studies confirmed that both separate space and time processing and the use of recurrent memory states are critical for performance, with ablation on either component leading to significant accuracy drops.
• The model demonstrated strong generalization, achieving competitive results across multiple backbone placements, with the best performance achieved when the graph was inserted at intermediate layers (res3 and res4).