[Paper Review] Predicting Human Activities Using Stochastic Grammar

Siyuan Qi, Siyuan Huang+2 more|arXiv (Cornell University)|Aug 2, 2017

Human Pose and Action Recognition2 references3 citations

TL;DR

This paper proposes a stochastic grammar-based model using a Spatial-Temporal And-Or Graph (ST-AOG) to predict future human activities from partially observed RGB-D videos. It integrates human actions, objects, and affordances via a temporal stochastic grammar and Earley parsing to infer future sub-activities and their associated labels, achieving state-of-the-art performance in semantic event parsing and future activity prediction on benchmark datasets.

ABSTRACT

This paper presents a novel method to predict future human activities from partially observed RGB-D videos. Human activity prediction is generally difficult due to its non-Markovian property and the rich context between human and environments. We use a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances. We represent the event by a spatial-temporal And-Or graph (ST-AOG). The ST-AOG is composed of a temporal stochastic grammar defined on sub-activities, and spatial graphs representing sub-activities that consist of human actions, objects, and their affordances. Future sub-activities are predicted using the temporal grammar and Earley parsing algorithm. The corresponding action, object, and affordance labels are then inferred accordingly. Extensive experiments are conducted to show the effectiveness of our model on both semantic event parsing and future activity prediction.

Motivation & Objective

• To model the hierarchical and compositional structure of human activities beyond Markovian assumptions.
• To enable online inference of current activity states and prediction of future actions from partial video observations.
• To jointly model human actions, interacting objects, and object affordances for richer spatial-temporal context capture.
• To address the challenge of predicting from a large future state space using symbolic reasoning and probabilistic grammar.

Proposed method

• Represent human activities using a Spatial-Temporal And-Or Graph (ST-AOG), combining a temporal stochastic grammar (T-AOG) with spatial graphs (S-AOG) for sub-activities.
• Model sub-activities as nonterminals in a stochastic context-free grammar, with terminals representing action-object-affordance triplets.
• Use the Earley parsing algorithm to symbolically predict the next sub-activity based on current observations and grammar rules.
• Compute parsing likelihoods using Viterbi-style inference to estimate probabilities of predicted sequences.
• Infer action, object, and affordance labels by combining parsed sub-activity predictions with learned feature representations.
• Employ a coverage threshold of 0.5 and context window size of 4 for bootstrapping equivalence classes during training.

Experimental results

Research questions

• RQ1 How can we model the non-Markovian, hierarchical structure of human activities in a way that supports future prediction?
• RQ2 What is the role of object affordances and spatial context in improving activity prediction accuracy?
• RQ3 Can a stochastic grammar model effectively capture compositional event structures in RGB-D video sequences?
• RQ4 How does symbolic parsing with Earley’s algorithm compare to end-to-end deep learning models in activity prediction?
• RQ5 To what extent can the model generalize to unseen future states using only partial video observations?

Key findings

• The proposed ST-AOG model outperforms baseline methods including SVM, LSTM, and VGG-16 in action detection and future activity prediction tasks.
• The use of stochastic grammar and Earley parsing enables effective symbolic prediction of future sub-activities with high accuracy.
• Incorporating object affordances and spatial context significantly improves prediction performance compared to models ignoring these factors.
• The model achieves strong results in semantic event parsing, demonstrating its ability to recover hierarchical activity structures from video.
• The parsing likelihood computation via Viterbi-style inference provides reliable probability estimates for predicted future states.
• The ablation study confirms that both the temporal grammar and spatial context components are essential for optimal performance.