[Paper Review] Temporal Dynamic Graph LSTM for Action-driven Video Object Detection

Yuan Yuan, Xiaodan Liang+3 more|arXiv (Cornell University)|Aug 2, 2017

Human Pose and Action Recognition43 references14 citations

TL;DR

This paper proposes a Temporal Dynamic Graph LSTM (TD-Graph LSTM) for action-driven weakly-supervised video object detection, leveraging action descriptions as supervision to overcome the missing label problem. By dynamically constructing a temporal graph based on visual correlations between object proposals across frames, the model enables global temporal reasoning and knowledge transfer, achieving state-of-the-art performance on the Charades dataset with a 1.98% detection mAP and 19.52% classification mAP.

ABSTRACT

In this paper, we investigate a weakly-supervised object detection framework. Most existing frameworks focus on using static images to learn object detectors. However, these detectors often fail to generalize to videos because of the existing domain shift. Therefore, we investigate learning these detectors directly from boring videos of daily activities. Instead of using bounding boxes, we explore the use of action descriptions as supervision since they are relatively easy to gather. A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label". To tackle this problem, we propose a novel temporal dynamic graph Long Short-Term Memory network (TD-Graph LSTM). TD-Graph LSTM enables global temporal reasoning by constructing a dynamic graph that is based on temporal correlations of object proposals and spans the entire video. The missing label issue for each individual frame can thus be significantly alleviated by transferring knowledge across correlated objects proposals in the whole video. Extensive evaluations on a large-scale daily-life action dataset (i.e., Charades) demonstrates the superiority of our proposed method. We also release object bounding-box annotations for more than 5,000 frames in Charades. We believe this annotated data can also benefit other research on video-based object recognition in the future.

Motivation & Objective

• To address the missing label problem in weakly-supervised video object detection, where object categories not involved in action descriptions are often undetected.
• To leverage temporal coherence in videos to transfer knowledge from action-labeled frames to unlabeled frames with uninvolved objects.
• To design a dynamic graph-based recurrent architecture that adaptively models temporal correlations between object proposals across frames.
• To improve detection performance on small and occluded objects in daily-life video scenarios with minimal supervision.
• To release a new annotated dataset of over 5,000 frames with bounding boxes to support future video recognition research.

Proposed method

• The TD-Graph LSTM constructs a dynamic temporal graph where nodes represent object proposals across all frames, and edges are adaptively formed based on visual similarity between proposals in consecutive frames.
• At each time step, only the most similar proposals are connected via edges, enabling adaptive and context-aware information propagation through the graph.
• The model uses an LSTM unit to process sequential features along the dynamic graph, allowing for long-range temporal reasoning and feature refinement.
• Graph edges are updated iteratively based on current feature representations, enabling the model to evolve its connectivity based on learned visual correlations.
• The framework jointly optimizes detection and classification by propagating action knowledge across the entire video through the dynamic graph structure.
• Ablation studies compare dynamic graphs against static, fully-connected, and mean-weighted graphs to validate the effectiveness of adaptive topology learning.

Experimental results

Research questions

• RQ1 Can temporal correlations between object proposals in videos be effectively modeled to alleviate the missing label problem in weakly-supervised detection?
• RQ2 Does a dynamic graph structure that evolves over time and model states improve detection performance compared to fixed or static graph topologies?
• RQ3 Can action descriptions be effectively leveraged to infer the presence of uninvolved objects across frames, even when they are not explicitly labeled?
• RQ4 How does the integration of temporal reasoning via LSTM on a dynamic graph impact detection accuracy for small and occluded objects?
• RQ5 To what extent does global video-level action knowledge improve frame-level object detection when only partial labels are available?

Key findings

• The proposed TD-Graph LSTM achieves a detection mAP of 1.98% and classification mAP of 19.52% on the Charades dataset, outperforming state-of-the-art methods by 3.85% in classification mAP.
• The model improves detection performance on small objects by over 14.13% for the pillow class and 6.93% for the cup class, demonstrating effectiveness in challenging detection scenarios.
• Ablation studies confirm that the dynamic graph structure significantly outperforms static and mean-weighted graph baselines, proving the value of adaptive edge selection.
• Removing the graph component reduces detection mAP to 1.55%, confirming that graph-based knowledge transfer is essential for performance gains.
• The model with LSTM units outperforms the variant without LSTM, validating the importance of sequential modeling in capturing complex motion patterns.
• Qualitative results show superior detection accuracy on small and occluded objects, such as cups and sofas, compared to baselines.