[Paper Review] Ego-Downward and Ambient Video based Person Location Association

Liang Yang, Hao Jiang+2 more|arXiv (Cornell University)|Dec 2, 2018

Video Surveillance and Tracking Methods28 references4 citations

TL;DR

This paper proposes a novel action and motion feature-based model for cross-view person localization and tracking using ego-downward and third-person static camera views. By leveraging 3D pose estimation and temporal-spatial alignment, the method achieves 18.32% higher accuracy than baseline models, with 74.22% verification accuracy and 67.767% average accuracy in multi-person scenarios, demonstrating robustness to occlusion and similar appearances.

ABSTRACT

Using an ego-centric camera to do localization and tracking is highly needed for urban navigation and indoor assistive system when GPS is not available or not accurate enough. The traditional hand-designed feature tracking and estimation approach would fail without visible features. Recently, there are several works exploring to use context features to do localization. However, all of these suffer severe accuracy loss if given no visual context information. To provide a possible solution to this problem, this paper proposes a camera system with both ego-downward and third-static view to perform localization and tracking in a learning approach. Besides, we also proposed a novel action and motion verification model for cross-view verification and localization. We performed comparative experiments based on our collected dataset which considers the same dressing, gender, and background diversity. Results indicate that the proposed model can achieve $18.32 \%$ improvement in accuracy performance. Eventually, we tested the model on multi-people scenarios and obtained an average $67.767 \%$ accuracy.

Motivation & Objective

• To address the challenge of person localization and tracking in GPS-denied environments such as indoor or urban areas where visual features are limited.
• To overcome the failure of traditional feature-based tracking in low-texture or occluded environments by leveraging motion and action cues.
• To improve cross-view verification between ego-downward and third-person static cameras, which are inherently challenging due to limited field of view and pose variance.
• To develop a robust, learning-based system that generalizes across diverse appearances, backgrounds, and motion patterns in real-world settings.

Proposed method

• A YOLO-based tracker is used to maintain continuous person detection across ego-downward and third-view video sequences.
• 3D human pose estimation is performed on both ego and third-view data to enable spatial alignment and motion modeling.
• A joint temporal-spatial siamese network is designed to learn action and motion features for cross-view verification, using pose and motion as key signals.
• The model integrates ego-view motion and translation features with third-view odometry to improve localization accuracy through multi-modal feature fusion.
• A Bayes filter with velocity prediction is applied to refine verification outputs, especially in multi-person and crossing scenarios.
• The system processes short video clips online, enabling real-time inference by leveraging geometric and motion consistency across views.

Experimental results

Research questions

• RQ1 Can action and motion features alone enable accurate cross-view person localization when visual appearance is unreliable due to similar clothing or occlusion?
• RQ2 How does the integration of 3D pose estimation improve alignment and verification between ego-downward and third-person camera views?
• RQ3 What is the relative contribution of ego-view motion versus third-view odometry in enhancing localization accuracy?
• RQ4 How does the model perform under challenging conditions such as multi-person interactions and body pose similarity?
• RQ5 Can the proposed method generalize across diverse backgrounds, clothing, and motion patterns without relying on visual context?

Key findings

• The proposed ETVVT model achieves 74.22% verification accuracy, representing an 18.32% improvement over the baseline model.
• The action model outperforms the motion model with 72.5% accuracy versus 70.03%, indicating that pose-based action recognition is more discriminative than motion-only features.
• Third-view translation features improve validation accuracy from 79.05% to 81.80%, showing that third-view motion cues are more reliable than ego-view odometry.
• In multi-person scenarios, the model achieves an average accuracy of 67.767%, with higher performance in non-crossing cases and lower accuracy in group crossing due to partial occlusion.
• The Bayes filter significantly improves results in low-complexity scenarios, achieving up to 96.17% accuracy in two-person non-crossing cases.
• The model fails when multiple people have identical actions and appearances, highlighting limitations in distinguishing identical motion patterns.