[Paper Review] EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset

Hao Tang, Kevin Liang+2 more|arXiv (Cornell University)|Jan 9, 2023

Video Surveillance and Tracking Methods6 citations

TL;DR

EgoTracks introduces the first large-scale, long-term egocentric visual object tracking dataset, sourced from the Ego4D benchmark, to address the critical challenges of re-detection, occlusion, and scale variation in first-person video. The paper proposes EgoSTARK, a modified STARK tracker with scale augmentation and improved search region design, achieving a 10% boost in AO and 8%+ improvement in F-score on EgoTracks, significantly outperforming standard trackers on egocentric data.

ABSTRACT

Visual object tracking is a key component to many egocentric vision problems. However, the full spectrum of challenges of egocentric tracking faced by an embodied AI is underrepresented in many existing datasets; these tend to focus on relatively short, third-person videos. Egocentric video has several distinguishing characteristics from those commonly found in past datasets: frequent large camera motions and hand interactions with objects commonly lead to occlusions or objects exiting the frame, and object appearance can change rapidly due to widely different points of view, scale, or object states. Embodied tracking is also naturally long-term, and being able to consistently (re-)associate objects to their appearances and disappearances over as long as a lifetime is critical. Previous datasets under-emphasize this re-detection problem, and their "framed" nature has led to adoption of various spatiotemporal priors that we find do not necessarily generalize to egocentric video. We thus introduce EgoTracks, a new dataset for long-term egocentric visual object tracking. Sourced from the Ego4D dataset, this new dataset presents a significant challenge to recent state-of-the-art single-object tracking models, which we find score poorly on traditional tracking metrics for our new dataset, compared to popular benchmarks. We further show improvements that can be made to a STARK tracker to significantly increase its performance on egocentric data, resulting in a baseline model we call EgoSTARK. We publicly release our annotations and benchmark, hoping our dataset leads to further advancements in tracking.

Motivation & Objective

• Address the lack of large-scale, long-term egocentric visual object tracking datasets that reflect real-world challenges like frequent occlusions, reappearances, and viewpoint changes.
• Identify why state-of-the-art short-term trackers fail on egocentric video due to over-reliance on spatiotemporal priors that do not generalize to egocentric dynamics.
• Develop a robust baseline model, EgoSTARK, tailored for egocentric tracking by adapting training and inference strategies to handle scale variation and long-term re-detection.
• Enable future research through public release of annotations, benchmark, and a planned challenge to advance long-term egocentric tracking.

Proposed method

• Source videos from the Ego4D dataset, a large-scale collection of unscripted, in-the-wild egocentric videos of daily-life activities.
• Annotate 22,028 object tracks across 5,708 videos with average duration of 6 minutes, emphasizing long-term tracking, re-detection, and occlusion events.
• Introduce scale augmentation during training by randomly resizing the search image by a factor of $ s ∈ [0.5, 1.5] $, improving robustness to rapid scale changes.
• Optimize the search region size via the search region ratio (SRR) and search image size (SIS), finding that larger search regions significantly improve re-detection performance.
• Use a fixed template size of $ 128 \times 128 $, and control context ratio (CR) to evaluate its impact, settling on CR = 2 as optimal based on F-score results.
• Propose and evaluate a multi-template fusion strategy in EgoSTARK to learn more global, view-variant object representations, improving tracking robustness.

Figure 1: A video from the proposed EgoTracks dataset, with yellow clip segments marking when the object ( blowtorch ) is visible. Note the frequent disappearances and reappearances of the object over an 8 minute video, with lengthy absences, necessitating re-detection to track accurately without fa

Experimental results

Research questions

• RQ1 Why do state-of-the-art single-object trackers perform poorly on egocentric video despite strong performance on existing benchmarks?
• RQ2 To what extent do spatiotemporal priors—such as gradual motion and consistent scale—fail to generalize to egocentric tracking scenarios?
• RQ3 How can training and inference strategies be adapted to improve re-detection and robustness to large viewpoint changes and occlusions in egocentric video?
• RQ4 What impact do scale augmentation and expanded search regions have on tracker performance in long-term egocentric tracking?
• RQ5 Can multi-template fusion improve long-term tracking performance by enabling more global, view-variant object representations in egocentric settings?

Key findings

• State-of-the-art trackers, including SOTA models, score significantly lower on EgoTracks than on traditional benchmarks, indicating a major performance gap due to egocentric-specific challenges.
• Scale augmentation during training improves STARK’s AO by nearly 10% and F-score by over 8%, demonstrating that scale variation is a key factor in egocentric tracking failure.
• Larger search region sizes (via SRR and SIS) have a higher impact on F-score than context ratio, confirming that re-detection requires broader spatial search capability.
• The context ratio has mixed effects on performance, so the common practice of using CR = 2 is retained as a stable baseline.
• Multi-template fusion in EgoSTARK leads to measurable performance gains, suggesting that global, view-variant representations are beneficial for egocentric tracking.
• The proposed EgoSTARK baseline significantly outperforms standard STARK on EgoTracks, establishing a new benchmark for long-term egocentric tracking.

Figure 2: EgoTracks is an order of magnitude larger than past long-term VOT datasets, with significantly more tracks (circle area) and object disappearances/appearances in longer videos.