[Paper Review] On the Role of Event Boundaries in Egocentric Activity Recognition from Photostreams

Alejandro Cartas, Estefanía Talavera+2 more|arXiv (Cornell University)|Sep 2, 2018

Human Pose and Action Recognition13 references3 citations

TL;DR

This paper investigates the impact of event boundaries on egocentric activity recognition from photostreams, proposing a method that leverages automatically extracted event segments via SR-Clustering to improve recognition performance. Using a new dataset of 15 users' full-day photostreams, it demonstrates that event-based temporal segmentation significantly boosts accuracy—achieving 75.59% with a CNN+Bidirectional LSTM model—outperforming fixed-size sliding windows and highlighting generalization to unseen users.

ABSTRACT

Event boundaries play a crucial role as a pre-processing step for detection, localization, and recognition tasks of human activities in videos. Typically, although their intrinsic subjectiveness, temporal bounds are provided manually as input for training action recognition algorithms. However, their role for activity recognition in the domain of egocentric photostreams has been so far neglected. In this paper, we provide insights of how automatically computed boundaries can impact activity recognition results in the emerging domain of egocentric photostreams. Furthermore, we collected a new annotated dataset acquired by 15 people by a wearable photo-camera and we used it to show the generalization capabilities of several deep learning based architectures to unseen users.

Motivation & Objective

• To investigate the role of event boundaries in improving activity recognition performance in egocentric photostreams.
• To evaluate the generalization capability of deep learning models to unseen users in egocentric activity recognition.
• To compare different temporal segmentation strategies—fixed-size windows, full sequences, and event-based segmentation—on recognition performance.
• To introduce a new annotated dataset of 102,227 images from 15 users wearing wearable cameras for full-day photostreams.
• To assess the effectiveness of deep learning architectures, particularly bidirectional LSTMs, when trained on event-segmented sequences.

Proposed method

• The authors collected a new egocentric photostream dataset with 102,227 images from 15 users, captured at ~30-second intervals.
• Event boundaries were automatically extracted using SR-Clustering, which groups temporally adjacent images with similar visual and semantic features.
• Three deep learning models were trained: CNN+LSTM, CNN+Bidirectional LSTM, and CNN+RF+LSTM, using both full sequences and segmented event sequences.
• The models were evaluated using accuracy, macro-precision, macro-recall, and macro-F1-score across different temporal segmentation strategies.
• A sliding window approach with fixed-size segments was used as a baseline for comparison.
• The Xception network was used as the base CNN for all models, with temporal modeling performed via recurrent networks on image embeddings.

Experimental results

Research questions

• RQ1 Does incorporating automatically detected event boundaries improve activity recognition performance in egocentric photostreams compared to fixed-size sliding windows?
• RQ2 How do different temporal segmentation strategies—full sequence, fixed-size segments, and event-based segmentation—affect recognition accuracy and robustness?
• RQ3 To what extent do deep learning models generalize to unseen users when trained on egocentric photostreams?
• RQ4 Can event-based segmentation enhance recognition performance, especially in cases with short test sequences?
• RQ5 What is the relative performance of bidirectional LSTMs versus unidirectional LSTMs and RF-augmented models in this setting?

Key findings

• The CNN+Bidirectional LSTM model achieved the highest accuracy of 75.59% when trained on event-segmented sequences, outperforming all other models.
• Event-based segmentation (SR-Clustering) improved performance over fixed-size sliding windows for two out of three LSTM models, indicating better temporal coherence capture.
• The best macro-F1-score of 51.21% was achieved with the CNN+Bidirectional LSTM on event-segmented data, significantly outperforming the baseline sliding window approach.
• The CNN+RF+LSTM model underperformed compared to its baseline, likely due to overfitting of the RF component, especially on unseen users.
• The model generalization was most effective when using event boundaries, as evidenced by improved macro-recall and F1-scores on unseen users.
• Event segmentation proved particularly beneficial for long sequences, while fixed-size windows performed better for short test sequences, highlighting the importance of temporal context.