[Paper Review] Recognizing Activities of Daily Living from Egocentric Images

Alejandro Cartas, Juan Marín+2 more|arXiv (Cornell University)|Apr 13, 2017

Context-Aware Activity Recognition Systems8 references4 citations

TL;DR

This paper proposes a novel ensemble method that combines feature maps from multiple layers of pre-trained Convolutional Neural Networks (CNNs) with a Random Forest classifier to improve activity recognition from low-temporal-resolution egocentric images. The approach achieves 86% overall accuracy on a 21-class dataset from the NTCIR-12 benchmark, significantly outperforming baseline CNNs by up to 8.07% through integration of global image features from fully connected layers.

ABSTRACT

Recognizing Activities of Daily Living (ADLs) has a large number of health applications, such as characterize lifestyle for habit improvement, nursing and rehabilitation services. Wearable cameras can daily gather large amounts of image data that provide rich visual information about ADLs than using other wearable sensors. In this paper, we explore the classification of ADLs from images captured by low temporal resolution wearable camera (2fpm) by using a Convolutional Neural Networks (CNN) approach. We show that the classification accuracy of a CNN largely improves when its output is combined, through a random decision forest, with contextual information from a fully connected layer. The proposed method was tested on a subset of the NTCIR-12 egocentric dataset, consisting of 18,674 images and achieved an overall accuracy of 86% activity recognition on 21 classes.

Motivation & Objective

• To improve activity recognition from egocentric images by leveraging contextual visual features beyond raw CNN predictions.
• To generalize activity recognition across multiple users with diverse daily routines, avoiding reliance on time or color-based cues.
• To reduce overfitting and improve performance on under-represented and overlapping activity classes (e.g., 'drinking/eating alone' vs. 'eating together').
• To explore the effectiveness of combining multiple CNN layers (e.g., FC6, pool5/7x7, softmax) as input to a Random Forest for improved classification.
• To develop a method robust to intra-class variability and limited training data in egocentric lifelogs.

Proposed method

• The method uses two pre-trained CNNs—AlexNet and GoogLeNet—as feature extractors from egocentric images.
• It combines the softmax probability output and the final fully connected layer (FC6 or pool5/7x7) into a single 1,045-dimensional feature vector per image.
• A Random Forest classifier with 500 trees is trained on these fused features to predict activity classes.
• The approach avoids reliance on time or color histograms by using internal CNN features as contextual priors.
• The method is evaluated on a subset of the NTCIR-12 dataset with 18,674 images and 21 activity labels.
• Ensemble performance is compared across different CNN layers and fusion strategies, with hyperparameters tuned for optimal accuracy.

Experimental results

Research questions

• RQ1 Can combining multiple CNN layers with a Random Forest improve activity recognition accuracy on egocentric images compared to using only the softmax layer?
• RQ2 Does using internal CNN features (e.g., FC6) as contextual information outperform traditional global image features like color histograms in multi-user settings?
• RQ3 How does the proposed ensemble method perform on overlapping or under-represented activity classes such as 'drinking/eating alone' and 'eating together'?
• RQ4 To what extent does the method reduce overfitting in low-data regimes, such as for the 'walking outdoor' class with few training samples?
• RQ5 Can the method generalize across users with diverse lifestyles without requiring user-specific adaptation?

Key findings

• The proposed ensemble method achieved an overall accuracy of 86% on the 21-class NTCIR-12 egocentric dataset, representing a significant improvement over baseline CNNs.
• The best-performing ensemble, GoogLeNet+RF on pool5/7x7+prob, achieved a 7.93% accuracy gain over its baseline CNN.
• The AlexNet+RF on FC6 ensemble achieved an 8.07% improvement over its baseline, demonstrating strong performance on challenging classes.
• The method significantly improved recall for low-frequency classes such as 'cooking' and 'cleaning and chores'.
• Overlapping classes like 'drinking/eating alone' and 'eating together' showed reduced confusion, indicating better discrimination.
• The only performance drop occurred on 'biking' (from 81.68% to lower), likely due to the random forest’s regularization balancing error across classes.