[Paper Review] Action2Activity: Recognizing Complex Activities from Sensor Data

Ye Liu, Liqiang Nie+3 more|arXiv (Cornell University)|Nov 7, 2016

Context-Aware Activity Recognition Systems25 references295 citations

TL;DR

The paper introduces temporal pattern mining to represent complex activities and an adaptive multi-task learning framework to recognize them from sensor data, demonstrated on a real-world dataset.

ABSTRACT

As compared to simple actions, activities are much more complex, but semantically consistent with a human's real life. Techniques for action recognition from sensor generated data are mature. However, there has been relatively little work on bridging the gap between actions and activities. To this end, this paper presents a novel approach for complex activity recognition comprising of two components. The first component is temporal pattern mining, which provides a mid-level feature representation for activities, encodes temporal relatedness among actions, and captures the intrinsic properties of activities. The second component is adaptive Multi-Task Learning, which captures relatedness among activities and selects discriminant features. Extensive experiments on a real-world dataset demonstrate the effectiveness of our work.

Motivation & Objective

• Motivate the need to bridge action recognition and high-level activities in real-life sensor data.
• Develop a temporal pattern mining approach that captures sequential, interleaved, and concurrent action relations to describe activities.
• Propose an adaptive multi-task learning model to capture relatedness among activities and select discriminant patterns.
• Evaluate the proposed methods on a real-world dataset to demonstrate effectiveness over baselines.

Proposed method

• Mine frequent temporal patterns from action sequences using Allen's temporal relations to create a pattern-based feature space for activities.
• Represent each activity by the pattern supports, forming a high-dimensional feature vector.
• Formulate activity recognition as multi-task learning across M activities with aW as weight matrix and learn task relatedness via a learned Omega matrix.
• Use an L2,1 (group Lasso) penalty to encourage shared and task-specific features while selecting discriminant patterns.
• Solve the optimization via alternating optimization: W with fixed Omega using FISTA, and Omega with fixed W using a closed-form solution Omega = (W^T W)^{1/2} / tr((W^T W)^{1/2}).
• Evaluate pattern dimensions up to 3 due to convergence and cost considerations.

Experimental results

Research questions

• RQ1 Can temporal patterns among actions capture the intrinsic semantics of complex activities?
• RQ2 Does adaptive multi-task learning improve activity recognition by leveraging relatedness among activities and selecting discriminant features?
• RQ3 What is the impact of pattern dimensionality and regularization parameters on recognition accuracy?

Key findings

• Temporal patterns (especially higher-order patterns like {1,2} and {1,2,3}) improve recognition over bag-of-actions representations.
• aMTL consistently outperforms single-task baselines and traditional MTL, with 2–4% higher accuracy than MTL and 3–6% over GL.
• On the Opportunity dataset, the {1,2}-patterns + aMTL approach achieves 98.0% accuracy, and {1,2,3}-patterns + aMTL achieves 99.2% accuracy.
• The learned task-relations matrix Omega reveals meaningful relatedness among activities (e.g., coffee time correlates with sandwich time).
• Compared to HMM, CRF, and ITBN, the proposed approach with temporal patterns and aMTL yields higher accuracy on the same data.