[Paper Review] Latent hypernet: Exploring all Layers from Convolutional Neural Networks

Artur Jordão, Ricardo Barbosa Kloss+1 more|arXiv (Cornell University)|Nov 7, 2017

Context-Aware Activity Recognition Systems15 references3 citations

TL;DR

This paper proposes Latent HyperNet (LHN), a method that enhances Convolutional Neural Networks (ConvNets) for human activity recognition (HAR) by projecting feature maps from all convolutional layers—via Partial Least Squares—into a shared latent space before concatenation and classification. The approach improves recognition accuracy by up to 5.1 percentage points over state-of-the-art methods without increasing inference time.

ABSTRACT

Since Convolutional Neural Networks (ConvNets) are able to simultaneously learn features and classifiers to discriminate different categories of activities, recent works have employed ConvNets approaches to perform human activity recognition (HAR) based on wearable sensors, allowing the removal of expensive human work and expert knowledge. However, these approaches have their power of discrimination limited mainly by the large number of parameters that compose the network and the reduced number of samples available for training. Inspired by this, we propose an accurate and robust approach, referred to as Latent HyperNet (LHN). The LHN uses feature maps from early layers (hyper) and projects them, individually, onto a low dimensionality space (latent). Then, these latent features are concatenated and presented to a classifier. To demonstrate the robustness and accuracy of the LHN, we evaluate it using four different networks architectures in five publicly available HAR datasets based on wearable sensors, which vary in the sampling rate and number of activities. Our experiments demonstrate that the proposed LHN is able to produce rich information, improving the results regarding the original ConvNets. Furthermore, the method outperforms existing state-of-the-art methods.

Motivation & Objective

• To address the limitations of standard ConvNets in HAR, such as high parameter count and sensitivity to small training sets.
• To explore whether early-layer features in ConvNets contain discriminative information that can improve classification when combined with deep-layer representations.
• To develop a lightweight, plug-and-play method that enhances ConvNet performance without modifying network architecture or retraining.
• To demonstrate the method’s robustness and generalization across diverse HAR datasets with varying sensor configurations and activity counts.

Proposed method

• The method extracts feature maps from each max-pooling layer in a pre-trained ConvNet.
• Each layer’s feature maps are projected into a low-dimensional latent space using Partial Least Squares (PLS), a supervised dimensionality reduction technique.
• The latent features from all layers are concatenated into a single representation for classification.
• The classifier is trained on the concatenated latent features, while the original ConvNet weights remain frozen.
• The approach is architecture-agnostic and can be applied to any ConvNet without retraining or architectural changes.
• The method is evaluated using five public HAR datasets with varying sampling rates and activity classes.

Experimental results

Research questions

• RQ1 Can early-layer features in ConvNets provide complementary discriminative information for HAR when combined with deep-layer features?
• RQ2 Does projecting multi-layer features into a shared latent space improve recognition accuracy compared to using only the final layer?
• RQ3 Can the proposed method enhance performance without increasing computational cost at inference time?
• RQ4 How does the LHN method compare to state-of-the-art ConvNet-based HAR approaches across diverse datasets?

Key findings

• The LHN method improved recognition accuracy by an average of 5.1 percentage points over existing state-of-the-art methods across five HAR datasets.
• On the USCHAD dataset, LHN outperformed the previous SOTA by 5.1 p.p., achieving a recognition rate of 98.7%.
• On the UTD-MHAD1 dataset, LHN achieved a 5.3 p.p. improvement over the prior best method, despite the high number of activity classes.
• The method maintained statistical equivalence in prediction time compared to the original ConvNet, with no significant increase in inference cost.
• Even without LHN, the proposed ConvNet3 architecture outperformed existing ConvNet architectures in the literature due to larger kernel sizes capturing longer temporal patterns.
• On the MHEALTH dataset, applying LHN to the best-performing baseline model (Ha and Choi, 2017) improved accuracy by 7.3 p.p., surpassing the previous SOTA.