[Paper Review] Extreme Low Resolution Activity Recognition with Multi-Siamese Embedding Learning

Michael S. Ryoo, Kiyoon Kim+1 more|arXiv (Cornell University)|Aug 3, 2017

Anomaly Detection Techniques and Applications22 citations

TL;DR

This paper proposes a two-stream multi-Siamese CNN for extreme low-resolution (16x12) human activity recognition, explicitly modeling transform-robust video embeddings to unify representations of the same high-resolution video under different low-resolution transformations. The method achieves state-of-the-art performance, outperforming prior approaches by over 8% on the 16x12 HMDB dataset with 37.70% accuracy using two-stream features.

ABSTRACT

This paper presents an approach for recognizing human activities from extreme low resolution (e.g., 16x12) videos. Extreme low resolution recognition is not only necessary for analyzing actions at a distance but also is crucial for enabling privacy-preserving recognition of human activities. We design a new two-stream multi-Siamese convolutional neural network. The idea is to explicitly capture the inherent property of low resolution (LR) videos that two images originated from the exact same scene often have totally different pixel values depending on their LR transformations. Our approach learns the shared embedding space that maps LR videos with the same content to the same location regardless of their transformations. We experimentally confirm that our approach of jointly learning such transform robust LR video representation and the classifier outperforms the previous state-of-the-art low resolution recognition approaches on two public standard datasets by a meaningful margin.

Motivation & Objective

• To address the challenge of recognizing human activities in extreme low-resolution videos (e.g., 16x12), where pixel values vary drastically under sub-pixel transformations.
• To enable privacy-preserving activity recognition by using anonymized low-resolution video data that prevents face identification.
• To overcome the limitations of prior methods that treat transformed low-res videos as independent samples, leading to poor generalization.
• To jointly learn a robust video representation and classifier that is invariant to low-resolution transformations such as translation, rotation, and scaling.
• To demonstrate superior performance on standard benchmarks like HMDB and DogCentric under extreme low-resolution settings.

Proposed method

• Designs a two-stream multi-Siamese convolutional neural network to learn shared embedding space for multiple low-resolution video clips derived from the same high-resolution source.
• Uses Siamese architecture to enforce that feature embeddings of the same content under different low-resolution transformations remain close in the embedding space.
• Applies data augmentation via different low-resolution transforms (e.g., sub-pixel translations) during training to simulate real-world LR variations.
• Jointly optimizes the video representation and classifier using contrastive loss to encourage identical embeddings for same-content LR clips.
• Employs optical flow features in the two-stream setup, using the Farneback algorithm for efficient real-time inference on edge devices.
• Trains on high-resolution video datasets (e.g., YouTube) and fine-tunes on low-resolution versions for downstream recognition.

Experimental results

Research questions

• RQ1 Can a deep learning model learn a shared, transform-robust embedding space for low-resolution videos derived from the same high-resolution source?
• RQ2 Does explicitly modeling the invariance to sub-pixel transformations improve recognition accuracy in extreme low-resolution settings?
• RQ3 How does the proposed multi-Siamese embedding learning compare to standard data augmentation and baseline CNNs in low-resolution activity recognition?
• RQ4 Can the method generalize across diverse datasets such as HMDB and DogCentric under extreme low resolution (16x12)?
• RQ5 What is the real-time inference performance of the model on edge hardware like NVIDIA Jetson TX2?

Key findings

• The proposed method achieves 37.70% accuracy on the 16x12 HMDB dataset, outperforming the previous state-of-the-art by more than 8 percentage points.
• On the DogCentric dataset, the method achieves 69.43% accuracy, significantly surpassing prior works such as ISR (67.36%) and PoT (64.6%).
• The method improves binary event detection F1-score to 0.885 on the Hardcore Henry dataset, outperforming baseline (0.838) and data augmentation (0.871).
• The model runs in real-time at approximately 50 fps on an NVIDIA Jetson TX2 GPU, demonstrating practical deployability.
• The multi-Siamese approach with shared embedding learning reduces overfitting to specific low-resolution transformations compared to standard data augmentation.
• The two-stream architecture with optical flow and spatial pyramid pooling further enhances performance, especially in scenarios with small human appearances.