[Paper Review] ReMotENet: Efficient Relevant Motion Event Detection for Large-scale Home Surveillance Videos

Ruichi Yu, Hongcheng Wang+1 more|arXiv (Cornell University)|Jan 6, 2018

Video Surveillance and Tracking Methods3 references3 citations

TL;DR

ReMotENet proposes a unified, end-to-end 3D ConvNet with spatial-temporal attention and frame differencing to detect relevant motion events (e.g., persons, vehicles) in home surveillance videos. It processes entire 15s video clips in a single forward pass, achieving up to 20,000× speedup over traditional two-stage methods while maintaining or improving accuracy, with inference in 4–8ms on GPU and under 0.4s on CPU using a model under 1MB.

ABSTRACT

This paper addresses the problem of detecting relevant motion caused by objects of interest (e.g., person and vehicles) in large scale home surveillance videos. The traditional method usually consists of two separate steps, i.e., detecting moving objects with background subtraction running on the camera, and filtering out nuisance motion events (e.g., trees, cloud, shadow, rain/snow, flag) with deep learning based object detection and tracking running on cloud. The method is extremely slow and therefore not cost effective, and does not fully leverage the spatial-temporal redundancies with a pre-trained off-the-shelf object detector. To dramatically speedup relevant motion event detection and improve its performance, we propose a novel network for relevant motion event detection, ReMotENet, which is a unified, end-to-end data-driven method using spatial-temporal attention-based 3D ConvNets to jointly model the appearance and motion of objects-of-interest in a video. ReMotENet parses an entire video clip in one forward pass of a neural network to achieve significant speedup. Meanwhile, it exploits the properties of home surveillance videos, e.g., relevant motion is sparse both spatially and temporally, and enhances 3D ConvNets with a spatial-temporal attention model and reference-frame subtraction to encourage the network to focus on the relevant moving objects. Experiments demonstrate that our method can achieve comparable or event better performance than the object detection based method but with three to four orders of magnitude speedup (up to 20k times) on GPU devices. Our network is efficient, compact and light-weight. It can detect relevant motion on a 15s surveillance video clip within 4-8 milliseconds on a GPU and a fraction of second (0.17-0.39) on a CPU with a model size of less than 1MB.

Motivation & Objective

• To address the inefficiency and high latency of traditional two-stage motion detection pipelines in large-scale home surveillance systems.
• To reduce computational cost and improve scalability by replacing sequential background subtraction and cloud-based object detection with a unified, lightweight neural network.
• To exploit spatial-temporal redundancies in surveillance videos by jointly modeling appearance and motion patterns of objects-of-interest.
• To enable real-time, edge-compatible detection with minimal model size (<1MB) and low inference latency on CPU and GPU.
• To develop a weakly supervised training strategy using pseudo-groundtruth from object detection to avoid costly manual annotation.

Proposed method

• ReMotENet uses a 3D ConvNet with spatial-temporal attention (STA) to jointly model appearance and motion in video clips, focusing on relevant moving objects.
• It applies frame differencing with a local reference frame (RefL) to suppress static background and highlight moving foreground, reducing input complexity.
• The network processes entire 15s video clips in a single forward pass, eliminating per-frame processing bottlenecks of traditional methods.
• A pseudo-groundtruth generation pipeline uses object detection (Faster R-CNN at 10 FPS) with confidence threshold >0.8 to create binary masks for training the attention layer.
• The spatial attention mechanism uses 5×3 grid pooling to align pseudo-labels with feature maps, enabling weakly supervised training of the attention module.
• The model is trained end-to-end using weak supervision from detection results, avoiding full annotation costs.

Experimental results

Research questions

• RQ1 Can a unified, end-to-end deep learning model outperform traditional two-stage motion detection pipelines in speed and accuracy for home surveillance?
• RQ2 How effective is frame differencing with a local reference frame (RefL) compared to global reference or raw frames in enhancing motion detection?
• RQ3 To what extent can spatial-temporal attention improve focus on relevant moving objects while reducing computational load?
• RQ4 Can weak supervision using object detection pseudo-groundtruth achieve performance comparable to fully supervised methods?
• RQ5 What is the inference latency and model size of a compact, efficient network suitable for real-time deployment on edge devices?

Key findings

• ReMotENet achieves comparable or better average precision than the two-stage object detection baseline, with 82.3% mAP on person motion and 73.0% on vehicle motion using RefL-C3D input.
• The model reduces inference time to 4–8ms on GPU and 0.17–0.39s on CPU for 15s clips, representing a 3–4 order of magnitude speedup (up to 20,000×) over traditional methods.
• Using local reference frame differencing (RefL) improves mAP by 4.5% over global reference and 5.5% over raw frames, demonstrating its effectiveness in background suppression.
• The network achieves good convergence with only a few hundred training iterations, showing stable and efficient training dynamics.
• The final model is compact, with a size of less than 1MB, making it suitable for deployment on low-resource edge devices.
• Pseudo-groundtruth from object detection significantly improves attention layer performance, validating the weakly supervised training approach.