[论文解读] An IoT Based Framework For Activity Recognition Using Deep Learning Technique

Ashwin Geet d'Sa, Bhanu Prasad|arXiv (Cornell University)|Jun 17, 2019

Human Pose and Action Recognition参考文献 25被引用 8

一句话总结

本文提出了一种基于物联网的低成本活动识别框架，采用3D卷积神经网络（3D-CNNs）并结合背景减除预处理，以提升人体活动识别的准确性。该系统通过树莓派处理视频流，实现在便携式硬件上的实时活动分类，展示了其在医疗保健、监控和人机交互应用中实际部署的可行性。

ABSTRACT

Activity recognition is the ability to identify and recognize the action or goals of the agent. The agent can be any object or entity that performs action that has end goals. The agents can be a single agent performing the action or group of agents performing the actions or having some interaction. Human activity recognition has gained popularity due to its demands in many practical applications such as entertainment, healthcare, simulations and surveillance systems. Vision based activity recognition is gaining advantage as it does not require any human intervention or physical contact with humans. Moreover, there are set of cameras that are networked with the intention to track and recognize the activities of the agent. Traditional applications that were required to track or recognize human activities made use of wearable devices. However, such applications require physical contact of the person. To overcome such challenges, vision based activity recognition system can be used, which uses a camera to record the video and a processor that performs the task of recognition. The work is implemented in two stages. In the first stage, an approach for the Implementation of Activity recognition is proposed using background subtraction of images, followed by 3D- Convolutional Neural Networks. The impact of using Background subtraction prior to 3D-Convolutional Neural Networks has been reported. In the second stage, the work is further extended and implemented on Raspberry Pi, that can be used to record a stream of video, followed by recognizing the activity that was involved in the video. Thus, a proof-of-concept for activity recognition using small, IoT based device, is provided, which can enhance the system and extend its applications in various forms like, increase in portability, networking, and other capabilities of the device.

研究动机与目标

• 开发一种基于物联网设备的低成本、便携式活动识别系统，用于实时监控。
• 通过将背景减除作为预处理步骤，提升基于3D-CNN的人体活动识别的准确性。
• 验证在资源受限的物联网设备（如树莓派）上部署训练好的3D-CNN模型的可行性。
• 实现在边缘设备上对视频进行实时处理与分类，适用于医疗保健、监控和人机交互应用。
• 证明较低分辨率的视频输入可在降低计算负载的同时保持分类性能。

提出的方法

• 对视频帧应用背景减除，以从静态背景中分离出运动的前景物体，从而增强后续处理的特征提取效果。
• 将预处理后的视频片段（从8秒视频中截取35帧，对应7秒）输入3D卷积神经网络（3D-CNN），以实现时空特征学习与分类。
• 使用高性能系统在KTH数据集上训练3D-CNN模型，以应对训练过程的计算密集性。
• 将训练好的模型权重传输至树莓派进行推理，避免因处理能力有限而在设备上进行训练。
• 使用树莓派摄像头模块以160×120分辨率捕获视频，帧采样确保与训练数据的尺寸保持一致。
• 通过物联网协议（如通过GSM发送短信、通过SMTP发送邮件）传输分类后的输出，实现实时警报或控制操作。

实验结果

研究问题

• RQ1 与直接使用原始视频帧相比，背景减除预处理是否能提升基于3D-CNN的人体活动识别的准确性？
• RQ2 在高性能硬件上训练的3D-CNN模型，是否能有效部署在低功耗物联网设备（如树莓派）上并实现可靠的推理？
• RQ3 在边缘计算的活动识别中，视频分辨率如何影响计算效率与分类准确率之间的权衡？
• RQ4 在具有静态背景的现实室内环境中，便携式物联网系统在多大程度上能维持实时性能与准确性？
• RQ5 在计算资源受限的边缘设备上部署深度学习模型存在哪些实际限制？

主要发现

• 与直接使用原始视频帧相比，背景减除预处理显著提升了3D-CNN在活动识别中的分类准确性。
• 系统在树莓派上成功实现了基于其内置摄像头模块捕获视频的正确活动分类，验证了在边缘设备上实时部署的可行性。
• 较低分辨率的视频输入（160×120）足以实现准确分类，并降低了计算负载，使设备端推理成为可能。
• 树莓派上的模型推理速度慢于训练系统，主要由于其计算能力有限，但对实时应用仍具功能性。
• 将预训练的模型权重从高性能系统迁移到树莓派，实现了无需设备端训练的有效部署。
• 该框架支持与物联网通信协议（如GSM、SMTP）的集成，可在监控和医疗保健应用中实现实时警报与控制操作。