[论文解读] OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments

Naoya Yoshimura, Jaime Gómez‐Morales+1 more|arXiv (Cornell University)|Dec 10, 2022

Industrial Vision Systems and Defect Detection被引用 11

一句话总结

本文介绍了OpenPack，这是一个大规模、多模态的数据集，用于在物联网（IoT）赋能的物流环境中识别包装作业，结合了16名受试者的惯性、深度、激光雷达（LiDAR）以及物联网设备读数。所提出的LTS-Net模型采用阶梯状双流融合架构，将物联网设备数据（主干流）与传感器模态数据相结合，实现了最先进性能，宏平均F1得分为0.830，其中仅使用物联网数据便使准确率最高提升了0.045 F1分。

ABSTRACT

OpenPack is an open-access logistics dataset for human activity recognition, which contains human movement and package information from 16 subjects in four scenarios. Human movement information is subdivided into three types of data, acceleration, physiological, and depth-sensing. The package information includes the size and number of items included in each packaging job. In the "Humanware laboratory" at IST Osaka University, with the supervision of industrial engineers, an experiment to mimic logistic center labor was designed. 12 workers with previous packaging experience and 4 without experience performed a set of packaging tasks according to an instruction manual from a real-life logistics center. During the different scenarios, subjects were recorded while performing packing operations using Lidar, Kinect, and Realsense depth sensors while wearing 4 ATR IMU devices and 2 Empatica E4 wearable sensors. Besides sensor data, this dataset contains timestamp information collected from the hand terminal used to register product, packet, and address label codes as well as package details that can be useful to relate operations to specific packages. The 4 different scenarios include; sequential packing, worker-decided sequence changes, pre-ordered item packing, and time-sensitive stressors. Each of the subjects performed 20 packing jobs in 5 work sessions for a total of 100 packing jobs. 53+ hours of packaging operations have been labeled into 10 global operation classes and 16 sub-action classes for this dataset. Action classes are not unique to each operation but may only appear in one or two operations. You can find information on how to use this dataset at: https://open-pack.github.io/. For details on how this dataset was collected please check the following publication "OpenPack: A Large-Scale Dataset for Recognizing Packaging Works in IoT-Enabled Logistic Environments" 10.1109/PerCom59722.2024.10494448. Full Dataset In this repository, the data and label files are contained in separate files for each worker. Each worker's file contains; IMU, E4, 2d keypoint, 3d keypoint, annotation, and system-related data. Preprocessed Dataset (IMU with operation and action Labels) We have received many comments that it was difficult to combine multiple workers' IMU and annotation data. Therefore, we have created several CSV files containing the four IMU's sensor data and the operation labels in a single file. These files are now included as "imu-with-operation-action-labels.zip". Preprocessed Dataset (Kinect 2D and 3D keypoint data with operation and action Labels) We have received several requests for a preprocessed dataset containing only specific types of keypoint data with its assigned operation and action labels. Two new preprocessed files have been added for 2D and 3D keypoint data extracted from the frontal view Kinect camera. These files are: "kinect-2d-kpt-with-operation-action-labels.zip", and "kinect-3d-kpt-with-operation-action-labels.zip". Work is continuously being done to update and improve this dataset. When downloading and using this dataset please verify that the version is up to date with the latest release. The latest release [1.1.0] was uploaded on 24/04/2024. Changes LOG: v1.0.0: Add tutorial preprocessed dataset for IMU data with operation labels. v1.1.0: Update preprocessed datasets. (Include Kinect 2d and 3d keypoint data with Operation and action labels) We hosted an activity recognition competition using this dataset (OpenPack v0.3.x) awarded at a PerCom 2023 Workshop! The task was very simple: Recognize 10 work operations from the OpenPack dataset. You can refer to this website for coding materials relevant to this dataset. https://open-pack.github.io/challenge2022

研究动机与目标

• 为解决物流与制造领域工业作业活动识别中缺乏大规模、真实且多模态数据集的问题。
• 克服现有数据集的局限性，包括模态覆盖稀疏、数据量不足以及缺乏物联网设备读数。
• 通过提供丰富的元数据（如工人经验与身体特征），推动跨模态与多模态活动识别的研究。
• 通过提供细粒度的真实世界传感器与设备数据，促进基于人工智能的工作流程优化工具的开发。

提出的方法

• OpenPack数据集从16名受试者收集了53.8小时的多模态数据，包括惯性测量单元（IMU）、深度图像、激光雷达点云以及实时物联网设备读数（如条形码扫描器）。
• 提出一种新颖的双流神经网络架构LTS-Net，通过独立流处理传感器数据与物联网设备读数，随后在阶梯状结构中进行融合。
• 主干流处理高置信度的物联网设备读数（如扫描事件），以引导识别过程，提升对噪声传感器输入的鲁棒性。
• 模型采用早期融合与晚期融合策略，双流方法在所有受试者中的F1得分相比早期融合提升了约0.02。
• 该架构采用Conformer与WPCP主干网络处理时间序列，实现对复杂包装作业的有效时间建模。
• 消融研究评估了物联网数据与融合策略的影响，以F1得分（宏平均与加权平均）为主要评估指标。

实验结果

研究问题

• RQ1 在工业环境中，整合物联网设备读数（如条形码扫描）如何影响包装作业活动识别的准确性？
• RQ2 不同传感器模态——惯性测量单元（IMU）、深度图像、激光雷达（LiDAR）和物联网设备——对识别性能的相对贡献是什么？
• RQ3 双流阶梯状融合架构是否优于多模态工业活动识别中的早期融合？
• RQ4 工人经验水平如何影响识别性能？模型能否在不同技能水平间实现泛化？
• RQ5 丰富的元数据（如身体特征、经验）在多大程度上提升了识别结果的可解释性与实用性？

主要发现

• 引入物联网设备读数（主干流）使宏平均F1得分最高提升0.045分，证明其在提升识别准确率方面具有关键作用。
• LTS-Net中的双流融合策略相比早期融合使F1得分提升约0.02分，表明其在特征交互方面更具优势。
• 该模型在所有受试者中实现了0.830的宏平均F1得分，各受试者得分范围为0.752（U0110）至0.847（U0104）。
• 对于初学者水平的受试者U0104，识别性能在不同方法间保持相对稳定，表明其动作较慢且更具可预测性，因而更容易识别。
• 消融研究证实，物联网数据是影响最大的模态，当其被排除时性能显著下降。
• 该数据集包含20,129个活动实例与52,529个动作实例，覆盖9种模态，包含丰富的元数据，如工人经验与身体特征。