[论文解读] Multimodal Fish Feeding Intensity Assessment in Aquaculture

Meng Cui, Xubo Liu+6 more|arXiv (Cornell University)|Sep 10, 2023

Water Quality Monitoring TechnologiesEnvironmental Science参考文献 67被引用 3

一句话总结

本文提出U-FFIA，一种用于鱼类摄食强度评估（FFIA）的统一多模态模型，通过引入模态丢弃和知识蒸馏技术，在单一架构中处理音频、视频或音视频输入。该模型在参数量减少54%、浮点运算量（FLOPs）降低77%的情况下，实现了最先进性能，同时通过新构建的大规模音视频数据集AV-FFIA（包含27,000个标注片段）在嘈杂环境中保持了鲁棒性。

ABSTRACT

Fish feeding intensity assessment (FFIA) aims to evaluate fish appetite changes during feeding, which is crucial in industrial aquaculture applications. Existing FFIA methods are limited by their robustness to noise, computational complexity, and the lack of public datasets for developing the models. To address these issues, we first introduce AV-FFIA, a new dataset containing 27,000 labeled audio and video clips that capture different levels of fish feeding intensity. Then, we introduce multi-modal approaches for FFIA by leveraging the models pre-trained on individual modalities and fused with data fusion methods. We perform benchmark studies of these methods on AV-FFIA, and demonstrate the advantages of the multi-modal approach over the single-modality based approach, especially in noisy environments. However, compared to the methods developed for individual modalities, the multimodal approaches may involve higher computational costs due to the need for independent encoders for each modality. To overcome this issue, we further present a novel unified mixed-modality based method for FFIA, termed as U-FFIA. U-FFIA is a single model capable of processing audio, visual, or audio-visual modalities, by leveraging modality dropout during training and knowledge distillation using the models pre-trained with data from single modality. We demonstrate that U-FFIA can achieve performance better than or on par with the state-of-the-art modality-specific FFIA models, with significantly lower computational overhead, enabling robust and efficient FFIA for improved aquaculture management.

研究动机与目标

• 通过利用多模态数据，解决鱼类摄食强度评估（FFIA）中鲁棒性与计算效率的双重挑战。
• 开发一种统一的深度学习模型，能够在无需单独模型专业化的情况下处理音频、视觉或音视频输入。
• 创建并发布一个大规模、公开可访问的音视频数据集（AV-FFIA），以支持未来在多模态FFIA领域的研究。
• 通过有效融合音频与视觉模态，提升模型在噪声环境下的性能。
• 通过参数高效的训练与知识蒸馏，降低设备端FFIA应用的计算开销。

提出的方法

• 作者构建了AV-FFIA，一个大规模音视频数据集，包含27,000个标注的音频与视频片段，用于捕捉不同鱼类摄食强度。
• 采用单模态预训练模型（如音频与视频模型）作为特征提取器，随后应用模态融合技术整合不同模态的表征。
• 提出一种新颖的统一模型U-FFIA，通过在训练过程中引入模态丢弃，使单一架构能够处理仅音频、仅视频或音视频输入。
• 从预训练的单模态模型中应用知识蒸馏，指导统一的U-FFIA模型，提升性能而不增加推理复杂度。
• 通过动态调整模态丢弃概率（P_av, P_a, P_v）进行训练，以平衡各模态的学习并防止过拟合。
• 通过下采样视频帧并使用紧凑的音频表征，提升计算效率，使FLOPs降低高达77%。

Figure 1: Experimental systems for data collection. A hydrophone was underwater and the camera was deployed on a tripod with a height of about two meters to capture the video data.

实验结果

研究问题

• RQ1 统一的深度学习模型能否在性能损失最小的情况下，有效处理多种模态（音频、视频、音视频）的鱼类摄食强度评估？
• RQ2 训练过程中模态丢弃对统一多模态FFIA模型泛化能力与鲁棒性有何影响？
• RQ3 从预训练的单模态模型中应用知识蒸馏，能在多大程度上提升统一FFIA模型的性能？
• RQ4 在设备端实时水产养殖应用中，使用音视频融合时，模型精度与计算效率之间存在何种权衡？
• RQ5 在噪声环境条件下，所提出的U-FFIA模型与模态专用模型相比，在精度与鲁棒性方面表现如何？

主要发现

• U-FFIA模型在参数量减少54%、FLOPs降低77%的情况下，性能达到或优于最先进模态专用模型。
• 与全Transformer基线模型相比，视频处理计算成本降低77%，音频处理降低50%，仅导致3%的性能下降。
• 训练过程中引入模态丢弃显著提升了泛化能力，尤其在单模态推理场景下，优于未使用丢弃的模型。
• 音视频融合模型在音频流与视觉流上的F1分数分别达到0.824和0.857，优于自注意力与交叉注意力基线模型。
• 所提模型在噪声环境中表现出更优的鲁棒性，可在单模态模型失效时维持高性能。
• AV-FFIA数据集包含27,000个标注片段，是首个大规模多模态FFIA数据集，已公开发布，以支持未来研究。

Figure 2: Video frames and mel spectrogram visualizations of four different fish feeding intensity: “ Strong ”, “ Medium ”, “ Weak ” and “ None ”.