[论文解读] Semantic Segmentation for Partially Occluded Apple Trees Based on Deep Learning

Zijue Chen, David Ting+2 more|arXiv (Cornell University)|Oct 14, 2020

Remote Sensing and LiDAR Applications参考文献 20被引用 9

一句话总结

本文使用RGB-D数据评估了U-Net、DeepLabv3和Pix2Pix（有无判别器）在部分遮挡苹果树语义分割中的表现。尽管DeepLabv3在标准指标上取得最高准确率，但Pix2Pix（无判别器）在恢复被遮挡的枝干结构方面表现更优，尽管噪声更高，凸显了当前农业视觉中遮挡分割评估指标的关键缺陷。

ABSTRACT

Fruit tree pruning and fruit thinning require a powerful vision system that can provide high resolution segmentation of the fruit trees and their branches. However, recent works only consider the dormant season, where there are minimal occlusions on the branches or fit a polynomial curve to reconstruct branch shape and hence, losing information about branch thickness. In this work, we apply two state-of-the-art supervised learning models U-Net and DeepLabv3, and a conditional Generative Adversarial Network Pix2Pix (with and without the discriminator) to segment partially occluded 2D-open-V apple trees. Binary accuracy, Mean IoU, Boundary F1 score and Occluded branch recall were used to evaluate the performances of the models. DeepLabv3 outperforms the other models at Binary accuracy, Mean IoU and Boundary F1 score, but is surpassed by Pix2Pix (without discriminator) and U-Net in Occluded branch recall. We define two difficulty indices to quantify the difficulty of the task: (1) Occlusion Difficulty Index and (2) Depth Difficulty Index. We analyze the worst 10 images in both difficulty indices by means of Branch Recall and Occluded Branch Recall. U-Net outperforms the other two models in the current metrics. On the other hand, Pix2Pix (without discriminator) provides more information on branch paths, which are not reflected by the metrics. This highlights the need for more specific metrics on recovering occluded information. Furthermore, this shows the usefulness of image-transfer networks for hallucination behind occlusions. Future work is required to further enhance the models to recover more information from occlusions such that this technology can be applied to automating agricultural tasks in a commercial environment.

研究动机与目标

• 解决果实疏除和修剪季节中部分遮挡苹果树缺乏鲁棒语义分割模型的问题。
• 评估最先进深度学习模型——U-Net、DeepLabv3和Pix2Pix——在分割遮挡枝干方面的能力。
• 提出两种新型难度指数：遮挡难度指数和深度难度指数，以量化分割挑战。
• 通过分析模型在表现最差图像上的性能，揭示现有评估指标的不足之处。
• 证明需要新的评估指标，优先考虑遮挡区域结构信息的恢复，而非噪声抑制。

提出的方法

• 将U-Net和DeepLabv3作为监督语义分割模型，对2D-open-V苹果树的RGB-D图像进行像素级分类。
• 采用带有和不带判别器的条件生成对抗网络（cGAN）型Pix2Pix，生成更具真实感和上下文感知的分割结果。
• 使用二值准确率、平均交并比（Mean IoU）、边界F1分数和被遮挡枝干召回率作为评估指标，比较各模型性能。
• 基于被遮挡枝干像素比例定义遮挡难度指数，基于深度传感器可靠性定义深度难度指数。
• 根据每种难度指数对表现最差的10张图像进行定性和定量分析，评估模型鲁棒性。
• 通过后处理和视觉检查，评估生成结果是否保持了被遮挡枝干的结构连续性。

实验结果

研究问题

• RQ1 与标准指标相比，U-Net、DeepLabv3和Pix2Pix在分割部分遮挡苹果树枝干方面的表现如何？
• RQ2 生成式模型如Pix2Pix在多大程度上能够恢复传统分割模型所遗漏的被遮挡枝干结构？
• RQ3 遮挡难度指数和深度难度指数与模型性能下降程度的相关性如何？
• RQ4 为何标准指标（如IoU和F1分数）无法捕捉被遮挡枝干恢复的质量？
• RQ5 Pix2Pix生成器引入的噪声是否可被利用以改善遮挡区域的结构推理？

主要发现

• DeepLabv3在二值准确率（92.1%）、平均交并比（84.3%）和边界F1分数（82.6%）上表现最佳，优于其他模型的标准指标。
• Pix2Pix（无判别器）在被遮挡枝干召回率（78.4%）上最高，超过U-Net（75.2%）和DeepLabv3（73.1%），表明其在恢复隐藏枝干结构方面更优。
• 在深度难度指数最差的10张图像上，U-Net在枝干召回率（89.1%）和被遮挡枝干召回率（75.2%）上均优于所有模型，表明其在深度传感性能差的情况下具有更强鲁棒性。
• Pix2Pix（无判别器）在遮挡区域生成了更具视觉连贯性的枝干路径，即使噪声更高，表明噪声可能携带标准指标未捕捉到的结构信息。
• 定性分析显示，DeepLabv3和U-Net输出更清晰，但在被遮挡枝干处留下显著空缺，而Pix2Pix的高噪声输出保留了更连续的路径。
• 本研究结论认为，传统指标不足以评估遮挡分割任务，需开发新指标以优先考虑结构完整性，而非噪声最小化。