[Paper Review] Semantics-Aware Next-best-view Planning for Efficient Search and Detection of Task-relevant Plant Parts

Akshay K. Burusa, Joost Scholten+4 more|arXiv (Cornell University)|Jun 16, 2023

Smart Agriculture and AI4 citations

TL;DR

This paper proposes a semantics-aware next-best-view (NBV) planning strategy for robotic perception in tomato greenhouses, using semantic class labels and an attention mechanism to prioritize detection of task-relevant plant parts (tomatoes, peduncles, petioles). In simulation, it achieved 85.5% detection accuracy—4 to 11 more parts per plant than baseline strategies—while remaining robust to occlusion, position uncertainty, and plant complexity.

ABSTRACT

Searching and detecting the task-relevant parts of plants is important to automate harvesting and de-leafing of tomato plants using robots. This is challenging due to high levels of occlusion in tomato plants. Active vision is a promising approach in which the robot strategically plans its camera viewpoints to overcome occlusion and improve perception accuracy. However, current active-vision algorithms cannot differentiate between relevant and irrelevant plant parts and spend time on perceiving irrelevant plant parts. This work proposed a semantics-aware active-vision strategy that uses semantic information to identify the relevant plant parts and prioritise them during view planning. The proposed strategy was evaluated on the task of searching and detecting the relevant plant parts using simulation and real-world experiments. In simulation experiments, the semantics-aware strategy proposed could search and detect 81.8% of the relevant plant parts using nine viewpoints. It was significantly faster and detected more plant parts than predefined, random, and volumetric active-vision strategies that do not use semantic information. The strategy proposed was also robust to uncertainty in plant and plant-part positions, plant complexity, and different viewpoint-sampling strategies. In real-world experiments, the semantics-aware strategy could search and detect 82.7% of the relevant plant parts using seven viewpoints, under complex greenhouse conditions with natural variation and occlusion, natural illumination, sensor noise, and uncertainty in camera poses. The results of this work clearly indicate the advantage of using semantics-aware active vision for targeted perception of plant parts and its applicability in the real world. It can significantly improve the efficiency of automated harvesting and de-leafing in tomato crop production.

Motivation & Objective

• Address the challenge of occlusion in robotic perception of task-relevant plant parts during automated harvesting and de-leafing in tomato greenhouses.
• Overcome limitations of conventional active-vision methods that treat all plant parts equally and fail to prioritize relevant objects-of-interest (OOIs).
• Improve detection efficiency and accuracy by integrating semantic information (class labels and confidence scores) into next-best-view planning.
• Develop an online, adaptive attention mechanism that dynamically guides viewpoint selection toward OOIs during perception.
• Demonstrate robustness to uncertainties in plant structure, plant-part positions, and viewpoint sampling strategies in real-world-like conditions.

Proposed method

• Integrate semantic segmentation outputs (class labels and confidence scores) from a neural network into a next-best-view (NBV) planning framework.
• Use an attention mechanism that assigns higher priority to regions containing OOIs (tomatoes, peduncles, petioles) during viewpoint selection.
• Implement a volumetric representation (OctoMap) to fuse multi-view observations and handle uncertainty in 3D plant structure.
• Define a novel semantic NBV planner that selects viewpoints to maximize novel information about OOIs, not just overall scene coverage.
• Use a simulated environment with 3D models of tomato plants of varying complexity to evaluate planner performance under controlled, repeatable conditions.
• Apply a 50% F1-score threshold to determine object detection completeness, with the planner continuing to perceive objects beyond this threshold to ensure completeness.

Figure 1 : Accurate and efficient detection of relevant plant-parts, such as tomatoes, peduncles, and petioles, is essential for automation of harvesting and de-leafing in tomato greenhouses.

Experimental results

Research questions

• RQ1 Can integrating semantic information into active-vision planning improve the detection efficiency and accuracy of task-relevant plant parts in occluded greenhouse environments?
• RQ2 How does a semantics-aware NBV planner compare to conventional volumetric, predefined, random, and non-semantic active-vision strategies in terms of detection performance?
• RQ3 To what extent is the proposed semantic NBV planner robust to uncertainties in plant-part positions, plant structural complexity, and viewpoint sampling constraints?
• RQ4 Does the use of an attention mechanism focused on OOIs lead to faster and more reliable detection of relevant plant parts compared to non-semantic planning?
• RQ5 How does the planner perform under varying thresholds for object detection completeness, and can it be adapted to stop once desired completeness is reached?

Key findings

• The semantic NBV planner detected 85.5% of all plant parts across 96 experiments, significantly outperforming the volumetric NBV planner, which detected 4 fewer parts per plant on average.
• Compared to two predefined viewpoint strategies, the semantic NBV planner detected 5 and 9 more plant parts per plant, respectively.
• The semantic NBV planner detected 11 more plant parts per plant than the random viewpoint strategy, demonstrating its superiority in targeted perception.
• The planner achieved a median detection accuracy of 88.9% per plant across 96 experiments, indicating high reliability under diverse conditions.
• The planner remained robust to uncertainties in plant-part positions, variations in plant complexity, and different viewpoint sampling strategies, confirming its real-world applicability.
• False negatives in object detection had a limited impact due to the multi-view nature of the planner, which allowed detection from multiple angles even if one view failed.

Figure 2 : Schematic diagram of the experimental setup in simulation, with an ABB IRB 1200 robotic arm and Realsense L515 camera. The plant was positioned in front of the robot with uncertainty (shown in pink) to test the robustness of the viewpoint planners. The robot was allowed to choose viewpoin