[Paper Review] Semantic Feature Matching for Robust Mapping in Agriculture

Mohamad Qadri, George Kantor|arXiv (Cornell University)|Jul 9, 2021

Robotics and Sensor-Based Localization25 references4 citations

TL;DR

This paper proposes an object-level SLAM system for agricultural robotics that uses deep learning-based detection and segmentation of sorghum seeds as semantic landmarks, leveraging the structured row-wise navigation of farm environments. By modeling seed positions as graph nodes and exploiting horizontal translation constraints between consecutive frames, the method achieves 78% average mapping coverage—significantly outperforming traditional visual SLAM (e.g., ORB-SLAM2) in low-texture, repetitive agricultural settings.

ABSTRACT

Visual Simultaneous Localization and Mapping (SLAM) systems are an essential component in agricultural robotics that enable autonomous navigation and the construction of accurate 3D maps of agricultural fields. However, lack of texture, varying illumination conditions, and lack of structure in the environment pose a challenge for Visual-SLAM systems that rely on traditional feature extraction and matching algorithms such as ORB or SIFT. This paper proposes 1) an object-level feature association algorithm that enables the creation of 3D reconstructions robustly by taking advantage of the structure in robotic navigation in agricultural fields, and 2) An object-level SLAM system that utilizes recent advances in deep learning-based object detection and segmentation algorithms to detect and segment semantic objects in the environment used as landmarks for SLAM. We test our SLAM system on a stereo image dataset of a sorghum field. We show that our object-based feature association algorithm enables us to map 78% of a sorghum range on average. In contrast, with traditional visual features, we achieve an average mapped distance of 38%. We also compare our system against ORB-SLAM2, a state-of-the-art visual SLAM algorithm.

Motivation & Objective

• Address the failure of traditional visual SLAM systems in low-texture, repetitive agricultural environments due to poor feature detection and matching.
• Leverage the structured row-wise navigation of agricultural robots to improve data association robustness.
• Develop an object-level SLAM system that uses semantic objects (sorghum seeds) as landmarks instead of traditional visual features.
• Enable accurate 3D reconstruction in challenging conditions such as low lighting, wind-induced motion, and low frame rates (5Hz).
• Outperform state-of-the-art visual SLAM systems like ORB-SLAM2 in mapping performance under these constraints.

Proposed method

• Detect and segment individual sorghum seeds in stereo images using deep learning-based instance segmentation models.
• Represent each detected seed as a node in a graph, where node positions are 2D image coordinates.
• Enforce geometric consistency by constraining inter-node distances across frames to remain approximately constant.
• Model inter-frame motion as predominantly horizontal translation, using this to filter spurious matches and improve data association.
• Apply the Hungarian algorithm to solve the data association problem by minimizing assignment cost based on spatial and motion constraints.
• Construct a 3D map using visual odometry from stereo images and semantic landmarks, without relying on inertial sensors.

Experimental results

Research questions

• RQ1 Can semantic object detection and segmentation improve data association robustness in visual SLAM for agricultural environments?
• RQ2 How does exploiting the row-wise navigation structure of agricultural robots enhance SLAM performance in low-texture, repetitive fields?
• RQ3 To what extent does object-level SLAM outperform traditional feature-based SLAM (e.g., ORB-SLAM2) in mapping accuracy under low frame rate and poor texture conditions?
• RQ4 Can geometric constraints on horizontal translation between consecutive frames reduce false positive matches in feature association?
• RQ5 How scalable is the proposed method for large-scale 3D reconstruction in diverse crop types with similar structural layouts?

Key findings

• The proposed system maps 78% of the total distance across eight sorghum field ranges on average, compared to only 38% with traditional feature-based methods.
• In eight test ranges, the system successfully maps 3 out of 8 ranges completely and achieves an average of 65% coverage in the remaining five.
• ORB-SLAM2 fails to map more than 0.35 meters in any range, indicating severe instability due to poor feature matching in low-texture environments.
• The method outperforms SIFT, SURF, ORB, and AKAZE in terms of maximum mapped distance, with SIFT being the next best-performing traditional method.
• The system achieves full mapping of 5.03 meters in one range and 4.43 meters in another, demonstrating robustness even at longer traversals.
• The use of semantic landmarks and geometric constraints significantly reduces false matches and improves data association reliability in challenging visual conditions.