[Paper Review] Bimanual crop manipulation for human-inspired robotic harvesting

Sotiris Stavridis, Dimitrios Papageorgiou+2 more|arXiv (Cornell University)|Sep 13, 2022

Smart Agriculture and AI26 citations

TL;DR

The paper presents a bimanual robotic harvesting approach where one arm unveils the stem with a camera and cutter while the other arm manipulates the grasped crop to create cutting affordances, demonstrated on a two-armed UR5e setup with a RealSense camera.

ABSTRACT

Most existing robotic harvesters utilize a unimanual approach; a single arm grasps the crop and detaches it, either via a detachment movement, or by cutting its stem with a specially designed gripper/cutter end-effector. However, such unimanual solutions cannot be applied for sensitive crops and cluttered environments like grapes and a vineyard where obstacles may occlude the stem and leave no space for the cutter's placement. In such cases, the solution would require a bimanual robot in order to visually unveil the stem and manipulate the grasped crop to create cutting affordances which is similar to the practice used by humans. In this work, a dual-arm coordinated motion control methodology for reaching a stem pre-cut state is proposed. The camera equipped arm with the cutter is reaching the stem, unveiling it as much as possible, while the second arm is moving the grasped crop towards the surrounding free-space to facilitate its stem cutting. Lab experimentation on a mock-up vine setup with a plastic grape cluster evaluates the proposed methodology, involving two UR5e robotic arms and a RealSense D415 camera.

Motivation & Objective

• Motivate robotic harvesting in sensitive, cluttered environments where unimanual systems fail.
• Propose a dual-arm coordination framework to reach a stem pre-cut state by un?elding the stem and repositioning the crop.
• Develop velocity-controlled bimanual control laws that coordinate camera and grasping arms in a shared scene.
• Leverage point-cloud processing to identify stem, obstacles, and free space for planning and control.

Proposed method

• Model the problem as a bimanual asymmetric task with a camera arm (with cutter) and a grasping arm (with a crop).
• Define camera-arm objectives: reaching and centering within a region of interest around the stem and unveiling the stem to maximize visibility.
• Define grasping-arm objectives: stretch the grape to maximize free room around the stem by moving the grasped crop toward a point in free space.
• Formulate a velocity-controlled control law using a combined extended Jacobian to map end-effector velocities to joint motions.
• Process scene point-clouds to identify stem, obstacles, and free space, and compute critical positions (e.g., stem base, ROI center) for control.
• Provide a lightweight decomposition of the grasping-arm motion into a direction along stem basis and its orthogonal subspace to implement force-position control.

Experimental results

Research questions

• RQ1 How can a dual-arm system coordinate to unveil the stem and create cutting affordances for grape harvesting?
• RQ2 What control strategy enables simultaneous reaching, centering, and unveiling of the stem while manipulating a grasped crop?
• RQ3 How can scene point-cloud processing support robust bimanual coordination in the presence of clutter and occlusions?
• RQ4 What is an effective decomposition of the grasping-end-effector motion to maximize stem-space without damaging the crop?
• RQ5 Can this bimanual approach outperform unimanual harvesting strategies in vine-like environments?

Key findings

• A dual-arm coordinated approach can position a camera-equipped end-effector to unveil the stem while another arm repositions the grasped crop to create cutting affordances.
• A velocity-controlled framework using an extended Jacobian enables mapping to joint commands for both arms in a bimanual setup.
• Point-cloud processing is used to identify stem, obstacles, and free space to compute ROI, stem base, and relevant trajectories.
• The method was evaluated on a lab mock-up with two UR5e arms and a RealSense D415, demonstrating the viability of bimanual stem unveiling and cutting preparation.