[Paper Review] Neuromorphic Vision Based Control for the Precise Positioning of Robotic Drilling Systems

Abdulla Ayyad, Mohamad Halwani+4 more|arXiv (Cornell University)|Jan 5, 2022

Advanced Surface Polishing Techniques4 citations

TL;DR

This paper presents the first neuromorphic vision-based control system for robotic drilling, using event-based cameras to achieve sub-millimeter positioning accuracy in unstructured, low-light environments. The method combines multi-view 3D reconstruction and event-based circular hole detection with hybrid position-based and image-based visual servoing, achieving an average positional error of less than 0.1 mm across multiple workpieces under varying lighting and motion conditions.

ABSTRACT

The manufacturing industry is currently witnessing a paradigm shift with the unprecedented adoption of industrial robots, and machine vision is a key perception technology that enables these robots to perform precise operations in unstructured environments. However, the sensitivity of conventional vision sensors to lighting conditions and high-speed motion sets a limitation on the reliability and work-rate of production lines. Neuromorphic vision is a recent technology with the potential to address the challenges of conventional vision with its high temporal resolution, low latency, and wide dynamic range. In this paper and for the first time, we propose a novel neuromorphic vision based controller for faster and more reliable machining operations, and present a complete robotic system capable of performing drilling tasks with sub-millimeter accuracy. Our proposed system localizes the target workpiece in 3D using two perception stages that we developed specifically for the asynchronous output of neuromorphic cameras. The first stage performs multi-view reconstruction for an initial estimate of the workpiece's pose, and the second stage refines this estimate for a local region of the workpiece using circular hole detection. The robot then precisely positions the drilling end-effector and drills the target holes on the workpiece using a combined position-based and image-based visual servoing approach. The proposed solution is validated experimentally for drilling nutplate holes on workpieces placed arbitrarily in an unstructured environment with uncontrolled lighting. Experimental results prove the effectiveness of our solution with an average positional errors of less than 0.1 mm, and demonstrate that the use of neuromorphic vision overcomes the lighting and speed limitations of conventional cameras.

Motivation & Objective

• To address the limitations of conventional frame-based cameras in robotic machining, particularly sensitivity to lighting changes and motion blur.
• To enable precise, real-time robotic drilling in unstructured industrial environments with uncontrolled lighting and dynamic conditions.
• To develop a complete neuromorphic vision pipeline for 3D workpiece localization and end-effector positioning using asynchronous event data.
• To validate the system’s robustness and accuracy in real-world drilling tasks, especially for critical aerospace and automotive applications.
• To demonstrate the feasibility of neuromorphic vision as a reliable perception technology for high-precision automated manufacturing.

Proposed method

• The system uses two-stage perception: first, multi-view reconstruction from event-based camera data to estimate initial 3D pose of the workpiece.
• Second, a novel event-based circular hole detection algorithm refines the pose estimate in a local region using contour fitting and model-based detection.
• A hybrid visual servoing strategy combines position-based visual servoing (PBVS) for coarse alignment and image-based visual servoing (IBVS) for fine error correction.
• The neuromorphic camera (iniVation) captures asynchronous events, enabling high temporal resolution and low latency, crucial for fast and accurate control.
• The robotic system integrates a collaborative robot arm, a custom-designed drilling end-effector, and real-time processing of event data for closed-loop control.
• The peg-in-hole mechanism provides passive compliance, further reducing residual positioning errors during final alignment.

Experimental results

Research questions

• RQ1 Can neuromorphic vision overcome the limitations of conventional frame-based cameras in robotic drilling under uncontrolled lighting and high-speed motion?
• RQ2 Can a two-stage perception pipeline effectively localize a workpiece in 3D using asynchronous event data from neuromorphic cameras?
• RQ3 Can hybrid PBVS and IBVS control achieve sub-millimeter accuracy in robotic drilling using event-based visual feedback?
• RQ4 How does the system perform in terms of positional error when drilling nutplate holes in real-world, unstructured environments?
• RQ5 To what extent does the compliance of the collaborative robot and peg-in-hole mechanism further reduce positioning errors?

Key findings

• The proposed neuromorphic vision-based system achieved an average positional error of less than 0.1 mm across all tested nutplate holes in five different workpieces.
• The maximum positional error across all experiments was 0.183 mm, well within the precision requirements of aerospace and automotive manufacturing.
• The system demonstrated robust performance under uncontrolled lighting and high-speed motion, overcoming key limitations of conventional frame-based cameras.
• The event-based circular hole detection algorithm successfully localized reference holes with high accuracy despite low spatial resolution and asynchronous data.
• The combination of multi-view reconstruction and local refinement significantly improved pose estimation accuracy compared to single-stage methods.
• The passive compliance of the collaborative robot during the peg-in-hole phase further reduced residual errors, enhancing overall precision.