[Paper Review] AES: Autonomous Excavator System for Real-World and Hazardous Environments

Jinxin Zhao, Pinxin Long+6 more|arXiv (Cornell University)|Nov 10, 2020

Industrial Automation and Control Systems18 references4 citations

TL;DR

This paper presents AES, an autonomous excavator system that integrates perception, learning-based motion planning, and optimization to enable long-duration, human-equivalent excavation in hazardous real-world environments. The system achieves 24 HPI (Hours Per Intervention), operating continuously for 24 hours without human intervention, while matching the productivity of experienced human operators in diverse outdoor and indoor scenarios including mining, construction, and waste handling.

ABSTRACT

Excavators are widely used for material-handling applications in unstructured environments, including mining and construction. The size of the global market of excavators is 44.12 Billion USD in 2018 and is predicted to grow to 63.14 Billion USD by 2026. Operating excavators in a real-world environment can be challenging due to extreme conditions and rock sliding, ground collapse, or exceeding dust. Multiple fatalities and injuries occur each year during excavations. An autonomous excavator that can substitute human operators in these hazardous environments would substantially lower the number of injuries and can improve the overall productivity.

Motivation & Objective

• To develop an autonomous excavator system capable of operating safely and efficiently in real-world, hazardous environments such as mines and construction sites.
• To address the high risk of injury and death in manual excavation by replacing human operators in extreme conditions like dust, temperature extremes, and unstable terrain.
• To achieve robust, long-duration autonomy (24 HPI) with performance equivalent to experienced human operators in material handling and soil excavation tasks.
• To enable the system to perceive dynamic material piles, detect impurities and obstacles, and adapt to changing terrain and material properties in real time.

Proposed method

• Uses a hybrid perception architecture combining RTK GNSS, inclinometers, LiDAR, and RGB-D cameras to generate 2.5D height maps and detect material texture and composition.
• Employs a deep neural network (MLP-based) to predict the point of attack (POA) and bucket travel length for excavation based on real-time environmental observations.
• Applies inverse reinforcement learning (IRL) to extract human motion patterns from demonstrated trajectories, learning feature weights that represent optimal motion cost functions.
• Integrates the learned cost function into a stochastic trajectory optimization (STOMP) framework to generate collision-free, feasible joint-space trajectories.
• Incorporates obstacle and impurity detection data from perception into the STOMP cost function to avoid digging into hard or unsafe materials.
• Deploys the system on both compact (6.5-ton) and standard (49-ton) hydraulic excavators with drive-by-wire control via CAN bus, using an industrial microcontroller for low-level actuation.

Experimental results

Research questions

• RQ1 Can an autonomous excavator system achieve sustained, human-equivalent performance in real-world, hazardous excavation environments?
• RQ2 How can perception and planning be tightly coupled to handle dynamic, uncertain, and unstructured excavation scenarios?
• RQ3 To what extent can learning-based motion planning improve robustness and efficiency compared to purely optimization-based or rule-based systems?
• RQ4 Can a unified system architecture support both compact and standard excavators across diverse material types and environmental conditions?
• RQ5 What level of autonomy (measured in HPI) can be achieved with integrated perception, planning, and control in real-world excavation tasks?

Key findings

• AES achieves 24 HPI, meaning it can operate continuously for 24 hours without human intervention, demonstrating high system reliability and autonomy.
• The system matches the material-handling throughput of an experienced human operator, confirming human-equivalent productivity in real-world excavation tasks.
• The perception module successfully detects material types, textures, and impurities (e.g., chemical waste) under extreme conditions such as dust and low light.
• The learning-based motion planning with IRL and STOMP enables robust trajectory generation that adapts to changing pile shapes and obstacle locations in real time.
• The system operates successfully on both compact and standard excavators, proving scalability and generalizability across different excavator sizes.
• The integration of perception, planning, and control into a single architecture significantly improves performance over prior autonomous excavator systems in complex, unstructured environments.