[Paper Review] Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy

Sergei V. Kalinin, Maxim Ziatdinov+7 more|arXiv (Cornell University)|Mar 22, 2021

Force Microscopy Techniques and Applications138 references4 citations

TL;DR

This paper proposes a framework for automated and autonomous experimentation in electron and scanning probe microscopy by integrating machine learning and artificial intelligence to streamline routine operations and enhance decision-making during imaging. It emphasizes the role of domain knowledge, human-ML collaboration, and system limitations to enable real-time, high-order experiment planning with reduced latency and improved efficiency in materials characterization.

ABSTRACT

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with domain applications ranging from theory and materials prediction to high-throughput data analysis. In parallel, the recent successes in applying ML/AI methods for autonomous systems from robotics through self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiment (AE) in imaging. Here, we aim to analyze the major pathways towards AE in imaging methods with sequential image formation mechanisms, focusing on scanning probe microscopy (SPM) and (scanning) transmission electron microscopy ((S)TEM). We argue that automated experiments should necessarily be discussed in a broader context of the general domain knowledge that both informs the experiment and is increased as the result of the experiment. As such, this analysis should explore the human and ML/AI roles prior to and during the experiment, and consider the latencies, biases, and knowledge priors of the decision-making process. Similarly, such discussion should include the limitations of the existing imaging systems, including intrinsic latencies, non-idealities and drifts comprising both correctable and stochastic components. We further pose that the role of the AE in microscopy is not the exclusion of human operators (as is the case for autonomous driving), but rather automation of routine operations such as microscope tuning, etc., prior to the experiment, and conversion of low latency decision making processes on the time scale spanning from image acquisition to human-level high-order experiment planning.

Motivation & Objective

• To develop a framework for automated and autonomous experimentation in electron and scanning probe microscopy using machine learning and AI.
• To address the limitations of current imaging systems, including latency, drift, and non-idealities, in the context of autonomous operation.
• To integrate human expertise and ML/AI decision-making in a synergistic manner, considering knowledge priors, biases, and real-time feedback loops.
• To enable high-order experiment planning on human-level time scales, moving beyond basic automation to intelligent, adaptive experimentation.
• To position autonomous experimentation not as full replacement of human operators, but as a tool to automate routine tasks and accelerate discovery in materials science.

Proposed method

• Leverages machine learning and AI to automate routine microscope tuning and calibration procedures prior to imaging.
• Integrates real-time decision-making systems that process image data sequentially, enabling adaptive control during experiment execution.
• Models the experiment as a feedback loop where domain knowledge and prior data inform both experimental design and real-time adjustments.
• Accounts for system-specific constraints such as intrinsic latencies, drift (both correctable and stochastic), and measurement non-idealities.
• Uses hierarchical decision-making architectures that distinguish between low-latency control actions and higher-level planning tasks.
• Emphasizes the co-evolution of knowledge: experiment informs model, and model guides future experiments, forming a closed-loop learning system.

Experimental results

Research questions

• RQ1 How can machine learning and AI be effectively integrated into scanning probe and transmission electron microscopy to enable autonomous experimentation?
• RQ2 What are the key system-level limitations—such as latency, drift, and non-idealities—that must be addressed for reliable autonomous operation?
• RQ3 How can human expertise and ML/AI decision-making be meaningfully coordinated during the experimental process?
• RQ4 What role does domain knowledge play in guiding autonomous experiment design and real-time adaptation?
• RQ5 To what extent can autonomous systems perform high-order experiment planning comparable to human-level reasoning in microscopy?

Key findings

• Autonomous experimentation in microscopy is feasible when ML/AI systems are integrated with domain knowledge and real-time feedback mechanisms.
• System-level constraints such as drift and latency significantly impact the reliability and performance of autonomous systems and must be explicitly modeled.
• The framework enables a shift from manual tuning to automated, intelligent control of experimental parameters, reducing human intervention.
• Autonomous systems can perform high-order experiment planning on human-level time scales, improving efficiency and discovery speed.
• The co-evolution of knowledge and experiment—where data from imaging refine models and vice versa—creates a closed-loop system that enhances long-term discovery.
• Human operators remain essential not as executors, but as curators and designers of experimental strategies, with AI handling routine and real-time decisions.