[Paper Review] Building Workflows for Interactive Human in the Loop Automated Experiment (hAE) in STEM-EELS

Utkarsh Pratiush, Kevin M. Roccapriore+4 more|arXiv (Cornell University)|Apr 10, 2024

Electron and X-Ray Spectroscopy TechniquesMaterials Science3 citations

TL;DR

This paper introduces a human-in-the-loop automated experiment (hAE) framework for STEM-EELS that uses deep kernel learning (DKL) to intelligently guide data acquisition by focusing on regions of interest, avoiding local minima through adaptive hyperparameter tuning and real-time monitoring. The approach enables efficient discovery of atomic-scale structures with reduced experimental time and improved knowledge acquisition.

ABSTRACT

Exploring the structural, chemical, and physical properties of matter on the nano- and atomic scales has become possible with the recent advances in aberration-corrected electron energy-loss spectroscopy (EELS) in scanning transmission electron microscopy (STEM). However, the current paradigm of STEM-EELS relies on the classical rectangular grid sampling, in which all surface regions are assumed to be of equal a priori interest. This is typically not the case for real-world scenarios, where phenomena of interest are concentrated in a small number of spatial locations. One of foundational problems is the discovery of nanometer- or atomic scale structures having specific signatures in EELS spectra. Here we systematically explore the hyperparameters controlling deep kernel learning (DKL) discovery workflows for STEM-EELS and identify the role of the local structural descriptors and acquisition functions on the experiment progression. In agreement with actual experiment, we observe that for certain parameter combinations the experiment path can be trapped in the local minima. We demonstrate the approaches for monitoring automated experiment in the real and feature space of the system and monitor knowledge acquisition of the DKL model. Based on these, we construct intervention strategies, thus defining human-in the loop automated experiment (hAE). This approach can be further extended to other techniques including 4D STEM and other forms of spectroscopic imaging.

Motivation & Objective

• Address the inefficiency of classical rectangular grid sampling in STEM-EELS, which treats all regions equally despite uneven distribution of phenomena of interest.
• Overcome the challenge of local minima trapping in automated experiment workflows during DKL-based discovery of atomic-scale features.
• Develop real-time monitoring tools in both real and feature space to track knowledge acquisition and model confidence during experiments.
• Design intervention strategies that allow human experts to guide the automated workflow, ensuring robust and adaptive exploration of complex materials.
• Establish a scalable framework for hAE applicable beyond STEM-EELS to other spectroscopic imaging techniques like 4D STEM.

Proposed method

• Employ deep kernel learning (DKL) to model spatial correlations and predict EELS spectral signatures across the sample with uncertainty quantification.
• Use acquisition functions—such as expected improvement and entropy-based criteria—to guide the selection of next sampling points based on model uncertainty and expected information gain.
• Integrate local structural descriptors (e.g., from atomistic simulations or image analysis) as input features to enhance DKL model generalization and localization.
• Implement real-time monitoring of model predictions and uncertainty in both real-space (sample coordinates) and feature space (latent representations of spectra).
• Design interactive intervention protocols allowing human experts to re-weight acquisition functions, reinitialize models, or redirect sampling paths when model behavior becomes suboptimal.
• Optimize hyperparameters of the DKL model (e.g., kernel length scales, noise variance) through systematic ablation to identify robust configurations that avoid premature convergence.

Experimental results

Research questions

• RQ1 How do different acquisition functions influence the path efficiency and convergence behavior of DKL-based automated experiments in STEM-EELS?
• RQ2 What role do local structural descriptors play in improving the accuracy and localization of DKL models during atomic-scale feature discovery?
• RQ3 In what ways can real-time monitoring in real and feature space detect and prevent the model from getting trapped in local minima?
• RQ4 How can human-in-the-loop interventions effectively correct or redirect automated experiment workflows when model performance degrades?
• RQ5 What hyperparameter configurations of DKL yield the most robust and efficient discovery of nanoscale features in experimental STEM-EELS data?

Key findings

• Certain combinations of acquisition functions and hyperparameters led to experiment paths becoming trapped in local minima, highlighting the need for adaptive strategies.
• Monitoring model uncertainty in both real and feature space enabled early detection of suboptimal exploration, allowing timely human intervention.
• The integration of local structural descriptors significantly improved the DKL model's ability to focus on regions with known atomic-scale features, increasing discovery efficiency.
• Human-in-the-loop intervention strategies successfully redirected experiments when model convergence was compromised, demonstrating the value of expert oversight.
• The framework demonstrated reduced experimental time and improved detection of rare or complex atomic structures compared to classical grid sampling.
• The approach is generalizable to other techniques such as 4D STEM and other forms of spectroscopic imaging, enabling broader adoption in materials characterization.