[Paper Review] EAA: Automating materials characterization with vision language model agents

Ming Du, Yanqi Luo+4 more|arXiv (Cornell University)|Feb 17, 2026

Advanced Electron Microscopy Techniques and Applications0 citations

TL;DR

The paper presents Experiment Automation Agents (EAA), a vision-language-model–driven system that automates complex beamline experiments with multimodal reasoning, tool-augmented actions, and optional long-term memory, demonstrated at the Advanced Photon Source.

ABSTRACT

We present Experiment Automation Agents (EAA), a vision-language-model-driven agentic system designed to automate complex experimental microscopy workflows. EAA integrates multimodal reasoning, tool-augmented action, and optional long-term memory to support both autonomous procedures and interactive user-guided measurements. Built on a flexible task-manager architecture, the system enables workflows ranging from fully agent-driven automation to logic-defined routines that embed localized LLM queries. EAA further provides a modern tool ecosystem with two-way compatibility for Model Context Protocol (MCP), allowing instrument-control tools to be consumed or served across applications. We demonstrate EAA at an imaging beamline at the Advanced Photon Source, including automated zone plate focusing, natural language-described feature search, and interactive data acquisition. These results illustrate how vision-capable agents can enhance beamline efficiency, reduce operational burden, and lower the expertise barrier for users.

Motivation & Objective

• Motivate the use of AI agents to automate beamline workflows and lower the expertise barrier for users.
• Describe a flexible, modular architecture that integrates vision-language models with instrument control tools.
• Showcase how tool-augmented reasoning and memory enable autonomous and interactive experimentation at a synchrotron beamline.

Proposed method

• Introduce Experiment Automation Agents (EAA) with a three-module architecture: task manager, agent, and tool library.
• Enable both in-process tools and MCP-wrapped external tools for cross-application compatibility.
• Support three workflow modes (logic-driven, hybrid, and agent-driven) with varying levels of LLM involvement.
• Incorporate optional long-term memory via a vector store for retrieval-augmented generation.
• Ensure safe, deterministic tool execution by prioritizing controlled tool calls and process isolation when interacting with instruments.
• Demonstrate two-way MCP compatibility so EAA tools can be used as MCP servers or consumed from external MCP clients.

Figure 1: The main components of EAA and their interactions. The task manager contains the chat loop or workflow, creates and holds the agent object, and maintains the context. New messages coming from the user, auto-generated by the workflow logic, or responded by the agent are added to the context

Experimental results

Research questions

• RQ1 How can vision-language models enable autonomous and interactive experimentation at synchrotron beamlines?
• RQ2 What architecture and workflow designs best balance LLM-driven control with explicit analytical routines for robust instrument automation?
• RQ3 How can tools be standardized and shared across applications (via MCP) while maintaining instrument safety and reliability?
• RQ4 Can memory mechanisms (RAG) improve multi-session beamline operations and knowledge retention for automation?
• RQ5 What are practical demonstrations of EAA in tasks like automated focusing, feature search, and interactive data acquisition?

Key findings

• EAA can drive automated focusing of a zone plate by iteratively scanning, adjusting optics, and using image-based feedback to minimize line-scan FWHM.
• A feature-search workflow can locate a language-described feature (e.g., Siemens star) through local fine scans and adaptive step-sizing.
• Interactive data acquisition can be guided by user-provided screenshots, enabling precise local scans and multi-step refinements.
• The system demonstrates robust interaction between vision, prompts, and instrument control, including image-based overlap checks and sub-agents for complex tasks.
• EAA supports both in-process tool calls and MCP-based tool servers, enabling interoperability with other AI and beamline software.

Figure 2: Three levels of LLM involvement in experiment automation tools. Examples are enumerated for each level.