[论文解读] EAA: Automating materials characterization with vision language model agents

Ming Du, Yanqi Luo+4 more|arXiv (Cornell University)|Feb 17, 2026

Advanced Electron Microscopy Techniques and Applications被引用 0

一句话总结

该论文提出 Experiment Automation Agents (EAA)，这是一个由视觉-语言模型驱动的系统，能够通过多模态推理、工具增强的行动以及可选的长期记忆来自动化复杂的光束线实验，并在 Advanced Photon Source 展示。

ABSTRACT

We present Experiment Automation Agents (EAA), a vision-language-model-driven agentic system designed to automate complex experimental microscopy workflows. EAA integrates multimodal reasoning, tool-augmented action, and optional long-term memory to support both autonomous procedures and interactive user-guided measurements. Built on a flexible task-manager architecture, the system enables workflows ranging from fully agent-driven automation to logic-defined routines that embed localized LLM queries. EAA further provides a modern tool ecosystem with two-way compatibility for Model Context Protocol (MCP), allowing instrument-control tools to be consumed or served across applications. We demonstrate EAA at an imaging beamline at the Advanced Photon Source, including automated zone plate focusing, natural language-described feature search, and interactive data acquisition. These results illustrate how vision-capable agents can enhance beamline efficiency, reduce operational burden, and lower the expertise barrier for users.

研究动机与目标

• Motivate the use of AI agents to automate beamline workflows and lower the expertise barrier for users.
• Describe a flexible, modular architecture that integrates vision-language models with instrument control tools.
• Showcase how tool-augmented reasoning and memory enable autonomous and interactive experimentation at a synchrotron beamline.

提出的方法

• Introduce Experiment Automation Agents (EAA) with a three-module architecture: task manager, agent, and tool library.
• Enable both in-process tools and MCP-wrapped external tools for cross-application compatibility.
• Support three workflow modes (logic-driven, hybrid, and agent-driven) with varying levels of LLM involvement.
• Incorporate optional long-term memory via a vector store for retrieval-augmented generation.
• Ensure safe, deterministic tool execution by prioritizing controlled tool calls and process isolation when interacting with instruments.
• Demonstrate two-way MCP compatibility so EAA tools can be used as MCP servers or consumed from external MCP clients.

Figure 1: The main components of EAA and their interactions. The task manager contains the chat loop or workflow, creates and holds the agent object, and maintains the context. New messages coming from the user, auto-generated by the workflow logic, or responded by the agent are added to the context

实验结果

研究问题

• RQ1 How can vision-language models enable autonomous and interactive experimentation at synchrotron beamlines?
• RQ2 What architecture and workflow designs best balance LLM-driven control with explicit analytical routines for robust instrument automation?
• RQ3 How can tools be standardized and shared across applications (via MCP) while maintaining instrument safety and reliability?
• RQ4 Can memory mechanisms (RAG) improve multi-session beamline operations and knowledge retention for automation?
• RQ5 What are practical demonstrations of EAA in tasks like automated focusing, feature search, and interactive data acquisition?

主要发现

• EAA can drive automated focusing of a zone plate by iteratively scanning, adjusting optics, and using image-based feedback to minimize line-scan FWHM.
• A feature-search workflow can locate a language-described feature (e.g., Siemens star) through local fine scans and adaptive step-sizing.
• Interactive data acquisition can be guided by user-provided screenshots, enabling precise local scans and multi-step refinements.
• The system demonstrates robust interaction between vision, prompts, and instrument control, including image-based overlap checks and sub-agents for complex tasks.
• EAA supports both in-process tool calls and MCP-based tool servers, enabling interoperability with other AI and beamline software.

Figure 2: Three levels of LLM involvement in experiment automation tools. Examples are enumerated for each level.