[Paper Review] Autonomous Laboratory Agent via Customized Domain-Specific Language Model and Modular AI Interface

Zhuo Diao, Kouma Matsumoto+3 more|arXiv (Cornell University)|Feb 24, 2026

Machine Learning in Materials Science0 citations

TL;DR

The paper presents a modular, domain-adapted language-model framework that autonomously orchestrates real-time scanning probe microscopy with constraint-aware execution on consumer GPUs, separating routing, knowledge, and command generation.

ABSTRACT

We introduce a system architecture that addresses a fundamental challenge in deploying language-model agents for autonomous control of scientific instrumentation: ensuring reliability in safety-critical environments. The framework integrates probabilistic reasoning by domain-specialized language models with deterministic execution layers that enforce constraints through structured validation and modular orchestration. By separating intent interpretation, experimental planning, and command verification, the architecture translates high-level scientific goals into verifiable experimental actions. We demonstrate this approach in real-time atomic-resolution scanning probe microscopy experiments operated at room temperature, where the system autonomously generates control strategies, invokes corrective modules, and maintains stable operation under experimentally challenging conditions. Quantitative evaluations show that domain-adapted small language models achieve high routing robustness and command accuracy while operating on consumer-grade hardware. Beyond a specific instrument, the framework establishes a general computational principle for deploying language-model agents in safety-critical experimental workflows, providing a pathway toward scalable autonomous laboratories.

Motivation & Objective

• Translate high-level scientific intent into verifiable experimental actions while ensuring safety in a real-time nanoscale instrument setting.
• Demonstrate autonomous control of scanning probe microscopy (SPM) at room temperature using domain-adapted small language models.
• Evaluate reliability, routing robustness, and command accuracy of domain-specific LLMs versus cloud-based models under hardware constraints.
• Show that a modular AI interface with constraint validation enables safe execution of experimental procedures.

Proposed method

• Fine-tune compact domain-specific language models on SPM-related data to enable local control.
• Implement a three-SLM architecture: Knowledge-base SLM, Command SLM, and Router SLM, with a dynamic adapter injection scheme for efficient memory use.
• Route user inputs to the appropriate SLM and transform outputs into instrument-executable commands via a Text Parser.
• Enforce constraint-aware validation to prevent execution of invalid or out-of-spec instructions.
• Use a modular AI interface to orchestrate AI modules and instrument APIs, enabling real-time, safe autonomous operation.

Experimental results

Research questions

• RQ1 Can domain-adapted small language models achieve high routing robustness and command accuracy in SPM experimentation?
• RQ2 Does a modular AI interface with constraint validation enable reliable autonomous control of real-time scientific instruments on consumer hardware?
• RQ3 How does Stage I direct command execution compare to Stage II autonomous planning in handling high-level user intents?
• RQ4 What are the main failure modes of LLM-based control in experimental workflows and how can fine-tuning mitigate them?

Key findings

• Domain-adapted SLMs achieve high routing accuracy across knowledge-base and command tasks.
• Quantized and fine-tuned Phi-4, Mistral-v0.3, and Llama-3.2 models show robust routing and planning performance, with Phi-4 often strongest.
• Fine-tuned SLMs provide near cloud-model performance on knowledge-base tasks and superior command-generation performance in Stage II relative to OpenAI o4-mini.
• 4-bit quantization reduces GPU memory usage ~3x enabling deployment on consumer GPUs while maintaining interactive speeds (>30 tokens/s).
• Stage II autonomous planning demonstrates the system can formulate and execute multi-step experimental plans, such as drift compensation and tip conditioning, to achieve atomic-resolution imaging at room temperature.
• The architecture separates probabilistic reasoning from deterministic execution, enabling safe real-time instrument control.