[Paper Review] PEAR: A Robust and Flexible Automation Framework for Ptychography Enabled by Multiple Large Language Model Agents

Xiangyu Yin, Chuqiao Shi+2 more|arXiv (Cornell University)|Oct 11, 2024

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

PEAR is a multi-agent LLM framework that automates ptychographic data analysis by orchestrating specialized large language model agents for knowledge retrieval, code generation, parameter recommendation, and image quality assessment. It achieves high success rates in reconstruction workflows—even with smaller open-weight models like LLaMA 3.1 8B—by enabling flexible, adaptive automation across diverse experimental settings.

ABSTRACT

Ptychography is an advanced computational imaging technique in X-ray and electron microscopy. It has been widely adopted across scientific research fields, including physics, chemistry, biology, and materials science, as well as in industrial applications such as semiconductor characterization. In practice, obtaining high-quality ptychographic images requires simultaneous optimization of numerous experimental and algorithmic parameters. Traditionally, parameter selection often relies on trial and error, leading to low-throughput workflows and potential human bias. In this work, we develop the "Ptychographic Experiment and Analysis Robot" (PEAR), a framework that leverages large language models (LLMs) to automate data analysis in ptychography. To ensure high robustness and accuracy, PEAR employs multiple LLM agents for tasks including knowledge retrieval, code generation, parameter recommendation, and image reasoning. Our study demonstrates that PEAR's multi-agent design significantly improves the workflow success rate, even with smaller open-weight models such as LLaMA 3.1 8B. PEAR also supports various automation levels and is designed to work with customized local knowledge bases, ensuring flexibility and adaptability across different research environments.

Motivation & Objective

• To address the high-throughput, low-bias challenge in ptychographic reconstruction by replacing manual, trial-and-error parameter tuning with automated, intelligent optimization.
• To develop a flexible and extensible framework that supports various automation levels and integrates with local knowledge bases for domain-specific adaptation.
• To demonstrate that multi-agent LLM systems can achieve robust and accurate parameter recommendation and reconstruction automation in ptychography, even when using smaller open-weight LLMs.
• To reduce reliance on computationally expensive black-box optimization methods like Bayesian optimization by leveraging LLMs’ reasoning and knowledge capabilities.
• To enable end-to-end automation of ptychographic workflows—from data preparation to reconstruction and quality feedback—minimizing human intervention.

Proposed method

• PEAR employs a multi-agent architecture with dedicated LLM agents for knowledge retrieval, code generation, parameter recommendation, image quality assessment, and feedback summarization.
• Each agent is prompted with structured, domain-specific instructions and access to a curated knowledge base, enabling task-specific reasoning and decision-making.
• The framework generates MATLAB scripts for ptychographic reconstruction based on LLM-generated parameter configurations and executes them via a script runner agent.
• A quality collector agent solicits user feedback on reconstruction outcomes, including convergence, artifacts, probe accuracy, and mode structure.
• An updates recommender agent summarizes feedback and suggests parameter modifications, such as increasing detector blur standard deviation, which are then passed to a params updater agent for next-cycle configuration.
• The system supports integration with local knowledge bases and can be adapted to different experimental setups, ensuring flexibility and reproducibility.

Experimental results

Research questions

• RQ1 Can a multi-agent LLM framework automate ptychographic reconstruction with high success rates while minimizing human input?
• RQ2 How effective is the PEAR framework when using smaller, open-weight LLMs like LLaMA 3.1 8B in real-world ptychographic workflows?
• RQ3 To what extent can LLM agents improve parameter selection and reconstruction quality compared to traditional trial-and-error or black-box optimization methods?
• RQ4 Can the framework adapt to diverse experimental conditions and support customizable knowledge bases for domain-specific applications?
• RQ5 How does the multi-agent design enhance robustness and accuracy in complex, multi-parameter optimization tasks in ptychography?

Key findings

• PEAR achieved successful reconstruction with a single run using the GPT-4o-mini model, demonstrating high reliability in a real-world ptychographic workflow.
• The system successfully identified that the initial probe was accurate and that no grid artifacts were present in the reconstructed object, confirming high-quality output.
• After user feedback confirmed convergence and absence of artifacts, PEAR recommended increasing the detector blur standard deviation by 0.5 to improve robustness.
• The framework reduced the need for iterative manual tuning by automating parameter updates and feedback collection, significantly improving workflow throughput.
• PEAR demonstrated that smaller open-weight LLMs such as LLaMA 3.1 8B can support high-accuracy automation when combined with a structured multi-agent design.
• The system achieved a complete automation cycle—from script generation to execution and feedback-driven parameter update—within approximately 1145 seconds for a single reconstruction.