[Paper Review] Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents

Jacopo Teneggi, S. M. Bargeen A. Turzo+5 more|arXiv (Cornell University)|Mar 16, 2026

Machine Learning in Materials Science0 citations

TL;DR

Introduces Agent Rosetta, an LLM-driven agent integrated with RosettaScripts to iteratively design proteins, achieving competitive performance with ML baselines and enabling non-canonical residues.

ABSTRACT

Large language models (LLMs) are capable of emulating reasoning and using tools, creating opportunities for autonomous agents that execute complex scientific tasks. Protein design provides a natural testbed: although machine learning (ML) methods achieve strong results, these are largely restricted to canonical amino acids and narrow objectives, leaving unfilled need for a generalist tool for broad design pipelines. We introduce Agent Rosetta, an LLM agent paired with a structured environment for operating Rosetta, the leading physics-based heteropolymer design software, capable of modeling non-canonical building blocks and geometries. Agent Rosetta iteratively refines designs to achieve user-defined objectives, combining LLM reasoning with Rosetta's generality. We evaluate Agent Rosetta on design with canonical amino acids, matching specialized models and expert baselines, and with non-canonical residues -- where ML approaches fail -- achieving comparable performance. Critically, prompt engineering alone often fails to generate Rosetta actions, demonstrating that environment design is essential for integrating LLM agents with specialized software. Our results show that properly designed environments enable LLM agents to make scientific software accessible while matching specialized tools and human experts.

Motivation & Objective

• Motivate the use of autonomous agents to automate complex Rosetta-based protein design tasks.
• Demonstrate that environment design is essential for bridging LLMs with domain-specific software.
• Show that Agent Rosetta can match specialized ML models on canonical design and outperform human baselines with non-canonical residues.
• Provide a framework for multi-turn optimization that adapts protocols based on intermediate design metrics.

Proposed method

• Develop Agent Rosetta, an LLM agent interfaced with a tailored RosettaScripts environment for robust action generation.
• Represent design states with surrogate metrics (radius of gyration, cavity volume, buried unsatisfied H-bonds, RMSD to target, pLDDT) to steer decisions without full Pose data in context.
• Define three action types (rotamer_change, backbone_change, go_back_to) to cover design pipelines within RosettaScripts.
• Use structured multi-turn reasoning: action selection, parameter generation with environment docs, execution, and feedback to refine designs.
• Abstract RosettaScripts penalties and operations into simplified templates to ensure semantically correct actions and enable reliable multi-turn interaction.

Figure 1 : Illustration of our multi-turn agentic system. (A) Schematics of Agent Rosetta’s interaction protocol. (B) Design refinement: the agent chooses the action, and, after the environment returns the action documentation, it generates the action call with its parameters.

Experimental results

Research questions

• RQ1 Can an LLM agent effectively control Rosetta via a structured environment to optimize protein design goals?
• RQ2 Does environment design (beyond prompting) enable reliable generation of domain-specific actions for RosettaScripts?
• RQ3 How does Agent Rosetta perform on canonical fixed-backbone design compared with ProteinMPNN and human protocols?
• RQ4 Can Agent Rosetta design a non-canonical amino acid in protein cores, a regime where ML models struggle?

Key findings

• Agent Rosetta achieves competitive design quality with ProteinMPNN on canonical amino acids (within 0.20 Å RMSD tolerance).
• Agent Rosetta outperforms expert human baselines in the non-canonical amino acid task, demonstrating capability beyond data-driven methods.
• Prompting alone is insufficient to reliably generate Rosetta actions; environment and syntax design are essential for robust agent performance.
• With a tailored environment, all tested LLMs achieved action success rates ≥ 86%.
• GPT-5-based configurations yielded the best cost‑performance tradeoffs in several experiments.
• In NCAA design, Agent Rosetta achieved higher AF3 RMSD and pLDDT than a human baseline, indicating improved structural validation on average.

Figure 2 : A failure example of prompting for generation of composition penalties. Even though Agent Rosetta wants to reduce proline content, the penalty block achieves the opposite effect.