[Paper Review] From Memorization to Creativity: LLM as a Designer of Novel Neural-Architectures

Waqas Khalid, Dmitry I. Ignatov+1 more|arXiv (Cornell University)|Jan 6, 2026

Machine Learning in Materials Science0 citations

TL;DR

This paper studies how a code-capable LLM evolves into an autonomous neural-architecture designer through a 22-cycle closed-loop generate–evaluate–select–fine-tune process using low-fidelity performance signals and a MinHash–Jaccard novelty filter.

ABSTRACT

Large language models (LLMs) excel in program synthesis, yet their ability to autonomously navigate neural architecture design--balancing syntactic reliability, performance, and structural novelty--remains underexplored. We address this by placing a code-oriented LLM within a closed-loop synthesis framework, analyzing its evolution over 22 supervised fine-tuning cycles. The model synthesizes PyTorch convolutional networks which are validated, evaluated via low-fidelity performance signals (single-epoch accuracy), and filtered using a MinHash-Jaccard criterion to prevent structural redundancy. High-performing, novel architectures are converted into prompt-code pairs for iterative fine-tuning via parameter-efficient LoRA adaptation, initialized from the LEMUR dataset. Across cycles, the LLM internalizes empirical architectural priors, becoming a robust generator. The valid generation rate stabilizes at 50.6 percent (peaking at 74.5 percent), while mean first-epoch accuracy rises from 28.06 percent to 50.99 percent, and the fraction of candidates exceeding 40 percent accuracy grows from 2.04 percent to 96.81 percent. Analyses confirm the model moves beyond replicating existing motifs, synthesizing 455 high-performing architectures absent from the original corpus. By grounding code synthesis in execution feedback, this work provides a scalable blueprint for transforming stochastic generators into autonomous, performance-driven neural designers, establishing that LLMs can internalize empirical, non-textual rewards to transcend their training data.

Motivation & Objective

• Motivate and assess whether an LLM can autonomously design novel neural architectures when trained iteratively on its own successful generations.
• Balance three objectives: syntactic validity of generated PyTorch code, early learning signal from single-epoch CIFAR-10 accuracy, and structural novelty to avoid duplicate motifs.
• Demonstrate a closed-loop framework that internalizes empirical architectural priors and expands a diverse, high-quality design corpus.

Proposed method

• Treat the LLM as a stochastic generator of PyTorch architectures under a fixed API contract.
• Use a 22-cycle generate–evaluate–select–fine-tune loop with validity checks, single-epoch CIFAR-10 training, and MinHash–Jaccard novelty filtering.
• Fine-tune the LLM with LoRA on accepted self-generated architectures initialized from the LEMUR dataset.
• Evaluate generated architectures via a low-fidelity proxy (first-epoch accuracy) and a novelty criterion before adding to the training corpus.
• Maintain fixed prompts, decoding, and training protocol to isolate effects of iterative fine-tuning and data growth.

Experimental results

Research questions

• RQ1 Can iterative fine-tuning on its own successful designs improve an LLM's ability to generate valid, high-quality, and structurally novel neural architectures?
• RQ2 Does grounding code synthesis in execution feedback plus novelty filtering yield a robust architectural prior within a scalable loop?
• RQ3 How do validity, early-epoch performance, and design diversity evolve over multiple synthesis cycles?

Key findings

• The valid generation rate averages 50.6% across 22 cycles (Wilson CI [45.0%, 56.1%]).
• Mean first-epoch CIFAR-10 accuracy rises from 28.06% to 50.99%.
• The fraction of candidates with ≥40% accuracy grows from 2.04% to 92.86% by cycle 22 (peaking at 96.81%).
• Across cycles, 455 structurally novel architectures are discovered and added to the training corpus.
• In total, 455 high-performing architectures are absent from the original corpus and incorporated into the self-generated set.
• The loop maintains significant architectural diversity while improving reliability and learning efficiency.