[Paper Review] A LLM Assisted Exploitation of AI-Guardian

Nicholas Carlini|arXiv (Cornell University)|Jul 20, 2023

Adversarial Robustness in Machine LearningComputer Science3 citations

TL;DR

This paper demonstrates that GPT-4 can effectively assist in breaking AI-Guardian, a recently proposed adversarial defense at IEEE S&P 2023, by generating full attack code from natural language instructions. The attack reduces AI-Guardian’s robustness from 98% to just 8%, proving the defense is ineffective under the original threat model, and highlights the growing power of LLMs as research assistants in adversarial machine learning.

ABSTRACT

Large language models (LLMs) are now highly capable at a diverse range of tasks. This paper studies whether or not GPT-4, one such LLM, is capable of assisting researchers in the field of adversarial machine learning. As a case study, we evaluate the robustness of AI-Guardian, a recent defense to adversarial examples published at IEEE S&P 2023, a top computer security conference. We completely break this defense: the proposed scheme does not increase robustness compared to an undefended baseline. We write none of the code to attack this model, and instead prompt GPT-4 to implement all attack algorithms following our instructions and guidance. This process was surprisingly effective and efficient, with the language model at times producing code from ambiguous instructions faster than the author of this paper could have done. We conclude by discussing (1) the warning signs present in the evaluation that suggested to us AI-Guardian would be broken, and (2) our experience with designing attacks and performing novel research using the most recent advances in language modeling.

Motivation & Objective

• To evaluate whether GPT-4 can act as a capable research assistant in generating adversarial attacks on modern defenses.
• To test the robustness of AI-Guardian, a recent defense against adversarial examples presented at IEEE S&P 2023.
• To identify warning signs in the original AI-Guardian paper that foreshadowed its vulnerability to attack.
• To explore the feasibility and efficiency of using large language models to automate complex machine learning research tasks.

Proposed method

• GPT-4 was prompted with natural language instructions to generate end-to-end attack code for breaking AI-Guardian, including pattern recovery and backdoor injection.
• The attack used a gradient-based optimization approach to recover the hidden trigger pattern in the defense, leveraging PyTorch’s autograd for backpropagation.
• A masked, permuted backdooring strategy was employed, where the model was trained to generate a pattern that, when applied, produced logits matching the defended model’s output.
• The attack code was iteratively refined through human-LLM interaction, with minimal manual corrections required.
• The final attack used SGD to minimize the mean squared error between predicted and target logits, with pattern values clipped to [-0.5, 0.5] to match the original model’s assumptions.
• The entire attack pipeline was implemented entirely by GPT-4, with human oversight for correctness and clarity.

Experimental results

Research questions

• RQ1 Can a large language model like GPT-4 reliably generate functional adversarial attack code from high-level natural language descriptions?
• RQ2 To what extent can LLMs automate the process of breaking modern machine learning defenses without human-written code?
• RQ3 What structural or evaluation flaws in a defense might signal its vulnerability to LLM-assisted attacks?
• RQ4 How effective is GPT-4 at reverse-engineering complex defense mechanisms such as AI-Guardian?

Key findings

• GPT-4 successfully generated complete, working attack code for breaking AI-Guardian using only natural language prompts, with minimal human intervention.
• The attack reduced AI-Guardian’s robustness from 98% to just 8% under the original threat model, demonstrating it offers no meaningful protection.
• The defense was vulnerable to a pattern recovery attack that exploited its reliance on a fixed, learnable trigger pattern in the input space.
• Even with ambiguous or imprecise instructions, GPT-4 often produced correct or near-correct code, and errors were quickly resolved through iterative refinement.
• The original AI-Guardian paper contained several red flags—such as claims of robustness to nearly unbounded perturbations—that should have signaled its fragility.
• The study confirms that LLMs can now serve as highly effective research assistants in adversarial machine learning, accelerating attack development and reducing the need for low-level coding expertise.