[Paper Review] A Watermark for Large Language Models

John Kirchenbauer, Jonas Geiping+4 more|arXiv (Cornell University)|Jan 24, 2023

Topic ModelingComputer Science113 citations

TL;DR

The paper proposes a probabilistic watermarking framework for LLM outputs that is detectable from short text spans without model access, using a green/red token scheme and statistical tests, and analyzes its robustness and impact on text quality.

ABSTRACT

Potential harms of large language models can be mitigated by watermarking model output, i.e., embedding signals into generated text that are invisible to humans but algorithmically detectable from a short span of tokens. We propose a watermarking framework for proprietary language models. The watermark can be embedded with negligible impact on text quality, and can be detected using an efficient open-source algorithm without access to the language model API or parameters. The watermark works by selecting a randomized set of "green" tokens before a word is generated, and then softly promoting use of green tokens during sampling. We propose a statistical test for detecting the watermark with interpretable p-values, and derive an information-theoretic framework for analyzing the sensitivity of the watermark. We test the watermark using a multi-billion parameter model from the Open Pretrained Transformer (OPT) family, and discuss robustness and security.

Motivation & Objective

• Motivate harms from unmarked AI-generated text and the need for auditable detection.
• Introduce a watermarking framework that does not require model access or retraining.
• Develop a statistical detection method with interpretable p-values for watermark presence.
• analyzes the information-theoretic sensitivity of the watermark and its robustness to attacks.

Proposed method

• Define a green/red token partition and either hard (Algorithm 1) or soft (Algorithm 2) watermark rule.
• Seed the partition with a hash of prior tokens to reproduce the rule without model access.
• Modify the sampling distribution to favor green tokens (soft) or forbid red tokens (hard).
• Detect watermark using a one-proportion z-test on the count of green tokens in a text window (H0: no watermark).
• Provide private watermark variants using PRFs to conceal the rule and discuss attack-resistance and API considerations.
• Theoretical analysis links watermark parameters to detection power via spike entropy and derived bounds (Theorems 4.2, 4.3).

Experimental results

Research questions

• RQ1 Can a watermark be embedded in LLM outputs without retraining or model access and yet be reliably detected by third parties?
• RQ2 What is the detection power and false-positive rate of the watermark under various decoding strategies (multinomial, greedy, beam search)?
• RQ3 How does text entropy affect watermark detection sensitivity and sample efficiency?
• RQ4 How robust is the watermark to adversarial modification of tokens and to private-key (API) implementations?

Key findings

• A simple hard red-list watermark can be detected with a one-proportion z-test from as few as 16 tokens and yields a false-positive rate around 3e-5 when z>4.
• A soft watermark boosts probabilities toward a green token set in high-entropy regions with limited impact on perplexity, and detection sensitivity scales with text entropy.
• Experiments with OPT-1.3B/OPT-2.7B show high detection rates: e.g., 98.4-99.6% empirical sensitivity across multinomial and beam-search decoding under z>4.
• The framework maintains detection without model access, supports public or private deployment, and remains robust to certain attack strategies, while highlighting limitations in low-entropy, memorized text scenarios.