[Paper Review] A Semantic Invariant Robust Watermark for Large Language Models

Aiwei Liu, Leyi Pan+3 more|arXiv (Cornell University)|Oct 10, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a semantic invariant watermarking method for large language models that generates watermark logits based on the semantic embeddings of preceding tokens using a dedicated watermark model. By leveraging a BERT-based encoder and a trained transformation network with similarity and normalization losses, the method achieves high attack robustness under paraphrasing and synonym substitution, while maintaining strong security robustness, resolving the long-standing trade-off between robustness types.

ABSTRACT

Watermark algorithms for large language models (LLMs) have achieved extremely high accuracy in detecting text generated by LLMs. Such algorithms typically involve adding extra watermark logits to the LLM's logits at each generation step. However, prior algorithms face a trade-off between attack robustness and security robustness. This is because the watermark logits for a token are determined by a certain number of preceding tokens; a small number leads to low security robustness, while a large number results in insufficient attack robustness. In this work, we propose a semantic invariant watermarking method for LLMs that provides both attack robustness and security robustness. The watermark logits in our work are determined by the semantics of all preceding tokens. Specifically, we utilize another embedding LLM to generate semantic embeddings for all preceding tokens, and then these semantic embeddings are transformed into the watermark logits through our trained watermark model. Subsequent analyses and experiments demonstrated the attack robustness of our method in semantically invariant settings: synonym substitution and text paraphrasing settings. Finally, we also show that our watermark possesses adequate security robustness. Our code and data are available at \href{https://github.com/THU-BPM/Robust_Watermark}{https://github.com/THU-BPM/Robust\_Watermark}. Additionally, our algorithm could also be accessed through MarkLLM \citep{pan2024markllm} \footnote{https://github.com/THU-BPM/MarkLLM}.

Motivation & Objective

• To address the inherent trade-off between attack robustness and security robustness in LLM watermarking.
• To develop a watermarking method that remains effective under semantically invariant text modifications such as paraphrasing and synonym substitution.
• To ensure watermark logits are diverse, unbiased, and resistant to frequency-based cracking attacks.
• To minimize latency impact during text generation by enabling parallel computation of watermark and LLM logits.
• To achieve high detection accuracy while preserving text quality, as measured by perplexity.

Proposed method

• The method uses a separate embedding LLM (e.g., Compositional-BERT) to extract semantic embeddings from preceding tokens.
• A trainable watermark model transforms these semantic embeddings into watermark logits via a four-layer feedforward network.
• The watermark model is trained with two objectives: a similarity loss to preserve semantic similarity in watermark logits, and a normalization loss to ensure balanced, bimodal logits centered at zero.
• Watermark logits are scaled using a tanh function to produce values close to ±1, minimizing bias and maximizing diversity.
• During generation, watermark logits are combined with LLM logits in parallel, with minimal latency overhead.
• Watermark detection is performed by averaging watermark logits across all tokens; a significantly positive average indicates presence of watermark.

Figure 1: An illustration of our semantic invariant robust watermarking method. Text is input into a generative LLM for token logits and an embedding LLM for text embedding. The embedding is converted into watermark logits via the Watermark Model. LLM logits and watermark logits are then combined fo

Experimental results

Research questions

• RQ1 Can a watermarking method maintain robustness under semantically invariant text perturbations such as synonym replacement and paraphrasing?
• RQ2 Can a watermarking scheme simultaneously achieve high attack robustness and strong security robustness, overcoming the traditional trade-off?
• RQ3 How can watermark logits be generated in a way that is invariant to semantic changes but sensitive to structural patterns in the text?
• RQ4 To what extent does the watermarking method affect text quality, as measured by perplexity?
• RQ5 Can the watermarking process be efficiently parallelized to minimize inference latency?

Key findings

• The proposed method achieves attack robustness comparable to KGW-1 (global watermark logits), approaching the theoretical upper bound for watermark logit-based methods under synonym substitution and paraphrasing.
• In spoofing attacks, the method demonstrates strong security robustness, with low decryption accuracy for adversaries attempting to extract the watermark, outperforming prior approaches.
• The watermark model maintains a strong correlation between input embedding similarity and output watermark logit similarity, especially for highly similar semantic inputs (similarity > 0.8).
• Text quality is minimally affected, with only a slight increase in perplexity—slightly lower than KGW-series methods—indicating better fluency and consistency.
• The method incurs only a marginal latency increase, with a 40% time overhead on LLaMA-7B when not parallelized, and near-identical generation speed when watermark and LLM logits are computed in parallel.
• The normalization loss effectively prevents extreme logit values and ensures symmetrical, unbiased token selection, enhancing security and diversity.