[Paper Review] DeepTextMark: A Deep Learning-Driven Text Watermarking Approach for Identifying Large Language Model Generated Text

Travis Munyer, Tanvir, Abdullah+2 more|arXiv (Cornell University)|May 9, 2023

Topic Modeling4 citations

TL;DR

DeepTextMark proposes a deep learning-based text watermarking method to reliably detect LLM-generated text by embedding imperceptible watermarks using Word2Vec and Universal Sentence Encoder for semantic similarity, then detecting them with a blind, transformer-based classifier. The approach achieves near-perfect detection accuracy (over 98%) with just 5–10 sentences, while maintaining robustness, imperceptibility, and scalability without requiring original text for detection.

ABSTRACT

The rapid advancement of Large Language Models (LLMs) has significantly enhanced the capabilities of text generators. With the potential for misuse escalating, the importance of discerning whether texts are human-authored or generated by LLMs has become paramount. Several preceding studies have ventured to address this challenge by employing binary classifiers to differentiate between human-written and LLM-generated text. Nevertheless, the reliability of these classifiers has been subject to question. Given that consequential decisions may hinge on the outcome of such classification, it is imperative that text source detection is of high caliber. In light of this, the present paper introduces DeepTextMark, a deep learning-driven text watermarking methodology devised for text source identification. By leveraging Word2Vec and Sentence Encoding for watermark insertion, alongside a transformer-based classifier for watermark detection, DeepTextMark epitomizes a blend of blindness, robustness, imperceptibility, and reliability. As elaborated within the paper, these attributes are crucial for universal text source detection, with a particular emphasis in this paper on text produced by LLMs. DeepTextMark offers a viable "add-on" solution to prevailing text generation frameworks, requiring no direct access or alterations to the underlying text generation mechanism. Experimental evaluations underscore the high imperceptibility, elevated detection accuracy, augmented robustness, reliability, and swift execution of DeepTextMark.

Motivation & Objective

• To address the growing challenge of detecting LLM-generated text due to its high quality and indistinguishability from human-written text.
• To overcome the limitations of existing binary classifiers, which are unreliable, non-blind, and vulnerable to evasion.
• To develop a scalable, automatic, and reliable text watermarking system that preserves semantic meaning while enabling detection without access to original text.
• To ensure the watermarking method is robust against minor text modifications and maintains high detection accuracy across diverse text inputs.
• To enable practical deployment in real-world applications such as plagiarism detection and misuse prevention in education and publishing.

Proposed method

• Watermark insertion uses Word2Vec and Universal Sentence Encoder embeddings to find semantically similar replacement words for selected tokens, ensuring imperceptibility.
• The method applies word substitution at sentence level, with semantic similarity scores computed using sentence embeddings to maintain fluency and meaning.
• A transformer-based deep learning classifier detects watermarks solely from the watermarked text, ensuring blindness and high detection accuracy.
• Watermark detection does not require the original text, making the system practical for real-time and large-scale deployment.
• The system supports multi-sentence texts like essays by aggregating detection confidence across sentences, improving reliability.
• The watermarking process is fully automatic and can be integrated as an add-on to existing LLM text generation pipelines without modifying the model.

Experimental results

Research questions

• RQ1 Can a deep learning-based watermarking system achieve high detection accuracy while remaining imperceptible and robust to minor text modifications?
• RQ2 Can a blind detection mechanism reliably identify watermarked LLM-generated text without access to the original input?
• RQ3 How does the detection accuracy scale with the number of sentences in a document, particularly in multi-sentence texts like essays?
• RQ4 Can the watermarking method be implemented as a plug-in to existing LLM systems without requiring model-level changes?
• RQ5 To what extent does the use of pre-trained embeddings (Word2Vec and Universal Sentence Encoder) improve semantic fidelity during watermark insertion?

Key findings

• The proposed method achieves a detection accuracy of 98.75% on a single sentence and exceeds 98% accuracy with just 5–10 sentences, approaching near-perfect reliability.
• The watermarking process maintains high imperceptibility, as evidenced by semantic similarity scores consistently above 0.96 for replacement words.
• The transformer-based classifier demonstrates robustness, maintaining high detection accuracy even when watermarked text undergoes minor edits or rephrasing.
• The system operates efficiently with fast running speed, suitable for real-time or batch processing of large text volumes.
• The method is fully blind—detection does not require the original text—making it practical for deployment in environments where original content is unavailable.
• The approach scales well to multi-sentence documents, with detection confidence increasing significantly as the number of sentences grows.