[Paper Review] Exploiting Language Model for Efficient Linguistic Steganalysis

Biao Yi, Hanzhou Wu+2 more|arXiv (Cornell University)|Jul 26, 2021

Advanced Steganography and Watermarking Techniques4 citations

TL;DR

This paper proposes two pre-training methods—language model-based RNN (LM-RNN) and autoencoder-based RNN (AE-RNN)—to enhance linguistic steganalysis by leveraging the statistical differences in word conditional probability distributions between stego and carrier texts. The methods significantly improve detection accuracy and convergence speed over randomly initialized RNNs, achieving state-of-the-art performance, especially in imbalanced data scenarios with more cover texts than stego texts.

ABSTRACT

Recent advances in linguistic steganalysis have successively applied CNN, RNN, GNN and other efficient deep models for detecting secret information in generative texts. These methods tend to seek stronger feature extractors to achieve higher steganalysis effects. However, we have found through experiments that there actually exists significant difference between automatically generated stego texts and carrier texts in terms of the conditional probability distribution of individual words. Such kind of difference can be naturally captured by the language model used for generating stego texts. Through further experiments, we conclude that this ability can be transplanted to a text classifier by pre-training and fine-tuning to improve the detection performance. Motivated by this insight, we propose two methods for efficient linguistic steganalysis. One is to pre-train a language model based on RNN, and the other is to pre-train a sequence autoencoder. The results indicate that the two methods have different degrees of performance gain compared to the randomly initialized RNN, and the convergence speed is significantly accelerated. Moreover, our methods achieved the best performance compared to related works, while providing a solution for real-world scenario where there are more cover texts than stego texts.

Motivation & Objective

• To address the challenge of detecting secret information in generative text steganography using deep learning.
• To investigate whether language models used in steganography can be repurposed to improve steganalysis performance.
• To develop efficient, pre-trained models that generalize well even when cover texts vastly outnumber stego texts.
• To accelerate model convergence and improve detection accuracy through pre-training on carrier text distributions.

Proposed method

• Pre-train an RNN-based language model on large-scale carrier text to capture the conditional probability distribution of words, which reflects statistical differences between stego and carrier texts.
• Fine-tune the pre-trained language model as a steganalysis classifier to detect hidden messages in generated texts.
• Use a sequence autoencoder for pre-training to learn robust sentence-level representations, improving modeling of global text structure beyond next-word prediction.
• Transfer knowledge from the pre-trained language model or autoencoder to a downstream RNN classifier via fine-tuning for steganalysis.
• Train the steganalysis model end-to-end using cross-entropy loss, with pre-training on large, unlabeled carrier text corpora to improve generalization.
• Evaluate performance across multiple datasets (MOVIE, TWITTER) and steganographic methods (VLC, FLC, Bins) at varying embedding rates (1, 2, 3 bpw).

Experimental results

Research questions

• RQ1 Can the statistical characteristics of a language model used in generative steganography be exploited to improve steganalysis performance?
• RQ2 Does pre-training a language model or autoencoder on carrier texts lead to better detection accuracy and faster convergence than random initialization?
• RQ3 How does the performance of pre-trained models scale with increasing pre-training data size?
• RQ4 Can pre-trained models generalize effectively in real-world scenarios with highly imbalanced datasets (many more cover texts than stego texts)?

Key findings

• The proposed LM-RNN and AE-RNN methods achieve state-of-the-art detection accuracy, outperforming existing methods such as GNN, CNN, FCN, and randomly initialized RNNs.
• AE-RNN achieves slightly better performance than LM-RNN, likely due to its ability to model full sentence representations, not just next-word prediction.
• Pre-training significantly accelerates convergence, as shown by faster loss reduction in training curves (Fig. 6).
• Performance improves with larger pre-training sets: mean accuracy on MOVIE dataset increased from 0.9474 to 0.9535 when pre-training data grew from 6.3k to 18k samples.
• The best results were achieved in data-balanced scenarios, but the method remains effective even with highly imbalanced data, making it suitable for real-world deployment.