[Paper Review] Language Model Inversion

John X. Morris, Wenting Zhao+3 more|arXiv (Cornell University)|Nov 22, 2023

Topic Modeling4 citations

TL;DR

This paper introduces a novel method for language model inversion that recovers hidden prompts from next-token probability distributions, using a fine-tuned encoder-decoder model to reconstruct input text. On Llama-2 7B, the approach achieves a BLEU score of 59 and 78% token-level F1, recovering 27% of prompts exactly, even under limited access scenarios such as text-only outputs or top-K probabilities.

ABSTRACT

Language models produce a distribution over the next token; can we use this information to recover the prompt tokens? We consider the problem of language model inversion and show that next-token probabilities contain a surprising amount of information about the preceding text. Often we can recover the text in cases where it is hidden from the user, motivating a method for recovering unknown prompts given only the model's current distribution output. We consider a variety of model access scenarios, and show how even without predictions for every token in the vocabulary we can recover the probability vector through search. On Llama-2 7b, our inversion method reconstructs prompts with a BLEU of $59$ and token-level F1 of $78$ and recovers $27\%$ of prompts exactly. Code for reproducing all experiments is available at http://github.com/jxmorris12/vec2text.

Motivation & Objective

• To investigate whether next-token probability distributions in language models leak sufficient information to reconstruct the original input prompt.
• To develop a method that inverts language model outputs to recover the hidden prompt, even when only partial or indirect access to probabilities is available.
• To evaluate the robustness of prompt recovery across diverse access patterns, including full probability vectors, top-K outputs, and discrete text generation.
• To assess the transferability of inversion models across different language model architectures within the same family.
• To measure the extent to which private information (e.g., names, dates, nationalities) is preserved or leaked during prompt inversion.

Proposed method

• Propose a two-stage inversion framework: first, represent the next-token probability vector as a dense embedding; second, use a fine-tuned encoder-decoder transformer to decode this embedding into a natural language prompt.
• Train the inversion model on synthetic prompt-probability pairs using a reconstruction loss to minimize the difference between original and reconstructed prompts.
• Enable inversion under various access constraints by simulating scenarios such as full probability access, top-K probability retrieval, and discrete text generation via iterative querying.
• Use iterative query-based probability estimation to reconstruct missing probability components when only partial outputs are available.
• Leverage transfer learning to apply inversion models trained on one model (e.g., Llama-2 7B) to another (e.g., Llama-chat 7B) within the same model family.
• Apply the inversion method to private attribute recovery by fine-tuning on synthetic prompts containing names, dates, and nationalities.

Experimental results

Research questions

• RQ1 Can next-token probability distributions from large language models be inverted to recover the original input prompt with high fidelity?
• RQ2 How effective is prompt inversion under limited model access, such as only observing discrete text outputs or top-K probabilities?
• RQ3 To what extent is private information (e.g., names, dates, nationalities) preserved in the inversion process?
• RQ4 Does the inversion method generalize across different models in the same architecture family, such as Llama-2 and Llama-chat?
• RQ5 How does the performance of inversion compare to text-based jailbreak attacks, especially in the presence of RLHF alignment techniques?

Key findings

• On Llama-2 7B, the inversion method achieves a BLEU score of 59 and a token-level F1 of 78%, demonstrating high-quality reconstruction of input prompts.
• The method recovers 27% of prompts exactly, even when the prompt is not directly visible to the user.
• Private attributes such as country and nationality are reconstructed with 87.2% and 64.5% exact-match accuracy on the synthbio dataset, respectively.
• The model performs poorly on reconstructing individual dates and years, with accuracy dropping to 1.0% for years on the synthbio dataset.
• Inversion remains effective even when only discrete text outputs are available, as iterative querying enables sufficient probability estimation for reconstruction.
• The inversion method is robust to RLHF alignment techniques, unlike text-based jailbreaks, which are often inhibited by such alignment strategies.