[Paper Review] SelfIE: Self-Interpretation of Large Language Model Embeddings

Haozhe Chen, Carl Vondrick+1 more|arXiv (Cornell University)|Mar 16, 2024

Topic Modeling4 citations

TL;DR

SelfIE enables large language models to self-interpret their own hidden embeddings in natural language by prompting the model to describe its internal representations without additional training. It achieves zero-shot, faithful interpretation of complex, open-world concepts—enabling control via Supervised Control (95% effectiveness in overriding ethical steering) and Reinforcement Control (84.66% reduction in prompt injection success), all using only gradient updates on individual layers.

ABSTRACT

How do large language models (LLMs) obtain their answers? The ability to explain and control an LLM's reasoning process is key for reliability, transparency, and future model developments. We propose SelfIE (Self-Interpretation of Embeddings), a framework that enables LLMs to interpret their own embeddings in natural language by leveraging their ability to respond to inquiries about a given passage. Capable of interpreting open-world concepts in the hidden embeddings, SelfIE reveals LLM internal reasoning in cases such as making ethical decisions, internalizing prompt injection, and recalling harmful knowledge. SelfIE's text descriptions on hidden embeddings also open up new avenues to control LLM reasoning. We propose Supervised Control, which allows editing open-ended concepts while only requiring gradient computation of individual layer. We extend RLHF to hidden embeddings and propose Reinforcement Control that erases harmful knowledge in LLM without supervision targets.

Motivation & Objective

• To enable large language models (LLMs) to interpret their own internal hidden representations in natural language without retraining.
• To provide faithful, open-world explanations of complex reasoning processes, including ethical decisions and prompt injection responses.
• To enable lightweight, gradient-based control of LLM behavior by editing specific hidden layer representations.
• To reduce reliance on supervised data or complex probes by leveraging the LLM’s own decoding capability.
• To demonstrate that self-interpretation enables effective editing of harmful knowledge and ethical steering without performance degradation.

Proposed method

• SelfIE inserts the hidden embedding of interest into a prompt as a placeholder token, then uses the LLM to generate a natural language description of that embedding in a forward pass.
• The method relies on the LLM’s inherent ability to summarize or repeat content when prompted, repurposed for interpreting internal representations.
• Supervised Control uses gradient updates on a single layer to align the model’s interpretation with a desired behavior, using SelfIE-generated descriptions as targets.
• Reinforcement Control uses a reward model (the LLM itself) to provide feedback based on whether the interpretation contains harmful content, enabling unsupervised editing.
• Control is applied via gradient descent on individual layers, minimizing the need for full fine-tuning or data collection.
• Interpretation quality is evaluated via downstream tasks such as world state tracking in TextWorld and prompt injection resistance.

Figure 1: SelfIE interpretation of latent embeddings in Large Language Models. SelfIE produces open-world text explanations for the internal states in LLM without any training.

Experimental results

Research questions

• RQ1 Can LLMs self-interpret their own hidden embeddings in natural language without additional training?
• RQ2 Can self-interpretations faithfully reveal complex reasoning processes such as ethical decisions or prompt injection responses?
• RQ3 Can self-interpretations enable effective, lightweight control of LLM behavior through gradient-based editing?
• RQ4 Can Reinforcement Control erase harmful knowledge without supervision or labeled data?
• RQ5 Does self-interpretation-based control preserve model capabilities while modifying specific behaviors?

Key findings

• SelfIE achieves performance comparable to a 100-shot supervised probe in tracking world states in TextWorld, demonstrating zero-shot faithfulness.
• Supervised Control achieved a 95% effective rate in overriding ethical steering by editing layer representations based on SelfIE interpretations.
• Reinforcement Control reduced the success rate of prompt injection attacks from 89.06% to 4.4%, achieving an 84.66% editing effectiveness rate.
• The edited model preserved 95.85% of factual knowledge on the Counterfact benchmark, indicating minimal negative side effects.
• Smaller LLMs (7B and 13B) produced lower-quality interpretations due to poor instruction following, but accuracy matched the 70B model when instruction compliance was ensured.
• Editing on a single harmful prompt (e.g., building a molotov cocktail) erased responses to 388 other unrelated harmful queries, indicating broad generalization of edits.

Figure 2: Interpretation procedure for SelfIE . By replacing placeholder token embedding in the interpretation prompt with embedding being interpreted in the interpretation forward pass, we can generate text descriptions for the embedding.