[Paper Review] Building Better Deception Probes Using Targeted Instruction Pairs

Vikram Natarajan, Devina Jain+3 more|arXiv (Cornell University)|Feb 1, 2026

Deception detection and forensic psychology0 citations

TL;DR

System prompt choices dominate deception-probe performance; taxonomy-aligned prompts improve detection; deception is heterogeneous, requiring targeted probes rather than a universal detector.

ABSTRACT

Linear probes are a promising approach for monitoring AI systems for deceptive behaviour. Previous work has shown that a linear classifier trained on a contrastive instruction pair and a simple dataset can achieve good performance. However, these probes exhibit notable failures even in straightforward scenarios, including spurious correlations and false positives on non-deceptive responses. In this paper, we identify the importance of the instruction pair used during training. Furthermore, we show that targeting specific deceptive behaviors through a human-interpretable taxonomy of deception leads to improved results on evaluation datasets. Our findings reveal that instruction pairs capture deceptive intent rather than content-specific patterns, explaining why prompt choice dominates probe performance (70.6% of variance). Given the heterogeneity of deception types across datasets, we conclude that organizations should design specialized probes targeting their specific threat models rather than seeking a universal deception detector.

Motivation & Objective

• Motivate robust monitoring of deceptive behavior in AI systems amid concerns about model misrepresentation and strategic deception.
• Investigate how instruction-pair probes detect deceptive intent using model activations rather than surface content.
• Assess how prompt design, training data, and model layer choice contribute to deception-detection performance.
• Propose taxonomy-aligned prompts to improve detection across diverse deception datasets.

Proposed method

• Train linear (logistic regression) probes on residual activations from a fixed layer (layer 20) using token-aggregated activations from the model’s responses.
• Use contrastive instruction pairs (honest vs dishonest prompts) to create training signals that target deceptive intent rather than content.
• Evaluate prompts across 10 deception datasets with on-policy model responses, comparing baseline, best-average, and taxonomy-based prompts.
• Generate two prompt families: taxonomy-based prompts aligned to a human deception taxonomy and framing variations to explore surface-structure effects.
• Perform ANOVA over 2000 probes to quantify variance contributions from system prompts, training data, layer, and token-aggregation strategy.

Experimental results

Research questions

• RQ1 Does system prompt design (honest vs dishonest) dominate deception-probe performance relative to data, layer, or aggregation choices?
• RQ2 Do taxonomy-aligned prompts improve deception detection across diverse datasets compared to generic prompts?
• RQ3 Is deception heterogeneous across datasets such that specialized probes outperform universal detectors?
• RQ4 What underlying mechanisms do taxonomy prompts capture in model activations (intent vs content)?

Key findings

• System prompts explain 70.6% of performance variance, far exceeding layer, data, or token-aggregation effects.
• Taxonomy-based prompts yield a mean AUC improvement of 0.108 (20.5% relative gain) across ten datasets.
• Every dataset showed at least one taxonomy-based probe outperforming the baseline.
• Prompt-type heterogeneity means different deception types require different probes, challenging universal detectors.
• Clustering reveals five probe groups with shared detection directions, suggesting taxonomy probes access core deceptive-intent directions.
• Best taxonomy probe per dataset can vastly outperform baselines (e.g., MASK known facts improved from 0.374 to 0.697 AUC).