[论文解读] Synthetic Image Detection with CLIP: Understanding and Assessing Predictive Cues

Marco Willi, Melanie Mathys+1 more|arXiv (Cornell University)|Feb 12, 2026

Generative Adversarial Networks and Image Synthesis被引用 0

一句话总结

本文分析基于 CLIP 的合成图像检测器，介绍 SynthCLIC 配对数据集，并解释 CLIP 在真实与合成分类中依赖的高层次语义线索，显示对生成器类型的泛化存在差异。

ABSTRACT

Recent generative models produce near-photorealistic images, challenging the trustworthiness of photographs. Synthetic image detection (SID) has thus become an important area of research. Prior work has highlighted how synthetic images differ from real photographs--unfortunately, SID methods often struggle to generalize to novel generative models and often perform poorly in practical settings. CLIP, a foundational vision-language model which yields semantically rich image-text embeddings, shows strong accuracy and generalization for SID. Yet, the underlying relevant cues embedded in CLIP-features remain unknown. It is unclear, whether CLIP-based detectors simply detect strong visual artifacts or exploit subtle semantic biases, both of which would render them useless in practical settings or on generative models of high quality. We introduce SynthCLIC, a paired dataset of real photographs and high-quality synthetic counterparts from recent diffusion models, designed to reduce semantic bias in SID. Using an interpretable linear head with de-correlated activations and a text-grounded concept-model, we analyze what CLIP-based detectors learn. CLIP-based linear detectors reach 0.96 mAP on a GAN-based benchmark but only 0.92 on our high-quality diffusion dataset SynthCLIC, and generalization across generator families drops to as low as 0.37 mAP. We find that the detectors primarily rely on high-level photographic attributes (e.g., minimalist style, lens flare, or depth layering), rather than overt generator-specific artifacts. CLIP-based detectors perform well overall but generalize unevenly across diverse generative architectures. This highlights the need for continual model updates and broader training exposure, while reinforcing CLIP-based approaches as a strong foundation for more universal, robust SID.

研究动机与目标

• 出于对高保真度生成模型的信任与安全问题，推动将合成图像检测（SID）作为一个问题。
• 引入 SynthCLIC 以减少语义偏差并实现对扩散模型的鲁棒评估。
• 通过可解释的线性头与基于概念的词汇表，研究 CLIP 基于检测器学到的内容。
• 评估 CLIP 基于 SID 在 GAN 与扩散模型生成器上的泛化能力。

提出的方法

• 使用冻结的 CLIP 视觉编码器（ViT-L/14-336），并再添加两个可学习的线性层，将 [CLS] 令牌投影到低维空间。
• 对投影激活施加正交性约束，鼓励去相关、可解释的特征。
• 应用概念建模框架（稀疏线性 CDMs）并使用以摄影为焦点的词汇表来识别视觉线索。
• 通过将投影方向与词汇嵌入（TextSpan 和基于反义词的词汇表）进行比较，将学习到的表征锚定在 CLIP 的文本空间中。
• 在三个数据集（CNNSpot、SynthBuster+、SynthCLIC）以及跨数据集/泛化测试下，使用 mAP 及消融评估。

Figure 1: Synthetic images—even those generated by recent, high-quality generative models—differ from real photographs in subtle aspects. The figure shows a real image (left) and four paired synthetic variants from the SynthCLIC dataset. Shown are the most relevant terms (absolute logit contribution

实验结果

研究问题

• RQ1 Q1 在现实配对数据集中，CLIP 基于检测器从 GAN 基于的合成图像到现代扩散基于的合成图像的扩展能力有多强？
• RQ2 Q2 是否能够通过正交线性头和/或针对 SID 的人可解释概念来解释 CLIP 基于的分类？
• RQ3 Q3 在不同数据集中，哪些在 CLIP 表征中的视觉和摄影属性驱动真实与合成的判别？

主要发现

• 基于 CLIP 的检测器在 CNNSpot（基于 GAN）上达到 0.96 mAP，在 SynthCLIC（基于扩散）上达到 0.92。
• 跨数据集泛化较弱，跨数据集的泛化下降至最低 0.37 mAP。
• 检测器依赖于高层次的摄影属性（如极简风格、镜头效果、深度分层），而非显式的生成器特定伪影。
• 对 CLIP 特征的线性头在很大程度上是正交的，表明有多种不同因素共同贡献于 SID。
• SynthCLIC 相比早期配对数据集降低了语义偏差，但跨生成器家族的泛化仍然不均匀。
• 结合词汇表来解释学习到的方向，与深度分层和极简主义等感知线索相关，与观察到的合成图像伪影一致。
• 在跨数据集中，将投影维度（k 从 2 变到 16）对 mAP 的影响很小（绝对值≤0.03）。

Figure 2: Examples from the SynthBuster+ dataset. Different paired images are shown in each row. Each column depicts a different image source, starting with real photographs from the RAISE-1K dataset [ undefo ] , followed by synthetic images from the Synthbuster dataset [ undefb ] and images added b