[Paper Review] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Richard Zhang, Phillip Isola+3 more|arXiv (Cornell University)|Jan 11, 2018

Advanced Image Processing Techniques54 references809 citations

TL;DR

Deep features from diverse architectures and supervision levels predict human perceptual similarity far better than traditional metrics, and can be calibrated to improve alignment with human judgments across distortions and real algorithms.

ABSTRACT

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

Motivation & Objective

• Motivate and quantify how well deep feature representations align with human perceptual similarity judgments.
• Create a large, varied perceptual similarity dataset (BAPPS) to evaluate metrics across distortions and real algorithms.
• Evaluate deep features from supervised, self-supervised, and unsupervised models, and test calibration of these features.
• Assess whether architecture or training signal (not just ImageNet) drives perceptual alignment.
• Explore whether linear calibration of pre-trained features suffices to improve perceptual distance measurements.

Proposed method

• Introduce the Berkeley-Adobe Perceptual Patch Similarity (BAPPS) dataset with 2AFC and JND judgments for 64x64 patches.
• Compute distances between patches in deep feature spaces by normalizing channel activations, applying layer-wise channel weights, and aggregating across space and layers (LPIPS framework).
• Evaluate multiple architectures (SqueezeNet, AlexNet, VGG) and supervision signals (supervised, self-supervised, unsupervised).
• Explore three training configurations for LPIPS: lin (linear calibration of fixed features), tune (fine-tune from pre-trained weights), scratch (train from random) .
• Test on traditional distortions, CNN-based distortions, and real algorithm outputs (superresolution, frame interpolation, video deblurring, colorization).
• Train a small predictor G to map distances to perceptual judgments to examine predictability of judgments from distance pairs.

Experimental results

Research questions

• RQ1 Do deep features pretrained on classification (and even self-supervised/unsupervised) align with human perceptual similarity across distortions?
• RQ2 Does the network architecture or training signal primarily determine perceptual alignment, and can simple calibration improve correlation with human judgments?
• RQ3 Can the learned perceptual distance generalize from distortions to real-world algorithm outputs and to different perceptual tests (2AFC and JND)?
• RQ4 Is a linear calibration of a pre-trained network sufficient to improve perceptual distance, or is full fine-tuning necessary?
• RQ5 How do deep-feature perceptual metrics compare to traditional FR-IQA metrics (e.g., SSIM, FSIM) across diverse distortions?

Key findings

• Deep features outperform traditional perceptual metrics by large margins on human judgments across a large, varied dataset.
• Humans are ~73.9% consistent with the evaluated metrics, while supervised networks around 67–69% and traditional metrics around 63% are lower.
• Across architectures (SqueezeNet, AlexNet, VGG) and supervision types (supervised, self-supervised, unsupervised), deep features yield strong perceptual distances without additional calibration.
• Linear calibration of pre-trained networks (LPIPS lin) improves performance over fixed features, and tuning further (LPIPS tune) yields the best results among the configurations tested; training from scratch (LPIPS scratch) is often worse than calibration.
• Calibration on perceptual judgments transfers to real-algorithm outputs, improving performance in 11 of 12 cases across three networks and four real-algorithm tasks.
• Perceptual similarity appears to be an emergent property of representations trained for semantic prediction or other natural tasks, with self-supervised and unsupervised representations performing comparably to supervised ones.