[Paper Review] E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles

Markus Kettunen, Erik Härkönen+1 more|arXiv (Cornell University)|Jun 10, 2019

Advanced Image Processing Techniques31 references33 citations

TL;DR

E-LPIPS extends LPIPS by averaging over an effectively infinite ensemble of random input transformations and architectural tweaks, yielding a robust perceptual similarity metric that preserves human judgment while being resistant to adversarial attacks. It also reveals perceptual convexity in image space and improves loss performance in restoration tasks.

ABSTRACT

It has been recently shown that the hidden variables of convolutional neural networks make for an efficient perceptual similarity metric that accurately predicts human judgment on relative image similarity assessment. First, we show that such learned perceptual similarity metrics (LPIPS) are susceptible to adversarial attacks that dramatically contradict human visual similarity judgment. While this is not surprising in light of neural networks' well-known weakness to adversarial perturbations, we proceed to show that self-ensembling with an infinite family of random transformations of the input --- a technique known not to render classification networks robust --- is enough to turn the metric robust against attack, while retaining predictive power on human judgments. Finally, we study the geometry imposed by our our novel self-ensembled metric (E-LPIPS) on the space of natural images. We find evidence of "perceptual convexity" by showing that convex combinations of similar-looking images retain appearance, and that discrete geodesics yield meaningful frame interpolation and texture morphing, all without explicit correspondences.

Motivation & Objective

• Demonstrate vulnerabilities of LPIPS-style perceptual similarity metrics to adversarial attacks that contradict human judgment.
• Propose E-LPIPS by ensembling random input transformations and network alterations to boost robustness without losing human-predicted judgment accuracy.
• Analyze the geometric properties induced by E-LPIPS on natural image space, including perceptual convexity and meaningful geodesics.

Proposed method

• Apply a randomized, effectively infinite ensemble of geometric and color transformations to input images before computing LPIPS distance.
• Compute distances over all convolutional layers rather than only the last layer, including input color terms with learned weights.
• Introduce dropout in all layers and replace max pooling with average pooling to improve gradient flow and robustness.
• Define E-LPIPS as the expectation of LPIPS over the transformation ensemble: d_E-LPIPS(x,y)=E[d_LPIPS(T(x),T(y))].
• Provide practical guidance: single random sample suffices per iteration, but N>=300 samples recommended for precision.
• Evaluate robustness against white-box adversarial attacks using EoT-style optimization and compare to LPIPS variants.

Experimental results

Research questions

• RQ1 Do LPIPS-style perceptual similarity metrics suffer from adversarial attacks that misalign with human judgment?
• RQ2 Does random transformation self-ensembling (E-LPIPS) improve robustness to attacks while preserving correlation with human perceptual judgments?
• RQ3 What geometric properties (e.g., perceptual convexity, meaningful geodesics) emerge under the E-LPIPS metric?
• RQ4 Can E-LPIPS serve as an effective loss function for image restoration and related tasks without sacrificing optimization stability?

Key findings

• E-LPIPS is significantly more robust to adversarial attacks than lpips-vgg and lpips-sqz, reducing L2-distance growth under attack (3.7–14.5 vs. 16.0–16.2 in LPIPS variants).
• Attacks constrained to small L2 perturbations increase LPIPS distances much more than E-LPIPS, demonstrating improved resilience.
• Ablation shows each transformation component contributes to robustness; full ensemble yields the strongest defense.
• E-LPIPS achieves similar human-judgment prediction accuracy to LPIPS (2AFC task: 69.16% vs. 68.65–70.07% for non-ensembled variants).
• E-LPIPS exhibits perceptual convexity: barycenters preserve appearance better than L2 or LPIPS, and discrete geodesics yield meaningful frame interpolation and texture morphing without explicit correspondences.
• Using E-LPIPS as a loss function yields consistently better restoration results than plain LPIPS in their experiments.