[Paper Review] Deep Network Interpolation for Continuous Imagery Effect Transition

Xintao Wang, Ke Yu+3 more|arXiv (Cornell University)|Nov 26, 2018

Advanced Image Processing Techniques37 references4 citations

TL;DR

This paper proposes Deep Network Interpolation (DNI), a simple yet general method for achieving smooth, continuous transitions between diverse imagery effects—such as super-resolution, style transfer, and image restoration—by linearly interpolating the parameters of two or more pre-trained, correlated deep neural networks. The approach enables fine-grained, real-time control over output effects without retraining, outperforming pixel-level interpolation and existing task-specific designs in visual quality and flexibility.

ABSTRACT

Deep convolutional neural network has demonstrated its capability of learning a deterministic mapping for the desired imagery effect. However, the large variety of user flavors motivates the possibility of continuous transition among different output effects. Unlike existing methods that require a specific design to achieve one particular transition (e.g., style transfer), we propose a simple yet universal approach to attain a smooth control of diverse imagery effects in many low-level vision tasks, including image restoration, image-to-image translation, and style transfer. Specifically, our method, namely Deep Network Interpolation (DNI), applies linear interpolation in the parameter space of two or more correlated networks. A smooth control of imagery effects can be achieved by tweaking the interpolation coefficients. In addition to DNI and its broad applications, we also investigate the mechanism of network interpolation from the perspective of learned filters.

Motivation & Objective

• To address the limitation of deterministic deep networks that produce only one fixed output, which fails to accommodate diverse user preferences or varying restoration strengths.
• To enable continuous, smooth transitions between different imagery effects (e.g., MSE vs. GAN in super-resolution, day vs. night photos) without requiring task-specific architectural modifications.
• To provide a generalizable, lightweight solution for controlling image effects via interpolation coefficients, applicable across low-level vision tasks.
• To investigate the effectiveness and underlying mechanism of parameter-space interpolation compared to pixel-level interpolation and instance normalization-based methods.

Proposed method

• DNI performs linear interpolation in the parameter space between two pre-trained, correlated deep neural networks—one for effect A and one fine-tuned for effect B.
• The interpolation coefficient α ∈ [0,1] controls the blend: θ_interpolated = α·θ_A + (1−α)·θ_B, where θ_A and θ_B are the model parameters of the two networks.
• The interpolated model is used for inference with the same input, generating a continuous transition between effects A and B without additional training.
• The method is compatible with standard architectures like VGG, ResNet, and DenseNet, and requires no architectural changes or retraining for each interpolation.
• Fine-tuning is applied only once to adapt a base model to a new effect, enabling interpolation between the original and fine-tuned models.
• The approach is evaluated across super-resolution, image-to-image translation, style transfer, and semantic attribute transfer, demonstrating broad applicability.

Experimental results

Research questions

• RQ1 Can linear interpolation in the parameter space of pre-trained deep networks achieve smooth, visually plausible transitions between diverse image effects?
• RQ2 How does DNI compare to pixel-level interpolation and instance normalization-based interpolation in terms of visual quality and artifact suppression?
• RQ3 To what extent can DNI generalize across different low-level vision tasks, including image restoration, style transfer, and semantic attribute manipulation?
• RQ4 What role do convolutional layer parameters play in style and texture transfer, compared to instance normalization parameters?

Key findings

• DNI achieves smooth, continuous transitions between image effects such as MSE and GAN-based super-resolution, with no abrupt changes or visual artifacts.
• DNI outperforms pixel-level interpolation, which produces unnatural ghosting artifacts, especially at object boundaries.
• Fine-tuning both convolutional layers and instance normalization layers leads to better texture transfer than fine-tuning only instance normalization layers, indicating the importance of convolutional features in style modeling.
• DNI enables smooth semantic transitions on face attributes (e.g., young to old, male to female) even when no intermediate training data exists, producing natural-looking intermediate faces.
• The method is computationally efficient, requiring no retraining for each interpolation, and supports real-time control via a single interpolation coefficient.
• DNI generalizes across tasks: it successfully enables continuous transitions in depth of field control, day-to-night photo transformation, and multi-style blending in image stylization.