[Paper Review] Generalized Deep Image to Image Regression

Venkataraman Santhanam, Vlad I. Morariu+1 more|arXiv (Cornell University)|Dec 10, 2016

Generative Adversarial Networks and Image Synthesis55 references4 citations

TL;DR

This paper proposes the Recursively Branched Deconvolutional Network (RBDN), a fully convolutional, end-to-end trainable deep network for generic image-to-image regression. By using recursive branching with learnable upsampling and parameter sharing, RBDN creates a cheap, rich multi-context representation early in the network, enabling strong performance across diverse tasks like relighting, denoising, and colorization without task-specific modifications or post-processing.

ABSTRACT

We present a Deep Convolutional Neural Network architecture which serves as a generic image-to-image regressor that can be trained end-to-end without any further machinery. Our proposed architecture: the Recursively Branched Deconvolutional Network (RBDN) develops a cheap multi-context image representation very early on using an efficient recursive branching scheme with extensive parameter sharing and learnable upsampling. This multi-context representation is subjected to a highly non-linear locality preserving transformation by the remainder of our network comprising of a series of convolutions/deconvolutions without any spatial downsampling. The RBDN architecture is fully convolutional and can handle variable sized images during inference. We provide qualitative/quantitative results on $3$ diverse tasks: relighting, denoising and colorization and show that our proposed RBDN architecture obtains comparable results to the state-of-the-art on each of these tasks when used off-the-shelf without any post processing or task-specific architectural modifications.

Motivation & Objective

• Address the lack of a generic, task-agnostic deep learning architecture for image-to-image regression that performs well across diverse tasks.
• Overcome the inherent locality-context trade-off in existing image-to-image networks by developing a multi-scale, early-context representation.
• Design a fully convolutional network that handles variable-sized inputs and requires no architectural modifications for different tasks.
• Demonstrate that a single, off-the-shelf architecture can achieve competitive performance on multiple distinct image-to-image regression tasks.

Proposed method

• Employ a recursive branching scheme with learnable upsampling to generate multi-scale features early in the network, enabling efficient multi-context representation.
• Use extensive parameter sharing across branches to reduce model complexity while maintaining rich feature representation.
• Construct a composite multi-context feature map by fusing outputs from all recursive branches at a shared pooling layer.
• Pass the composite representation through a stack of 9 fully convolutional layers with no spatial downsampling to preserve locality and apply strong non-linear transformations.
• Train the entire network end-to-end using standard backpropagation with a loss function appropriate to each task (e.g., MSE for denoising, cross-entropy for colorization).
• Utilize a fixed-depth architecture across tasks, with no residual connections, to evaluate the effectiveness of recursive branching over depth.

Experimental results

Research questions

• RQ1 Can a single, generic deep network architecture achieve state-of-the-art performance across diverse image-to-image regression tasks without task-specific architectural modifications?
• RQ2 Does early, recursive multi-context feature learning via branching and learnable upsampling effectively alleviate the locality-context trade-off in image-to-image networks?
• RQ3 Can a fully convolutional, end-to-end trainable network handle variable-sized inputs and generalize across different image-to-image tasks?
• RQ4 How does the performance of a shallow, recursively branched network compare to deeper, residual-based networks like DnCNN in tasks like denoising and relighting?
• RQ5 To what extent does the choice of loss function (e.g., MSE vs. cross-entropy) impact the perceptual quality of outputs in ambiguous tasks like colorization?

Key findings

• The 3-branch RBDN achieves near-perfect relighting results on the validation set, demonstrating strong generalization and robustness to illumination variations.
• The 3-branch RBDN outperforms DnCNN in denoising for noise levels between σ=25 and σ=55, even though it is shallower (9 layers vs. 18 layers) and lacks residual connections.
• The RBDN generalizes effectively to high noise levels (σ=55), which were outside the training distribution, indicating strong generalization capability.
• For colorization, the RBDN with a softmax cross-entropy loss (RBDN-Lab) produces more saturated and perceptually pleasing results than MSE-based models, overcoming the under-saturation problem.
• The RBDN-Lab model achieves colorization performance comparable to a more complex linear dilated convolutional network from prior work, despite a simpler architecture.
• The RBDN’s performance is competitive with state-of-the-art models across all three tasks—relighting, denoising, and colorization—when used off-the-shelf without any post-processing or architectural tuning.