[Paper Review] Learning Continuous Image Representation with Local Implicit Image Function

Yinbo Chen, Sifei Liu+1 more|arXiv (Cornell University)|Dec 16, 2020

Advanced Vision and Imaging50 references32 citations

TL;DR

LIIF represents images as a continuous function using local latent codes and a shared decoder, enabling arbitrary-resolution rendering and better handling of size-varied ground-truths without resizing. It supports extrapolation to very high scales (up to ×30) and bridges discrete and continuous 2D representations.

ABSTRACT

How to represent an image? While the visual world is presented in a continuous manner, machines store and see the images in a discrete way with 2D arrays of pixels. In this paper, we seek to learn a continuous representation for images. Inspired by the recent progress in 3D reconstruction with implicit neural representation, we propose Local Implicit Image Function (LIIF), which takes an image coordinate and the 2D deep features around the coordinate as inputs, predicts the RGB value at a given coordinate as an output. Since the coordinates are continuous, LIIF can be presented in arbitrary resolution. To generate the continuous representation for images, we train an encoder with LIIF representation via a self-supervised task with super-resolution. The learned continuous representation can be presented in arbitrary resolution even extrapolate to x30 higher resolution, where the training tasks are not provided. We further show that LIIF representation builds a bridge between discrete and continuous representation in 2D, it naturally supports the learning tasks with size-varied image ground-truths and significantly outperforms the method with resizing the ground-truths.

Motivation & Objective

• Motivate representing images as continuous functions rather than fixed-resolution grids.
• Propose Local Implicit Image Function (LIIF) using local latent codes and a shared decoder for continuous RGB prediction.
• Demonstrate that LIIF enables arbitrary-resolution rendering and extrapolation beyond training scales.
• Show that LIIF naturally handles size-varied ground-truths without resizing and improves fidelity over ground-truth resizing.

Proposed method

• Represent each image with a 2D feature map M(i) of latent codes distributed in the image domain.
• Use a shared MLP decoder f_theta that takes a latent code z and a coordinate x to predict RGB values, enabling x -> s predictions.
• Apply feature unfolding to enrich local codes by concatenating neighboring latent vectors (3x3 neighborhood).
• Mitigate prediction discontinuities with a local ensemble that combines four neighboring latent-code-based predictions weighted by an area-based confidence.
• Introduce cell decoding to condition the decoder on the query pixel size c, improving rendering for arbitrary resolutions.
• Train an encoder E_phi end-to-end with a self-supervised super-resolution task where the LIIF representation must predict high-resolution ground-truths from down-sampled inputs.

Experimental results

Research questions

• RQ1 Can a continuous, coordinate-based implicit representation recover natural and complex images with high fidelity?
• RQ2 Does a shared decoding function plus local latent codes enable arbitrary high-resolution rendering and extrapolation beyond training scales?
• RQ3 Can LIIF effectively leverage size-varied ground-truths without resizing while maintaining fidelity?
• RQ4 What design choices (feature unfolding, local ensemble, cell decoding, depth of decoder) most influence performance and generalization?
• RQ5 How does LIIF compare to fixed-scale up-sampling methods and image-conditioned implicit models in both in-distribution and out-of-distribution scaling tasks?

Key findings

• LIIF achieves competitive PSNR on in-distribution scales relative to up-sampling baselines and outperforms some baselines on larger, unseen scales.
• LIIF with local ensemble and deep decoding generalizes better to out-of-distribution high-resolution scales (e.g., ×6 to ×30) than some alternatives.
• Cell decoding improves visual quality at larger scales, especially for unseen high-resolution renderings, though PSNR effects can vary."
• Training with random continuous scales (×1 to ×4) enables extrapolation to much higher resolutions (up to ×30) without task-specific ground-truths.
• LIIF naturally handles size-varied ground-truths, outperforming resizing-based up-sampling methods in CelebA-HQ-based image-to-image tasks with different input/output resolutions.