[Paper Review] 3D Appearance Super-Resolution with Deep Learning

Yawei Li, Vagia Tsiminaki+3 more|arXiv (Cornell University)|Jun 3, 2019

Advanced Image Processing Techniques48 references4 citations

TL;DR

This paper introduces the 3DASR dataset and a deep learning-based framework for 3D appearance super-resolution, leveraging 2D super-resolution networks enhanced with 3D geometric priors via normal maps. It achieves performance comparable to model-based methods while enabling fast, feed-forward inference, marking the first integration of deep learning into multi-view 3D texture super-resolution with real and synthetic data.

ABSTRACT

We tackle the problem of retrieving high-resolution (HR) texture maps of objects that are captured from multiple view points. In the multi-view case, model-based super-resolution (SR) methods have been recently proved to recover high quality texture maps. On the other hand, the advent of deep learning-based methods has already a significant impact on the problem of video and image SR. Yet, a deep learning-based approach to super-resolve the appearance of 3D objects is still missing. The main limitation of exploiting the power of deep learning techniques in the multi-view case is the lack of data. We introduce a 3D appearance SR (3DASR) dataset based on the existing ETH3D [42], SyB3R [31], MiddleBury, and our Collection of 3D scenes from TUM [21], Fountain [51] and Relief [53]. We provide the high- and low-resolution texture maps, the 3D geometric model, images and projection matrices. We exploit the power of 2D learning-based SR methods and design networks suitable for the 3D multi-view case. We incorporate the geometric information by introducing normal maps and further improve the learning process. Experimental results demonstrate that our proposed networks successfully incorporate the 3D geometric information and super-resolve the texture maps.

Motivation & Objective

• To bridge the gap between deep learning-based 2D super-resolution and 3D multi-view appearance reconstruction by adapting 2D methods to the 3D texture domain.
• To address the lack of large-scale, diverse 3D texture datasets with paired high- and low-resolution maps for training and evaluation.
• To incorporate 3D geometric information—specifically normal maps—into 2D deep learning networks to improve texture super-resolution quality.
• To demonstrate that deep learning-based 3D appearance super-resolution can achieve performance close to computationally heavy model-based methods while enabling fast, feed-forward inference.
• To open a new research direction in 3D content creation by enabling data-driven, scalable, and efficient super-resolution of 3D object appearances.

Proposed method

• The authors construct the 3DASR dataset by combining real and synthetic 3D scenes from ETH3D, SyB3R, MiddleBury, TUM, Fountain, and Relief, providing HR and LR texture maps, 3D meshes, images, projection matrices, and normal maps.
• They model the image formation process as a composition of 3D geometry-to-texture mapping and texture-to-image projection, enabling consistent generation of LR and HR texture pairs from the same underlying HR texture map.
• The proposed framework adapts 2D deep learning super-resolution networks (e.g., EDSR) by incorporating normal maps as additional input channels to encode local 3D surface geometry.
• The method includes a fine-tuning strategy (EDSR-FT) to adapt pre-trained 2D models to the texture domain, improving performance over standard interpolation and baseline networks.
• The framework evaluates multiple variants: EDSR-FT, NLR (low-res normal maps), and NHR (high-res normal maps), to study the impact of geometric priors and data precision.
• The training and evaluation pipeline uses cross-validation on subsets and full-dataset training to assess the influence of data size and subset characteristics.

Experimental results

Research questions

• RQ1 Can deep learning-based super-resolution methods be effectively adapted to the 3D multi-view appearance super-resolution task, given the lack of domain-specific data?
• RQ2 How does incorporating 3D geometric information—via normal maps—improve the quality of super-resolved texture maps compared to standard 2D SR methods?
• RQ3 What is the relative impact of training data size versus subset-specific characteristics on model generalization across diverse 3D scenes?
• RQ4 How does the performance of the proposed deep learning-based approach compare to established model-based super-resolution methods in terms of PSNR and inference speed?
• RQ5 Can high-resolution normal maps further enhance reconstruction quality compared to low-resolution counterparts?

Key findings

• The proposed deep learning-based 3DASR framework achieves PSNR values comparable to model-based methods like HRST, particularly on MiddleBury and Collection datasets, despite being a fast feed-forward network.
• Incorporating normal maps significantly improves reconstruction quality: NHR (high-res normal maps) outperforms NLR (low-res normal maps), demonstrating that geometric detail enhances texture recovery.
• The training data size is a more critical factor than subset-specific characteristics; NLR (full dataset) outperforms NLR-Sub (cross-validated subset), especially on larger datasets like ETH3D.
• Fine-tuning the EDSR model on the texture domain (EDSR-FT) yields better results than using the pre-trained model, indicating domain adaptation is essential for texture-specific performance.
• On ETH3D, the model-based HRST method outperforms the deep learning approach, but the PSNR gap is small, suggesting the deep learning method is competitive.
• Visual results confirm that the proposed method recovers finer texture details and avoids artifacts like blurring and white texels seen in bicubic upsampling and standard EDSR.