[Paper Review] Implicit Transformer Network for Screen Content Image Continuous Super-Resolution

Jingyu Yang, Sheng Shen+2 more|arXiv (Cornell University)|Dec 12, 2021

Advanced Image Processing TechniquesComputer Science25 references32 citations

TL;DR

It proposes ITSRN, an implicit transformer-based method for continuous-scale super-resolution of screen content images, outperforming prior continuous and discrete SR methods especially on sharp, high-contrast content.

ABSTRACT

Nowadays, there is an explosive growth of screen contents due to the wide application of screen sharing, remote cooperation, and online education. To match the limited terminal bandwidth, high-resolution (HR) screen contents may be downsampled and compressed. At the receiver side, the super-resolution (SR) of low-resolution (LR) screen content images (SCIs) is highly demanded by the HR display or by the users to zoom in for detail observation. However, image SR methods mostly designed for natural images do not generalize well for SCIs due to the very different image characteristics as well as the requirement of SCI browsing at arbitrary scales. To this end, we propose a novel Implicit Transformer Super-Resolution Network (ITSRN) for SCISR. For high-quality continuous SR at arbitrary ratios, pixel values at query coordinates are inferred from image features at key coordinates by the proposed implicit transformer and an implicit position encoding scheme is proposed to aggregate similar neighboring pixel values to the query one. We construct benchmark SCI1K and SCI1K-compression datasets with LR and HR SCI pairs. Extensive experiments show that the proposed ITSRN significantly outperforms several competitive continuous and discrete SR methods for both compressed and uncompressed SCIs.

Motivation & Objective

• Motivate high-quality SCI SR to handle arbitrary magnification factors for screen contents dominated by text and graphics.
• Address poor generalization of natural-image SR methods to SCI due to sharp edges and high contrast.
• Develop a continuous-scale SR framework that leverages coordinates (not just pixel values) to render high-fidelity SCI results.
• Create SCI1K and SCI1K-compression benchmarks to evaluate SCI SR methods under different degradation.

Proposed method

• Introduce implicit transformer to map query coordinates in HR space to key coordinates and features in LR space for pixel value generation.
• Predict transform weights via a nonlinear mapping of coordinate differences and a scale token, enabling coordinate-to-value learning.
• Augment the model with implicit position encoding to aggregate similar neighboring pixel values and refine predictions.
• Use a CNN backbone (RDN-based) to extract features, followed by feature unfolding to enlarge receptive fields, and an MLP to compute the transform weights.
• Normalize coordinates to a [-1,1] range to stabilize learning and enable continuous magnification.
• Incorporate a local 3x3 neighborhood encoding (implicit position encoding) learned by an MLP to refine the coarse SR output.

Experimental results

Research questions

• RQ1 Can an implicit transformer operating on coordinates outperform existing continuous SR methods (LIIF, MetaSR) on screen content images?
• RQ2 Does introducing an implicit position encoding improve edge preservation and edge sharpness for SCI SR across arbitrary scales?
• RQ3 How do scale tokens and IPE affect performance under both in-training and out-of-training magnification factors?
• RQ4 What is the impact of using learnable vs fixed neighbor weighting in implicit position encoding for compressed and uncompressed SCI data?

Key findings

• ITSRN consistently outperforms continuous SR methods (MetaSR, LIIF) and discrete SR baselines (RDN, RCAN) on SCI1K and SCI1K-compression across multiple scales.
• Ablation shows scale token and implicit position encoding contribute substantial gains (about 0.39 dB PSNR at x4; 0.06 dB at out-of-training scales).
• Learnable weights in implicit position encoding outperform fixed distances, especially on JPEG-compressed inputs.
• Training with a wide range of scales improves performance at larger magnification ratios (x6–x10).
• ITSRN achieves better edge reconstruction for thin characters, particularly at high magnification, compared to competing methods.