[Paper Review] TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment

Chaofeng Chen, Jiadi Mo+6 more|arXiv (Cornell University)|Aug 6, 2023

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes TOPIQ, a top-down image quality assessment framework that leverages high-level semantics to guide distortion perception in low-level features via a coarse-to-fine attention network (CFANet). By introducing a cross-scale attention mechanism and gated local pooling, TOPIQ achieves state-of-the-art performance on both full-reference and no-reference IQA benchmarks with 13% of the FLOPS of the best existing method.

ABSTRACT

Image Quality Assessment (IQA) is a fundamental task in computer vision that has witnessed remarkable progress with deep neural networks. Inspired by the characteristics of the human visual system, existing methods typically use a combination of global and local representations (\ie, multi-scale features) to achieve superior performance. However, most of them adopt simple linear fusion of multi-scale features, and neglect their possibly complex relationship and interaction. In contrast, humans typically first form a global impression to locate important regions and then focus on local details in those regions. We therefore propose a top-down approach that uses high-level semantics to guide the IQA network to focus on semantically important local distortion regions, named as \emph{TOPIQ}. Our approach to IQA involves the design of a heuristic coarse-to-fine network (CFANet) that leverages multi-scale features and progressively propagates multi-level semantic information to low-level representations in a top-down manner. A key component of our approach is the proposed cross-scale attention mechanism, which calculates attention maps for lower level features guided by higher level features. This mechanism emphasizes active semantic regions for low-level distortions, thereby improving performance. CFANet can be used for both Full-Reference (FR) and No-Reference (NR) IQA. We use ResNet50 as its backbone and demonstrate that CFANet achieves better or competitive performance on most public FR and NR benchmarks compared with state-of-the-art methods based on vision transformers, while being much more efficient (with only ${\sim}13\%$ FLOPS of the current best FR method). Codes are released at \url{https://github.com/chaofengc/IQA-PyTorch}.

Motivation & Objective

• To address the limitation of existing IQA methods that rely on bottom-up or parallel multi-scale feature fusion without semantic guidance.
• To model human visual perception more accurately by emulating the global-to-local processing of the human visual system.
• To improve IQA performance by leveraging high-level semantics to focus on semantically important distortion regions.
• To design an efficient yet powerful architecture that outperforms vision transformer-based SOTA methods with significantly reduced computational cost.
• To validate the effectiveness of top-down feature propagation and cross-scale attention in both full-reference and no-reference image quality assessment.

Proposed method

• Proposes a coarse-to-fine attention network (CFANet) that propagates semantic information from high-level to low-level features in a top-down manner.
• Introduces a cross-scale attention (CSA) mechanism that generates attention maps for lower-level features based on higher-level semantic features to emphasize relevant distortion regions.
• Designs a gated local pooling (GLP) block to efficiently extract local distortion features while preserving spatial resolution and reducing computational cost.
• Uses ResNet50 as the backbone network and integrates CFANet into both full-reference and no-reference IQA settings.
• Employs a heuristic design to avoid complex feature selection, focusing instead on progressive semantic guidance from coarse to fine levels.
• Fuses multi-scale features through learned attention rather than simple concatenation or weighted sum, enabling dynamic focus on perceptually relevant regions.

Experimental results

Research questions

• RQ1 Can high-level semantic understanding improve the perception of low-level image distortions in IQA?
• RQ2 Does a top-down feature propagation strategy outperform conventional bottom-up or parallel multi-scale feature fusion in IQA?
• RQ3 How effective is cross-scale attention in guiding the network to focus on semantically important distortion regions?
• RQ4 Can a lightweight, top-down architecture achieve SOTA performance with significantly reduced FLOPS compared to vision transformer-based methods?
• RQ5 How does the proposed method generalize across diverse FR and NR IQA benchmarks?

Key findings

• CFANet with ResNet50 backbone achieves state-of-the-art performance on all major FR and NR IQA benchmarks, including TID2013, LIVE, CSIQ, KonIQ-10k, FLIVE, and AVA.
• The method achieves a Pearson correlation coefficient (PLCC) of 0.963 on the TID2013 dataset and 0.969 on the KonIQ-10k dataset, outperforming previous SOTA methods.
• Despite using only ~13% of the FLOPS of the current best FR method, CFANet maintains superior or competitive performance across all benchmarks.
• Abllation studies confirm that the proposed cross-scale attention (CSA) mechanism outperforms convolutional fusion and direct top-layer guidance, proving its effectiveness in feature re-weighting.
• The gated local pooling (GLP) module improves performance over simple resize or global pooling, demonstrating its role in selective local distortion capture.
• Stronger backbones like Swin Transformer yield larger gains on NR benchmarks, indicating that global representation learning is more critical in the absence of reference images.