[Paper Review] EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature Refinement and Regularized Image-Text Alignment

Mykola Lavreniuk, Shariq Farooq Bhat+2 more|arXiv (Cornell University)|Dec 13, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes EVP, a novel vision model that enhances visual perception through inverse multi-attentive feature refinement (IMAFR) and regularized image-text alignment (RITA), leveraging the Stable Diffusion U-Net backbone. EVP achieves state-of-the-art performance in single-image depth estimation (11.8% RMSE reduction on NYU Depth v2 and new SOTA on KITTI) and referring segmentation (2.53 IoU improvement on RefCOCO).

ABSTRACT

This work presents the network architecture EVP (Enhanced Visual Perception). EVP builds on the previous work VPD which paved the way to use the Stable Diffusion network for computer vision tasks. We propose two major enhancements. First, we develop the Inverse Multi-Attentive Feature Refinement (IMAFR) module which enhances feature learning capabilities by aggregating spatial information from higher pyramid levels. Second, we propose a novel image-text alignment module for improved feature extraction of the Stable Diffusion backbone. The resulting architecture is suitable for a wide variety of tasks and we demonstrate its performance in the context of single-image depth estimation with a specialized decoder using classification-based bins and referring segmentation with an off-the-shelf decoder. Comprehensive experiments conducted on established datasets show that EVP achieves state-of-the-art results in single-image depth estimation for indoor (NYU Depth v2, 11.8% RMSE improvement over VPD) and outdoor (KITTI) environments, as well as referring segmentation (RefCOCO, 2.53 IoU improvement over ReLA). The code and pre-trained models are publicly available at https://github.com/Lavreniuk/EVP.

Motivation & Objective

• To improve metric depth estimation from single images by enhancing feature learning and image-text alignment beyond prior methods.
• To address the limitations of VPD, particularly in feature alignment and boundary sharpness, by introducing new architectural components.
• To enable robust zero-shot and few-shot generalization in vision tasks using free-form, vision-language-generated image descriptions instead of rigid class templates.
• To demonstrate the effectiveness of the proposed architecture across diverse vision tasks, including depth estimation and referring segmentation.
• To provide a publicly available, high-performance model that leverages pre-trained diffusion features with minimal fine-tuning.

Proposed method

• Proposes the Inverse Multi-Attentive Feature Refinement (IMAFR) module, which aggregates features across multiple pyramid levels using multi-head attention to enhance spatial and semantic representation.
• Introduces a Regularized Image-Text Alignment (RITA) module that uses free-form captions from BLIP-2 to generate unified, enriched text embeddings, improving cross-attention alignment with image features.
• Employs a classification-based decoder with metric bins inspired by ZoeDepth, enabling more accurate depth prediction by learning discrete depth distributions.
• Utilizes the pre-trained Stable Diffusion U-Net as the visual encoder, leveraging its rich multi-modal features from large-scale image-text pre-training.
• Applies a multi-scale feature refinement strategy that operates across the U-Net’s skip connections, enhancing feature refinement at all scales.
• Fine-tunes CLIP text encoder weights during training to improve alignment between image and text representations, especially when using free-form descriptions.

Experimental results

Research questions

• RQ1 Can inverse multi-attentive feature aggregation improve feature representation in single-image depth estimation compared to hierarchical or fixed aggregation?
• RQ2 Does using free-form, vision-language-generated captions instead of template-based class descriptions lead to better image-text alignment and downstream performance?
• RQ3 Can a classification-based depth decoder with metric bins significantly improve depth estimation accuracy when combined with a pre-trained diffusion backbone?
• RQ4 To what extent does the proposed architecture generalize across diverse vision tasks, including depth estimation and referring segmentation?
• RQ5 How does the integration of IMAFR and RITA modules affect model performance compared to the baseline VPD architecture?

Key findings

• EVP achieves a 11.8% relative reduction in RMSE (from 0.254 to 0.224) on the NYU Depth v2 benchmark, setting a new state-of-the-art for indoor monocular depth estimation.
• On the KITTI dataset, EVP outperforms the previous SOTA method GEDepth across all 7 evaluation metrics, establishing a new SOTA for outdoor depth estimation.
• For referring segmentation on RefCOCO, EVP achieves an IoU of 76.35, representing a 2.53-point improvement over the previous SOTA method ReLA.
• The ablation study confirms that the combination of IMAFR, metric bins, and individual CLIP embeddings (per-image) yields the best performance, with the individual CLIP vector strategy (row 12) being optimal.
• Using frozen CLIP weights during training leads to suboptimal performance, while fine-tuning CLIP significantly improves alignment and results, especially with free-form captions.
• The model’s performance is most sensitive to the image-text alignment strategy, with free-form captions from BLIP-2 outperforming template-based descriptions, even without explicit class labels.