[Paper Review] Open Vocabulary Multi-Label Classification with Dual-Modal Decoder on Aligned Visual-Textual Features

Shichao Xu, Yikang Li+3 more|arXiv (Cornell University)|Aug 19, 2022

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes ADDS, a novel open-vocabulary multi-label classification framework that uses a Dual-Modal Decoder (DM-Decoder) to fuse aligned visual and textual features, enabling state-of-the-art performance on open-vocabulary, single-to-multi, and conventional multi-label classification. The method achieves SOTA results by leveraging visual-textual alignment, Pyramid-Forwarding for high-resolution adaptation, and Selective Language Supervision, outperforming prior methods on NUS-WIDE, ImageNet-1k, ImageNet-21k, and MS-COCO benchmarks.

ABSTRACT

In computer vision, multi-label recognition are important tasks with many real-world applications, but classifying previously unseen labels remains a significant challenge. In this paper, we propose a novel algorithm, Aligned Dual moDality ClaSsifier (ADDS), which includes a Dual-Modal decoder (DM-decoder) with alignment between visual and textual features, for open-vocabulary multi-label classification tasks. Then we design a simple and yet effective method called Pyramid-Forwarding to enhance the performance for inputs with high resolutions. Moreover, the Selective Language Supervision is applied to further enhance the model performance. Extensive experiments conducted on several standard benchmarks, NUS-WIDE, ImageNet-1k, ImageNet-21k, and MS-COCO, demonstrate that our approach significantly outperforms previous methods and provides state-of-the-art performance for open-vocabulary multi-label classification, conventional multi-label classification and an extreme case called single-to-multi label classification where models trained on single-label datasets (ImageNet-1k, ImageNet-21k) are tested on multi-label ones (MS-COCO and NUS-WIDE).

Motivation & Objective

• To address the challenge of classifying previously unseen labels in multi-label image recognition, especially in open-vocabulary settings where test labels are not seen during training.
• To improve generalization beyond seen labels by leveraging pre-trained vision-language models with aligned visual and textual embeddings.
• To enable effective multi-label classification on high-resolution images without retraining, using a resolution-adaptive inference mechanism.
• To enhance model performance through selective language supervision, particularly in low-resource or single-to-multi label transfer scenarios.

Proposed method

• The core of the method is the Dual-Modal Decoder (DM-Decoder), a transformer-based decoder that progressively fuses visual features with textual embeddings to improve semantic understanding.
• Visual and textual features are aligned using a contrastive learning objective during pretraining, ensuring that semantically similar image-text pairs are close in the shared embedding space.
• Pyramid-Forwarding enables high-resolution inference by processing feature maps at multiple scales, preserving spatial details and improving performance on large inputs.
• Selective Language Supervision enhances training by focusing on relevant labels during fine-tuning, improving generalization to unseen labels.
• The framework is trained on ImageNet-1k and ImageNet-21k and evaluated on NUS-WIDE and MS-COCO, with zero-shot transfer to multi-label and open-vocabulary settings.
• The model uses ViT-L-336 as the image encoder and is fine-tuned with a cross-entropy loss on label embeddings derived from pre-trained text encoders.

Experimental results

Research questions

• RQ1 Can a dual-modality decoder that fuses visual and textual features improve zero-shot and open-vocabulary multi-label classification beyond existing methods?
• RQ2 How effective is Pyramid-Forwarding in enabling high-resolution inference without retraining for models pre-trained at lower resolutions?
• RQ3 Does increasing the number of training classes via selective language supervision improve performance in single-to-multi label classification?
• RQ4 To what extent does visual-textual alignment, rather than the image or text encoder alone, drive performance gains in open-vocabulary settings?
• RQ5 How does the proposed method compare to SOTA models like ML-Decoder and Q2L across diverse benchmarks including NUS-WIDE, ImageNet-1k, ImageNet-21k, and MS-COCO?

Key findings

• ADDS achieves a 39.28% mAP on NUS-WIDE for open-vocabulary multi-label classification using the DM-Decoder with ViT-L-336, outperforming previous SOTA methods.
• The DM-Decoder improves mAP by 2.5 points over the ML-Decoder baseline, with a 38.68% mAP when using three stacked decoder layers.
• Pyramid-Forwarding boosts mAP from 91.76% to 93.54% on MS-COCO at 1344×1344 resolution, demonstrating its effectiveness in high-resolution inference.
• Training on 15,000 classes from ImageNet-21k instead of 1,000 classes from ImageNet-1k increases mAP from 31.02% to 37.92% on NUS-WIDE, showing the benefit of more training classes.
• Models without visual-textual alignment, such as a fine-tuned BLIP model without contrastive loss, drop to 2.52% mAP, proving that alignment is critical for performance.
• The method achieves SOTA results across all three tasks: open-vocabulary multi-label, conventional multi-label, and single-to-multi label classification, demonstrating broad applicability.