[Paper Review] OvarNet: Towards Open-vocabulary Object Attribute Recognition

Keyan Chen, Xiaolong Jiang+5 more|arXiv (Cornell University)|Jan 23, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

OvarNet proposes a unified, end-to-end framework for open-vocabulary object detection and attribute recognition, leveraging CLIP-based vision-language pretraining with knowledge distillation into a Faster R-CNN backbone. It achieves state-of-the-art performance across VAW, MS-COCO, LSA, and OVAD benchmarks by jointly training detection and attribute prediction, demonstrating strong zero-shot generalization to unseen categories and attributes.

ABSTRACT

In this paper, we consider the problem of simultaneously detecting objects and inferring their visual attributes in an image, even for those with no manual annotations provided at the training stage, resembling an open-vocabulary scenario. To achieve this goal, we make the following contributions: (i) we start with a naive two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes; (ii) we combine all available datasets and train with a federated strategy to finetune the CLIP model, aligning the visual representation with attributes, additionally, we investigate the efficacy of leveraging freely available online image-caption pairs under weakly supervised learning; (iii) in pursuit of efficiency, we train a Faster-RCNN type model end-to-end with knowledge distillation, that performs class-agnostic object proposals and classification on semantic categories and attributes with classifiers generated from a text encoder; Finally, (iv) we conduct extensive experiments on VAW, MS-COCO, LSA, and OVAD datasets, and show that recognition of semantic category and attributes is complementary for visual scene understanding, i.e., jointly training object detection and attributes prediction largely outperform existing approaches that treat the two tasks independently, demonstrating strong generalization ability to novel attributes and categories.

Motivation & Objective

• To address the challenge of simultaneously detecting objects and recognizing their visual attributes in an open-vocabulary setting, where novel categories and attributes are unseen during training.
• To overcome the bias of CLIP models toward semantic categories by improving visual-attribute feature alignment through learnable prompt vectors and weakly supervised data.
• To develop an efficient, end-to-end detection framework that jointly predicts object categories and attributes without requiring manual annotations for all attributes.
• To demonstrate that joint training of detection and attribute recognition improves generalization and performance on novel attributes and categories.

Proposed method

• A two-stage baseline, CLIP-Attr, first generates object proposals via an offline RPN and then classifies them using CLIP by comparing visual features with text embeddings of attributes and categories.
• Learnable prompt vectors are introduced on the textual encoder side to better align attribute concepts with visual features, improving zero-shot attribute recognition.
• A federated fine-tuning strategy combines object detection and attribute prediction datasets to align CLIP’s visual representations with attribute semantics.
• Weakly supervised learning is applied using freely available image-caption pairs to enhance robustness for recognizing novel attributes.
• Knowledge distillation is used to train a Faster R-CNN-style model (OvarNet) end-to-end, enabling class-agnostic proposals and joint classification using text-encoder-generated classifiers.
• The final OvarNet model integrates detection and attribute prediction in a single, efficient architecture, trained with distilled knowledge from CLIP-Attr and external image-caption data.

Figure 1 : The first row depicts the tasks of object detection and attribute classification in a close-set setting, i.e. , train and test on the same vocabulary set. The second row gives qualitative results from our proposed OvarNet, which simultaneously localizes, categorizes, and characterizes arb

Experimental results

Research questions

• RQ1 Can joint training of object detection and attribute recognition improve zero-shot generalization to unseen categories and attributes?
• RQ2 How can CLIP-based models be adapted to reduce bias toward semantic categories and improve alignment with visual attributes?
• RQ3 To what extent can weakly supervised image-caption pairs improve attribute recognition robustness in open-vocabulary settings?
• RQ4 Does end-to-end knowledge distillation from a two-stage CLIP-Attr model into a Faster R-CNN backbone yield better efficiency and performance?
• RQ5 How does the proposed OvarNet framework generalize across diverse benchmarks, including those with long-tail attribute distributions?

Key findings

• OvarNet achieves 28.6 mAP on the VAW benchmark under the 'given' setting, significantly outperforming previous SOTA methods such as X-VLM (28.1) and OVAD (21.4).
• On the OVAD benchmark, OvarNet achieves 35.5 mAP for all attributes, surpassing CLIP-Attr (29.3) and other baselines, with a notable 9.5 mAP gain on tail attributes.
• In cross-dataset transfer evaluation on OVAD, OvarNet achieves 33.6 mAP under the 'free' setting, outperforming OV-Faster-RCNN (18.3) and Detic (13.4).
• On the LSA benchmark, OvarNet achieves 14.84 mAP on base attributes and 8.05 mAP on novel attributes when using category prior, outperforming OpenTAP (13.59 and 7.62) under the same setting.
• Even without category priors, OvarNet achieves 8.52 mAP on LSA common and 6.17 mAP on rare attributes, demonstrating strong zero-shot generalization.
• The ablation study confirms that knowledge distillation from CLIP-Attr into OvarNet improves performance, especially on rare and novel attributes, validating the effectiveness of the unified training paradigm.

Figure 2 : An overview of the proposed method. Left: the two-step training procedure for finetuning the pre-trained CLIP to get CLIP-Attr that better aligns the regional visual feature to attributes. Step-I: naïve federate training by base attribute annotations. Step-II: training by image-caption pa