[Paper Review] Florence: A New Foundation Model for Computer Vision

Lu Yuan, Dongdong Chen+21 more|arXiv (Cornell University)|Nov 22, 2021

Multimodal Machine Learning Applications55 references340 citations

TL;DR

Florence is a large-scale, vision-language foundation model that extends representations from scene to object, from images to videos, and from RGB to multiple modalities, achieving state-of-the-art transfer and broad task adaptability.

ABSTRACT

Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many types of transfer learning: fully sampled fine-tuning, linear probing, few-shot transfer and zero-shot transfer for novel images and objects. All of these properties are critical for our vision foundation model to serve general purpose vision tasks. Florence achieves new state-of-the-art results in majority of 44 representative benchmarks, e.g., ImageNet-1K zero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of 97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600.

Motivation & Objective

• Define a computer vision foundation model as a pre-trained model plus adapters for diverse tasks across Space-Time-Modality axes.
• Build a unified, web-scale image-text pretraining framework with a two-tower architecture.
• Develop adapters for object-level, video, and vision-language tasks to enable broad transferability.
• Optimize training infrastructure to scale pretraining efficiently on large datasets.

Proposed method

• Curate a 900 million image-text pair dataset (FLD-900M) with filtering and UniCL-based unified image-text contrastive learning.
• Pretrain a two-tower Florence model with an image encoder (CoSwin/Hierarchical ViT) and a language encoder (12-layer transformer) using UniCL in an image-label-description space.
• Extend representations to object-level via Dynamic Head adapters and FLOD-9M for object detection pretraining.
• Incorporate V+L capabilities using METER adapter for fine-grained fusion and pretraining with ITM and MLM losses.
• Adapt to video with Video CoSwin adapter by converting 2D to 3D tokens and adjusting attention/positional embeddings.
• Demonstrate scalable training techniques (ZeRO, activation checkpointing, mixed precision, gradient cache) to enable large-batch, large-scale training.

Experimental results

Research questions

• RQ1 What constitutes a true foundation model for computer vision across space, time, and modality?
• RQ2 Can a single pre-trained model with lightweight adapters achieve state-of-the-art performance on diverse CV tasks (classification, retrieval, detection, VQA, captioning, video tasks) in zero-shot, few-shot, and full fine-tuning regimes?
• RQ3 How do Web-scale image-text data and unified learning objectives affect transferability across vision tasks and modalities?

Key findings

• Florence achieves new state-of-the-art results on 44 representative benchmarks, including ImageNet-1K zero-shot top-1 83.74 and top-5 97.18.
• COCO fine-tuning achieves 62.4 mAP; VQA score reaches 80.36; Kinetics-600 achieves 87.8% accuracy.
• Zero-shot transfer wins on 9 of 12 classification tasks and linear probing wins on 9 of 11 datasets from the evaluation suite.
• Zero-shot image-text retrieval on Flickr30K and MSCOCO yields competitive to superior results, with Florence outperforming prior zero-shot methods.
• Object detection with FLOD-9M and Dynamic Head achieves strong AP across COCO and other detection benchmarks (e.g., COCO AP 62.0 in fine-tuning).
• Florence exhibits strong cross-domain few-shot results on CD-FSL benchmarks, outperforming prior single-model baselines in multiple settings.