[Paper Review] FLAIR: Federated Learning Annotated Image Repository

Congzheng Song, Filip Granqvist+1 more|arXiv (Cornell University)|Jul 18, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

FLAIR is a large-scale, multi-label image dataset with 429,078 images from 51,414 real Flickr users, designed to benchmark federated learning under realistic conditions such as long-tailed label distributions, user data heterogeneity, and feature skew. The paper introduces reproducible baselines for centralized, federated, and differentially private learning, demonstrating that model performance degrades significantly on rare classes under federated and private settings due to data imbalance and noise.

ABSTRACT

Cross-device federated learning is an emerging machine learning (ML) paradigm where a large population of devices collectively train an ML model while the data remains on the devices. This research field has a unique set of practical challenges, and to systematically make advances, new datasets curated to be compatible with this paradigm are needed. Existing federated learning benchmarks in the image domain do not accurately capture the scale and heterogeneity of many real-world use cases. We introduce FLAIR, a challenging large-scale annotated image dataset for multi-label classification suitable for federated learning. FLAIR has 429,078 images from 51,414 Flickr users and captures many of the intricacies typically encountered in federated learning, such as heterogeneous user data and a long-tailed label distribution. We implement multiple baselines in different learning setups for different tasks on this dataset. We believe FLAIR can serve as a challenging benchmark for advancing the state-of-the art in federated learning. Dataset access and the code for the benchmark are available at \url{https://github.com/apple/ml-flair}.

Motivation & Objective

• Address the lack of realistic, large-scale image datasets tailored for federated learning by creating a dataset that reflects real-world data heterogeneity and scale.
• Provide a benchmark for evaluating federated learning models under practical challenges such as non-IID data, long-tailed label distributions, and differential privacy.
• Enable systematic evaluation of privacy-utility-efficiency trade-offs in federated learning through standardized metrics and training setups.
• Support research into class imbalance, few-shot learning, and noise-robust training in distributed settings with real-world data characteristics.

Proposed method

• Curated 429,078 images from 51,414 real Flickr users, ensuring diverse user-level data distributions and realistic user partitioning.
• Annotated images with a two-level label hierarchy: 17 coarse-grained classes and 1,628 fine-grained classes, enabling multi-level benchmarking.
• Implemented multiple training setups: centralized learning, standard federated learning, and federated learning with central differential privacy.
• Applied differential privacy to model updates during federated training to evaluate privacy-utility trade-offs.
• Used ResNet18 as the base model architecture and evaluated performance using mean average precision (mAP) on validation and test sets.
• Provided open-source code and dataset access via GitHub to ensure reproducibility and community adoption.

Experimental results

Research questions

• RQ1 How does model performance vary across different learning paradigms—centralized, federated, and differentially private—on a realistic, heterogeneous image dataset?
• RQ2 To what extent does label imbalance and data heterogeneity in user-partitioned data degrade model performance in federated learning?
• RQ3 How does differential privacy affect model accuracy, especially on rare and fine-grained classes in federated learning?
• RQ4 Can pre-trained models (e.g., ImageNet) improve performance on long-tailed, fine-grained classification tasks in federated settings?
• RQ5 How do cohort size and noise scale impact model convergence and accuracy in differentially private federated learning on this dataset?

Key findings

• Model performance on the fine-grained 1,628-class taxonomy was significantly lower than on the coarse 17-class taxonomy, with 11 classes in validation/test sets being completely unseen during training.
• 134 of the 1,628 fine-grained classes had fewer than 20 positive examples in the training set, making them challenging for both centralized and federated learning.
• Differential privacy exacerbated performance degradation on rare classes due to poor signal-to-noise ratio in sparse gradients, especially in federated settings.
• Larger cohort sizes in differentially private federated learning reduced noise and improved model accuracy, but increased training latency.
• Performance on low-frequency classes was worse in federated learning than in centralized learning, highlighting the compounding effect of data heterogeneity and privacy mechanisms.
• The long-tailed label distribution in FLAIR makes it a suitable benchmark for advancing research in few-shot and zero-shot federated learning.