[Paper Review] Federated Learning with Only Positive Labels

Felix X. Yu, Ankit Singh Rawat+2 more|arXiv (Cornell University)|Apr 21, 2020

Privacy-Preserving Technologies in Data26 references40 citations

TL;DR

The paper introduces FedAwS, a federated learning framework where each client only has positive labels, and a spreadout regularizer is applied on the server to prevent collapsing class embeddings, achieving performance close to standard settings with negative labels.

ABSTRACT

We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class. As a result, during each federated learning round, the users need to locally update the classifier without having access to the features and the model parameters for the negative classes. Thus, naively employing conventional decentralized learning such as the distributed SGD or Federated Averaging may lead to trivial or extremely poor classifiers. In particular, for the embedding based classifiers, all the class embeddings might collapse to a single point. To address this problem, we propose a generic framework for training with only positive labels, namely Federated Averaging with Spreadout (FedAwS), where the server imposes a geometric regularizer after each round to encourage classes to be spreadout in the embedding space. We show, both theoretically and empirically, that FedAwS can almost match the performance of conventional learning where users have access to negative labels. We further extend the proposed method to the settings with large output spaces.

Motivation & Objective

• Motivate and formalize learning a multi-class classifier in federated settings where each client has only positive data for one class.
• Propose FedAwS, a FedAvg-based framework with a geometric spreadout regularizer to keep class embeddings well-separated.
• Theoretically relate FedAwS to conventional full-label training via a cosine contrastive loss and surrogate risk analysis.
• Extend FedAwS to large output spaces and provide empirical validation on standard datasets.

Proposed method

• Embed inputs via g_theta and classify with a class embedding matrix W, where class i has embedding w_i.
• In each round, clients update using only the positive-class data and the positive-part loss l_pos (e.g., l_cl^pos).
• The server applies a spreadout regularizer reg_sp(W) to encourage pairwise separation of class embeddings with margin nu.
• FedAwS updates theta via federated averaging and updates W using the collected client embeddings, followed by a server-side gradient step on reg_sp(W).
• A stochastic negative mining variant (reg_top_sp) adaptively selects close class embeddings to approximate the spreadout effect in large C scenarios.
• Theoretical justification connects FedAwS to cosine contrastive loss and proves surrogate-consistency and near equivalence to full-label empirical risk under certain conditions.

Experimental results

Research questions

• RQ1 Can effective multi-class learning be achieved in federated settings when each client only has positive data for a single class?
• RQ2 Does introducing a spreadout geometric regularizer on the server prevent embedding collapse and enable competitive performance compared to full-label training?
• RQ3 How does FedAwS scale to large output spaces, and can stochastic negative mining help in extreme multiclass settings?
• RQ4 What is the theoretical relationship between FedAwS objectives and conventional cosine/contrastive losses with access to negative labels?

Key findings

• FedAwS prevents collapse of class embeddings and yields competitive accuracy close to standard full-label training in embedding-based classifiers.
• The spreadout regularizer improves stability and separation of class embeddings, improving learning when negatives are unavailable.
• Cosine contrastive loss is established as a surrogate for misclassification error, underpinning the FedAwS objective.
• Under balanced data assumptions (n_i equal) and a specific lambda, FedAwS objective aligns with an empirical risk using a surrogate loss that includes negative components.
• Empirical results on CIFAR-10/100 show FedAwS surpasses two baselines and approaches Softmax (oracle) performance on Precision@1.