[Paper Review] Learning Federated Representations and Recommendations with Limited Negatives

Lin Ning, Karan Singhal+2 more|arXiv (Cornell University)|Aug 18, 2021

Privacy-Preserving Technologies in Data21 references4 citations

TL;DR

This paper proposes batch-insensitive losses—specifically hinge loss with spreadout regularization and global softmax—to address performance degradation in federated deep retrieval models caused by non-IID and limited negative samples on clients. The method reduces the recall gap between federated and centralized models from 27.22% to 0.53%, improving relative recall by up to 93.15%.

ABSTRACT

Deep retrieval models are widely used for learning entity representations and recommendations. Federated learning provides a privacy-preserving way to train these models without requiring centralization of user data. However, federated deep retrieval models usually perform much worse than their centralized counterparts due to non-IID (independent and identically distributed) training data on clients, an intrinsic property of federated learning that limits negatives available for training. We demonstrate that this issue is distinct from the commonly studied client drift problem. This work proposes batch-insensitive losses as a way to alleviate the non-IID negatives issue for federated movie recommendations. We explore a variety of techniques and identify that batch-insensitive losses can effectively improve the performance of federated deep retrieval models, increasing the relative recall of the federated model by up to 93.15% and reducing the relative gap in recall between it and a centralized model from 27.22% - 43.14% to 0.53% - 2.42%. We also open-source our code framework to accelerate further research and applications of federated deep retrieval models.

Motivation & Objective

• Address the severe performance drop in federated deep retrieval models due to non-IID and scarce negative samples on clients.
• Distinguish the negative sampling issue from client drift, a well-known problem in federated learning.
• Develop loss functions robust to data distribution shifts in federated settings to improve representation learning and recommendation quality.
• Demonstrate that batch-insensitive losses significantly narrow the performance gap between federated and centralized training.
• Release an open-source framework to accelerate research and practical deployment of federated deep retrieval models.

Proposed method

• Introduce batch-insensitive losses that do not rely on in-batch negative samples, reducing sensitivity to client data distribution.
• Combine hinge loss with a spreadout regularizer to push apart embeddings of unrelated item-context pairs, even when negatives are limited.
• Apply global softmax loss using the full item vocabulary as negatives, eliminating dependence on mini-batch negatives.
• Use a dual-encoder architecture with separate encoders for user context and items, trained via contrastive learning.
• Train models using federated averaging with multiple local steps, while evaluating performance on centralized and federated test sets.
• Implement a flexible code framework based on TensorFlow Federated for reproducibility and extensibility to new tasks.

Experimental results

Research questions

• RQ1 To what extent does non-IID data with limited negative samples degrade federated deep retrieval model performance compared to centralized training?
• RQ2 Is the performance degradation primarily due to data distribution shifts in negatives, or is it driven by client drift or other factors?
• RQ3 Can batch-insensitive loss functions mitigate performance degradation in federated deep retrieval models under non-IID data?
• RQ4 How do different loss functions—batch-sensitive vs. batch-insensitive—affect the recall gap between federated and centralized models?
• RQ5 Can batch-insensitive losses be combined effectively with existing client drift mitigation techniques to further improve performance?

Key findings

• The performance gap in recall between federated and centralized models drops from 27.22% (baseline) to 0.53% when using global softmax loss.
• Hinge loss with spreadout regularization reduces the recall gap to 2.42%, improving relative recall by up to 93.15% compared to baseline.
• Batch-insensitive losses like H+S and GS significantly outperform batch-sensitive baselines (BS and BS+S) across all recall metrics (R@1, R@5, R@10).
• Global softmax achieves the smallest performance gap between centralized and federated models, while H+S performs slightly better in absolute recall.
• The improved centralized model with H+S achieves R@10 of 11.15, while the federated model reaches 10.43, indicating strong generalization under federated training.
• The open-source framework enables reproducible training and evaluation of federated deep retrieval models on MovieLens for next-movie prediction.