[Paper Review] Federated Learning Of Out-Of-Vocabulary Words

Chen, Mingqing, Rajiv Mathews+2 more|arXiv (Cornell University)|Mar 26, 2019

Speech Recognition and SynthesisComputer Science19 references104 citations

TL;DR

The paper demonstrates that a character-level LSTM learned via federated learning can generate and rank OOV words on-device, achieving high precision/recall in simulated FL and meaningful on-device OOV word learning.

ABSTRACT

We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers. High-frequency words can be sampled from the trained generative model by drawing from the joint posterior directly. We study the feasibility of the approach in two settings: (1) using simulated federated learning on a publicly available non-IID per-user dataset from a popular social networking website, (2) using federated learning on data hosted on user mobile devices. The model achieves good recall and precision compared to ground-truth OOV words in setting (1). With (2) we demonstrate the practicality of this approach by showing that we can learn meaningful OOV words with good character-level prediction accuracy and cross entropy loss.

Motivation & Objective

• Motivate expanding mobile keyboard vocabularies by learning OOV words without sending sensitive text to servers.
• Show feasibility of on-device federated learning to learn OOV words using a character-based RNN.
• Evaluate performance in simulated FL on public data and in real on-device FL settings across multiple languages.
• Demonstrate that sampled OOV words are meaningful and reflect real-world trends.

Proposed method

• Train a multi-layer LSTM with CIFG, peephole connections, and a projection layer on device.”],
• method_note_1_2
• 3-6
• 1
• 2
• 3
• 4
• 5
• 6

Experimental results

Research questions

• RQ1 Can a federated learning setup learn useful OOV words without transferring raw user text to servers?
• RQ2 How well does the generated distribution match true word frequencies for OOV words?
• RQ3 Does on-device FL converge to accurate character-level predictions and acceptable cross-entropy loss across languages?

Key findings

• In simulated FL on Reddit data, the model achieves 90.56% precision and 81.22% recall for top 10^5 unique words.
• On-device FL shows 55.8% top-3 character-level prediction accuracy and 2.35 cross-entropy in en_US, pt_BR, and in_ID.
• The approach learns meaningful OOV words, including trending words, across three languages.
• Adaptive gradient clipping and momentum improve convergence over SGD baselines.
• The top-sampled OOV words align with ground-truth tendencies and include slang, abbreviations, and names.