[Paper Review] Gmail Smart Compose: Real-Time Assisted Writing

Mia Xu Chen, Benjamin N. Lee+10 more|arXiv (Cornell University)|May 17, 2019

Topic Modeling46 references45 citations

TL;DR

Smart Compose provides real-time, context-aware writing suggestions in Gmail using large-scale neural language models, with production deployment, personalization, multilingual support, and privacy considerations. It balances quality with strict latency constraints for billions of users.

ABSTRACT

In this paper, we present Smart Compose, a novel system for generating interactive, real-time suggestions in Gmail that assists users in writing mails by reducing repetitive typing. In the design and deployment of such a large-scale and complicated system, we faced several challenges including model selection, performance evaluation, serving and other practical issues. At the core of Smart Compose is a large-scale neural language model. We leveraged state-of-the-art machine learning techniques for language model training which enabled high-quality suggestion prediction, and constructed novel serving infrastructure for high-throughput and real-time inference. Experimental results show the effectiveness of our proposed system design and deployment approach. This system is currently being served in Gmail.

Motivation & Objective

• Motivate and enable faster, more accurate email drafting by reducing repetitive typing at scale.
• Address challenges of latency, scale, personalization, fairness, and privacy in a production neural language model system.
• Evaluate model architectures and serving strategies to achieve production-ready performance.
• Explore personalization and multilingual extensions to improve user experience across diverse users.

Proposed method

• Train and compare neural language models (RNN/LSTM and Transformer) using large English email corpora.
• Incorporate contextual features (subject, previous email, date/time, locale) via embeddings and context encoders.
• Formulate prediction as language modeling or seq2seq with attention; use beam search for top-n suggestions.
• Evaluate using Log Perplexity and ExactMatch@N to correlate offline metrics with online performance.
• Deploy production system with streaming RPC server, latency goals (90th percentile < 60 ms), and TPU-based acceleration.

Experimental results

Research questions

• RQ1 How do different model architectures (LM-A, LM-B, Seq2Seq) perform for real-time email composition tasks?
• RQ2 What are the latency and throughput requirements and how can they be met at Gmail scale?
• RQ3 Does personalization improve suggestion quality without compromising privacy and efficiency?
• RQ4 Can multilingual models provide competitive quality while maintaining production feasibility across languages?

Key findings

• Contextual features improve language model perplexity compared to context-free baselines.
• Transformer models offer better perplexity at similar capacity, but latency penalties make LM-A more production-suited. For LM-A, Transformer-1536-8192 achieves 2.90 test perplexity and 67.73% Overall ExactMatch, though with higher latency.
• Production deployment prioritizes latency; LM-A on CPU/TPU shows substantially better throughput and lower relative latency with Cloud TPU accelerators.
• Personalized n-gram models with linear interpolation to global models yield relative gains in ExactMatch and CTR (~6% CTR, ~10% ExactMatch in production).
• Multilingual wordpiece models are effective; in Spanish, French, Italian, Portuguese, monolingual word models tend to outperform multilingual variants in ExactMatch, except Portuguese where multilingual helps; overall deployment used multilingual wordpiece with language-specific thresholds.
• The system achieves practical real-time inference with end-to-end latency targets and scales to billions of users.