[Paper Review] Apple Intelligence Foundation Language Models

Tom Gunter, Z. Wang+153 more|arXiv (Cornell University)|Jul 29, 2024

Advanced Database Systems and Queries4 citations

TL;DR

This paper presents Apple's foundation language models—AFM-on-device (~3B parameters) and AFM-server—designed for on-device and private cloud inference, respectively, with a focus on efficiency, accuracy, and responsible AI. The models leverage optimized Transformer architectures, high-quality curated data, and rigorous red teaming to achieve state-of-the-art safety and performance, outperforming competitors in human preference evaluations on harmful content and helpfulness.

ABSTRACT

We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute. These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly. This report describes the model architecture, the data used to train the model, the training process, how the models are optimized for inference, and the evaluation results. We highlight our focus on Responsible AI and how the principles are applied throughout the model development.

Motivation & Objective

• To develop highly capable, efficient, and privacy-preserving foundation language models for integration into Apple Intelligence features across iOS, iPadOS, and macOS.
• To ensure responsible AI by embedding core principles—user empowerment, representation, design with care, and privacy—throughout model development.
• To optimize models for low-latency, on-device inference while maintaining high performance on diverse tasks like text generation, summarization, and tool use.
• To evaluate model safety rigorously using adversarial prompts and human preference comparisons against competitor models.
• To create a scalable, privacy-first AI system that respects user data and avoids biases through proactive design and data curation.

Proposed method

• Adopting a dense decoder-only Transformer architecture with RMSNorm, pre-normalization, and query/key normalization for training stability.
• Employing grouped-query attention (GQA) with 8 key-value heads to reduce KV-cache memory footprint and improve inference efficiency.
• Using shared input/output embeddings and SwiGLU activations to reduce parameter count and enhance computational efficiency.
• Implementing RoPE positional embeddings with a 500k base frequency to support long-context reasoning.
• Curating a high-quality, diverse training dataset from licensed publisher content, open-source datasets, and Applebot-crawled web pages, with strict filtering for PII, profanity, and unsafe content.
• Applying automated prompt generation and human red teaming with safety taxonomies to identify and mitigate model vulnerabilities.

Experimental results

Research questions

• RQ1 How can foundation language models be architected to achieve high performance while remaining efficient and power-optimized for on-device inference?
• RQ2 What data curation and filtering strategies are most effective in ensuring high-quality, safe, and privacy-preserving pre-training data?
• RQ3 To what extent do responsible AI principles, including bias mitigation and privacy protection, improve model safety and user trust?
• RQ4 How do the models perform in human preference evaluations compared to commercial and open-source models on safety and helpfulness?
• RQ5 Can automated and human red teaming processes effectively identify and reduce model vulnerabilities to adversarial prompts?

Key findings

• AFM-on-device and AFM-server achieve significantly lower violation rates on adversarial safety prompts than open-source and commercial models, demonstrating robustness to harmful content.
• Human graders preferred responses from Apple’s models over competitor models in 85% of side-by-side evaluations, citing both safety and helpfulness.
• The models maintain high performance across diverse tasks, including text generation, summarization, and tool use, with the on-device model achieving ~2.58B non-embedding parameters and efficient inference on Apple silicon.
• Rigorous data decontamination and filtering reduced exposure to PII, profanity, and unsafe content, with no private user data used in training.
• Red teaming with internal and external crowds, supported by health resources and time limits, enabled comprehensive probing of safety risks without compromising well-being.
• The use of automated prompt generation for data collection proved effective in identifying model gaps and improving evaluation set coverage.