Apple WWDC On-Device AI Deep Dive - Google Docs · Gist Share

Apple's

Full transcript (Instant)

Gist

1. Apple's 20-billion-parameter AI model now runs on an iPhone by patching in only 1 to 4 billion weights at a time from NAND flash — turning a memory-bandwidth wall into an I/O scheduling problem. WWDC 2026 wasn't an AI announcement. It was a declaration that the operating system itself is now a hypervisor for large language models, and the developer who controls the hypervisor controls the ecosystem.

Logic

2. The OS is now a hypervisor for large language models

iOS 27, macOS 27 (Golden Gate), iPadOS 27, and visionOS 27 abandon discrete ML tasks for a unified, generative-native compute fabric

The system search index was completely overhauled to process text, images, and data instantly using the Neural Engine

Siri AI was restructured into a system-wide semantic interface, not a standalone app — it pulls itineraries from email, cross-references photos, and generates map routes through natural language

Apple Foundation Models 3 (AFM 3) — five models spanning 3 billion parameters on-device to undisclosed cloud-pro — are the intelligence layer the hypervisor manages

3. Core AI and Core ML fork the paradigm, they don't replace it

Core ML, the standard inference framework for nearly a decade, remains the recommendation for tabular feature engineering, gradient-boosted decision trees, and traditional CNNs

Core AI is strictly required for transformer architectures, diffusion pipelines, and any neural network demanding extensive attention-mechanism computation — it's the "SwiftUI moment" for generative AI

Core AI establishes a memory-safe Swift API with zero network dependencies and zero token latency, keeping user data on-device by design

Core ML was simultaneously modernized with granular weight compression, stateful model artifacts for transformer adapters, and the new MLTensor type — the old framework got better, not killed

4. Ahead-of-time compilation eliminates the cold-start problem

coreai-build, a command-line tool integrated into Xcode 27, shifts the exhaustive compilation and hardware specialization of .aimodel files from the user's device at runtime to the developer's build environment

AOT compilation ensures virtually instantaneous model load times upon application launch — a non-negotiable requirement for background agentic tasks and synchronous UI updates

During initialization, Core AI evaluates the host device's available compute units — CPU, GPU, Neural Engine — and automatically specializes the graph execution for that specific hardware topology

Zero-copy data paths via NDArray.MutableView and NDArray.View prevent massive data matrices from duplicating across CPU and GPU memory addresses, preserving unified memory bandwidth and reducing thermal footprint

5. The Neural Engine and GPU Neural Accelerator split the workload

The Apple Neural Engine handles instantaneous completions and low-latency background tasks; Xcode 27's inline code completion runs entirely on the ANE, never touching the cloud

The GPU Neural Accelerator, a new hardware block inside each GPU shader core, accelerates the "prefill" stage of LLMs — the initial ingestion of the user's prompt and context window

Unified memory eliminates the CPU RAM/GPU VRAM division, and generative AI inference is overwhelmingly constrained by memory bandwidth during auto-regressive decoding, not raw compute

Metal 4's TensorOps library natively accelerates matrix multiplication and convolutions, routing instructions to the Neural Accelerator when present, with native hardware support for INT4, INT8, FP4, and FP8 quantization

6. AFM 3 Core Advanced stores 20 billion parameters in NAND flash and patches in only 1 to 4 billion at a time

Instruction-Following Pruning (IFP) analyzes the semantic intent of a prompt with a lightweight dense block, then selects a predetermined set of active parameters tailored to that domain task

A core set of shared experts remains resident in DRAM at all times for baseline linguistic coherence; during token generation, the model periodically reselects and updates activated experts, streaming weights asynchronously in staggered, predictive bursts

The 1-billion active parameter configuration achieved a 4.15 Mean Opinion Score for expressive text-to-speech and a 44.7% win rate against previous cloud-based production baselines for dictation and formatting

Users preferred local AFM 3 models over the previous generation more than 61% of the time for image understanding — sparse on-device execution matches or exceeds legacy cloud capabilities

7. Private Cloud Compute extends the privacy perimeter without breaking it

When on-device models hit their heuristic capacity, the OS transparently routes workloads to PCC — stateless servers built entirely on custom Apple silicon in Apple-owned data centers

PCC guarantees cryptographic non-retention: user context is processed in volatile memory and...