I Benchmarked Every Model That Fits on an iPhone

Every iOS team is having the same argument right now: use Apple's free on-device system<br>model, bring your own open model via MLX, or pay for a cloud API. Most of the arguments<br>run on leaderboard folklore and numbers measured on someone else's hardware. So I<br>measured it myself: the Foundation Models system model plus Qwen3 1.7B/4B/8B,<br>Llama 3.2 3B, and Gemma 3 4B, on the devices they'd actually ship on.

TL;DR

Apple's system model is the throughput king on iPhone (~149 tok/s) and its weights live outside your app's memory budget. Peak app memory during inference: 12MB.

There is a thermal cliff at the 4B class on iPhone: Qwen3 4B and Llama 3.2 3B pushed the device to a "fair" thermal state mid-run and throughput collapsed to 17–24 tok/s.

Qwen3 1.7B is the bring-your-own-model sweet spot on iPhone: 0.33s to first token, ~46 tok/s, 1.2GB peak, and it matched the system model on quality checks.

4-bit quantization cost nothing detectable vs 8-bit on our eval set. The judge-score deltas straddled zero across all five models.

Method, before the numbers

Fixed prompt set, cold and warm runs separated, median of 5, thermal state recorded per<br>run. The harness is a small Swift package (modelbench) that drives<br>Foundation Models directly and open models through MLX Swift; quality is checked with a<br>5-case golden set: typed tier-1 property assertions plus an LLM judge<br>(Qwen3 32B, thinking off, temp 0.1) running locally on a Mac Studio. Judge scores<br>are a ranking, not absolutes; the system-model baseline itself moved a point between<br>runs, so read ±1–2 as the measurement floor. iPhone numbers are an A17 Pro on OS 26.x;<br>iPad (M4) numbers are iPadOS 27 beta.

The iPhone table

ModelWeights @4-bitTTFT warmtok/s (est.)Peak app memQuality

FM system (~3B)0 (system)0.38s~149n/a (system)5/5 tier-1 · judge 9/10<br>Qwen3 1.7B~1.0GB0.33s~461.2GB5/5 tier-1 · judge 10/10<br>Llama 3.2 3B~1.8GB0.73s~24 (thermal: fair)2.0GB5/5 tier-1 · judge 10/10*<br>Qwen3 4B~2.3GB1.52s~17 (thermal: fair)2.4GB5/5 tier-1 · judge 8/10<br>Gemma 3 4B~2.4GB1.71s~122.7GB5/5 tier-1 · judge 10/10

* Llama 3.2 3B judged on 3/5 golden cases (judge-load timeouts on the rest). FM cold<br>TTFT is 2.05s; all open-model runs shown warm. tok/s is estimated from output tokens<br>over generation time, hence "est."

Finding 1: the system model's real advantage isn't speed, it's memory

~149 tok/s is more than 3× the fastest open model we ran on the same phone, but the<br>number that changes architectures is the memory one. The system model's weights and KV<br>cache are Apple's problem, not yours: our app's peak memory during inference was 12MB.<br>Every MLX model, by contrast, carries its full weight file inside your jetsam<br>budget. On an 8GB phone, a 2.4GB model plus your app plus a WebView is a crash report.

Finding 2: the 4B thermal cliff

The 4B class technically fits on an A17 Pro. It also cooked the phone. Qwen3 4B and<br>Llama 3.2 3B both degraded the thermal state to "fair" during a 5-run median,<br>and sustained throughput landed at 12–24 tok/s, slower reading speed than most users<br>tolerate for streaming UI. The same Qwen3 4B on a fanless iPad (M4) did 36 tok/s at<br>0.33s TTFT without breaking a sweat. The class boundary is real: on iPhone today, bring<br>a ~2B model or use the system one.

Finding 3: 4-bit is free (on this task)

Chapter-8-of-every-blog wisdom says 4-bit quantization visibly hurts quality. We ran<br>the full golden set at q8_0 against the Q4 defaults for all five open<br>models. Tier-1 typed assertions: 5/5 for every model at both quants. Judge-median<br>deltas: −1, −1, +2, 0, 0. That straddles zero with no consistent direction, and ±1–2 is<br>the measurement floor. On a structured-output task, 4-bit cost nothing we could detect,<br>and it halves the weight file you're carrying inside your memory budget.

The Mac numbers, for scale

ModelDeviceTTFT warmtok/s (est.)Peak mem

Qwen3 4BiPad (M4)0.33s~362.5GB<br>Gemma 3 4BiPad (M4)0.50s~372.8GB<br>Qwen3 8BM4 Max0.14s~1054.8GB<br>Qwen3 32BM2 Ultra0.84s~3717.6GB

The 32B row is there because it's the judge for the quality scores above: a local<br>LLM-as-judge on a Mac Studio, no cloud in the eval loop. If your team has one big-RAM<br>Mac on the LAN, you have an eval rig.

Reproduce it

Every number here states its method, and the harness builds with<br>swift build: no cloud keys, no telemetry. Licenses differ per model<br>(Apache-2.0 for Qwen3, Llama Community for 3.2, Gemma Terms for Gemma 3); check them<br>before shipping weights in an app. If you re-run this on other hardware, I want the<br>numbers: send them.

Where this came from

This table is Appendix A of a 12-chapter book.

On-Device Intelligence covers the decision matrix behind these numbers,<br>Foundation Models in production, MLX when you need model control, memory budgets,<br>App Review, and the local eval rig that produced the quality scores. Every code<br>snippet is compiler-verified against the shipping SDK. Chapter 3 is free.

Get the book<br>Read Chapter 3 free