MediumRunning Gemma4 on Apple Neural Engine | by MLBoy | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

Gemma 4

IOS

Coreml

Apple Neural Engine

LLM

Running Gemma4 on Apple Neural Engine

MLBoy

13 min read·<br>Apr 21, 2026

Listen

Share

Silent Calculator<br>— The job of running Gemma 4 on Apple Neural Engine —

one<br>Apple devices contain a computing unit called the Neural Engine, or ANE for short. It first appeared in iPhones with the A11 in 2017, and has since been updated with subsequent versions. The A19 Pro, released in 2026, is officially claimed to have a computing performance of approximately 19 teraflops in FP16.<br>This figure is more than double (!) the theoretical maximum GPU performance of the iPhone 17 Pro (approximately 9 TFLOPS) .<br>Nevertheless, the ANE is a device that is almost invisible from the outside. When you use Face ID, when the Photos app automatically searches for people, when voice memos isolate speakers, the ANE is quietly working in the background. Its power consumption is less than one-third of that of performing equivalent matrix operations on the GPU. It also generates almost no heat. Furthermore, third-party developers can only access it, in principle, via CoreML .<br>Why isn’t this incredibly efficient computing unit a topic of discussion in the world of large-scale language models?<br>The answer is that ANE does not have a straightforward instruction for matmul — matrix multiplication, which is central to large-scale language models. ANE’s native operation is convolution, and matmul is represented as a special case (kernel size 1x1). The computational path is roundabout, even though it goes through the same mathematics. tensor_ops::matmul2dWhile the A19 Pro GPU can handle a 16x16 fragment MMA via Metal 4, ANE expands the same matrix multiplication as Conv1x1. Hardware arithmetic unit utilization drops to roughly 1/3.<br>The task of running LLM on ANE begins only after overcoming this gap.

two<br>The target is Google’s Gemma 4 E2B and E4B.<br>E2B has 35 Transformer layers, 1536 hidden layers, and one KV head num_key_value_heads=1. With effective parameters of 2B and INT4 quantization, it is approximately 3.1 GB. E4B has 42 layers, 2560 hidden layers, two KV heads, effective parameters of 4B, and INT4 quantization,<br>resulting in 5.5 GB.<br>Both are small models marketed as “edge-oriented,” yet they cannot be run on an iPhone in their uncompressed state. iOS’s jetsamprocess termination mechanism for low memory will kill them the moment they switch from the background. An iPhone 17 Pro (8 GB RAM) has just os_proc_availableover 5 GB of RAM, and LLM alone cannot consume that much.<br>phys_footprintThis repository ultimately achieved 873 MB after loading and 981 MB during inference . Xcode's memory gauge is unreliable as it underestimates the INT4 palettized tensors that CoreML holds internally, so task_vm_infowe took it directly. To keep this under 1 GB, the model was split into four CoreML chunks, each compressed to INT4 (4-bit palettization).<br>Just by describing the settings up to this point, we can already set up three inequalities.<br>Firstly, the entire model cannot fit into a single CoreML graph .<br>Secondly, there is no room to maintain weights with a precision of INT8 or higher .<br>Thirdly, the inference path must not monopolize the GPU (the GPU needs to be free for the camera, UI, and video).<br>The only option that satisfies all three conditions simultaneously is practically ANE.<br>And from here, the maze begins.

three<br>Anyone who has tried to install LLM on ANE will encounter the following phenomenon at least once.<br>Attempting to directly convert a deep Transformer like Gemma 4 as a single CoreML model will cause the conversion process to die without error . More precisely, the MIL (Model Intermediate Language, an intermediate representation of CoreML) compiler goes silent the moment it exceeds a certain depth in the graph, and the resulting model fails to initialize .mlpackageon the iPhone error code: -14, returning an empty string.<br>This threshold is not documented. All we know empirically is that failures occur at a depth equivalent to approximately 8–10 connected Attention layers. Apple does not specify where in the MIL (Multiple Increment Layer) failure occurs or under what conditions. Reporting it through Radar (Apple’s bug reporting system) yields no response.<br>Therefore, we divide the model by hand.<br>In the case of Gemma 4 E2B, there are four ways to split the code.<br>The token embeddings and several layers of attention are grouped together as chunk1,<br>the middle section as chunk2, chunk3,<br>and then lm_headchunk4 which includes (output projection and conversion to vocabulary).<br>Each chunk .mlpackageis converted as an individual, and the Swift side ChunkedEngineaccesses all four in order.<br>Herein lies another unavoidable cost.<br>A CoreML MLModel.predictioncall itself ANECompilerServiceawakens a daemon via XPC (Inter-Process Communication), and that daemon then...