The best AI model in my benchmark crashed my iPhone
In August I wrote about why I believe in local AI on phones. A belief is cheap, so this month I measured it. Nine open AI models, one iPhone 17e with 8 GB of memory, 220 questions from public benchmarks, the same settings for every model, all of it running inside the app I ship. The full results table and method are on the Ragondin Assistant blog. This post is about what surprised me.
1. The best model crashed the phone
Bonsai 27B is a 27-billion-parameter model squeezed to about one bit per weight. Its file is 3.8 GB, so it loads on the phone, and it chats fine. It also has the best answers of the whole test. But a benchmark is not a chat. A chat is short bursts of work; a benchmark keeps the GPU busy without a break. Early in the run, the GPU hit a hardware error and iOS restarted it. Every answer after that came back empty, in 40 milliseconds.
I ran the same 220 questions on my laptop instead, and it scored 76.1, first overall. The lesson I keep coming back to: a model that fits in memory is not a model that survives sustained load. Those are two different tests, and only one of them shows up in a demo.
2. My own benchmark lied to me first
The first full run had Gemma 4 much lower than it deserved. The cause was in the inference stack, not the model: Gemma uses sliding-window attention, and the cache that speeds up consecutive prompts was being reused across unrelated questions. Bits of one answer leaked into the next. Once I cleared the cache between questions for that kind of model, Gemma jumped from 120 to 147 correct answers out of 220.
If you publish benchmark numbers, read the raw answers before you trust the totals. The totals looked perfectly plausible. The answers did not.
3. One rule reshuffled the ranking
Every answer can run to 2,048 tokens. I decided that an answer still being written at that limit counts as wrong, because on a phone an answer you never get is no answer. That single rule moved one model more than any other: LFM2.5 2.6B would have scored 75.5 without it, first on the phone. With it, 71.9, third. Thirty-three of its answers hit the limit, against one for Qwen 3.5 4B.
4. Thinking has a price, and it's paid in minutes
LFM2.5 reasons before almost every answer, even with its thinking mode switched off. The reasoning is smart, but it is long: a typical answer runs to 661 tokens, four times more than Qwen 3.5 4B. The whole test took it four hours and ten minutes. MiniCPM5 2B, less than two points behind on the same questions, finished in 33 minutes. On a phone, a minute per answer is a different product from eight seconds per answer.
5. The top is a tie
With 220 questions, every score is good to about five points either way. Qwen 3.5 4B (74.5), LFM2.5 2.6B (71.9), MiniCPM5 2B (70.3) and Gemma 4 E2B (69.1) are inside that margin. So the honest answer to "which model is best on an iPhone" is: for an 8 GB phone, Qwen 3.5 4B if you want the best answers, MiniCPM5 2B if you want them twice as fast. Below two billion parameters, the scores drop off a cliff, whatever the speed.
What I take from it
Phones are good enough now for the private, everyday layer I wrote about in August, and the gap between a 2B and a 4B model matters less than the gap between eight seconds and a minute. The hard part of shipping local AI is not picking the smartest model. It is picking the one that answers fast enough, fits in memory, and keeps working when someone uses it for an hour straight.
The video version, in English and French: I Tested 9 AI Models on an iPhone. The Best One Crashed It.