Local AI On Apple Silicon uses 7X Less RAM

Turbo Fieldfare: Local AI on Apple Silicon

🎯 Introduction: Turbo Fieldfare is a Swift and Metal-based Mac app that runs a 26-billion-parameter Gemma 4 mixture-of-experts model entirely locally. Its breakthrough is cutting memory usage from roughly 14.3 GB down to just 2.15 GB, while still achieving 23.4 tokens per second on an M3 Max—fast enough for smooth interactive use.

💡 Key Technical Innovation: Gemma 4 is a mixture-of-experts model: each of its 30 layers contains 128 small feed-forward blocks (experts) plus one router. For every token, the router picks only the top 8 experts, so 85% of the 26 billion parameters stay idle. Turbo Fieldfare exploits this by splitting the model into two piles. The resident pile—about 1.35 GB—holds attention, the router, embeddings, and a shared expert that always runs; it is memory-mapped from disk and stays in RAM. The rest—12.9 GB of experts, each roughly 3.36 MB—never loads, sitting on the SSD until needed. When generating a token, attention runs entirely in the resident pile without touching disk. Then the router names the 8 required experts, and because that choice depends on the current token and all tokens before it, prefetching is impossible. The CPU must stop and fetch a few megabytes from SSD, 30 times per token. To hide this latency, the shared expert from the resident pile executes concurrently, making the disk read almost free. Each layer also keeps 16 recently popular experts in an LFU cache; the least-frequently-used expert is evicted, which works better than an LRU cache because routing is highly predictable—some experts are always selected while others almost never are.

⚙️ Why Apple Silicon?

• On a discrete GPU, you’d need SSD → system RAM → PCIe → VRAM, two copies and a bus hop.
• Apple Silicon unifies memory: CPU and GPU see the same RAM, so a Metal buffer is directly accessible, letting SSD reads go straight into the GPU’s working memory, skipping the bus hop.
• The custom Metal file format stores weights exactly as the kernel consumes them—including 4-bit quantized values—so reading the file is loading the weight, with zero conversion or unpacking.

🏁 Conclusion: This proves the potential for MoE models on edge devices like phones or watches. The presenter encourages trying the repo, subscribing, and watching the companion video.