A recent technical demonstration has successfully deployed Google's 26-billion-parameter Gemma 4 mixture-of-experts model on a 13-year-old Xeon with no GPU [1]. The result is a decode speed of about 5 tokens per second, which is slow but usable for batch work. The journey to get there reveals the state of local LLM inference on non-standard hardware. The author of the demonstration notes that the real skill is not just paying for a subscription, but knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is actually correct [2].
The demonstration used the ik_llama.cpp fork, which adds optimizations for Gemma 4's MoE inference [3]. The fix for a silent MoE bug is now upstream, and the patch is available for anyone with similar hardware [4]. The root cause of the bug was a silent bug in how the code handled Mixture-of-Experts operations on non-AVX2 hardware [5]. The author confirmed the upstream fix, and the patch is available for anyone with similar hardware [6].
This achievement shows that it is possible to run modern language models on aging hardware, and that the real challenge is not just the hardware, but the willingness to open the hood and debug the code. As the author notes, "the subscription is the easy part. The rest is the willingness to open the hood, read a stranger's code, and keep asking until a thirteen-year-old CPU does something it was never meant to" [7].
