I wanted to understand how modern AI models actually generate text, but most inference codebases are tens or hundreds of thousands of lines long. They’re incredibly impressive, but they’re optimized for flexibility and performance.
So I implemented a complete CPU runtime for Google’s latest open language model, Gemma 4 E2B, in about 700 lines of C.
The whole point is that you can open one file, start at main() , and follow a prompt all the way through the program. You can see every buffer that’s allocated, every mathematical operation that transforms the activations, every update to the KV cache, and every step that eventually produces the next token.
Along the way I kept optimizing the CPU implementation for this one model. Specializing the runtime let me push performance surprisingly far, and it now outperforms llama.cpp on my Ryzen 7 7700.
I wanted to understand how modern AI models actually generate text, but most inference codebases are tens or hundreds of thousands of lines long. They’re incredibly impressive, but they’re optimized for flexibility and performance.
So I implemented a complete CPU runtime for Google’s latest open language model, Gemma 4 E2B, in about 700 lines of C.
The whole point is that you can open one file, start at main() , and follow a prompt all the way through the program. You can see every buffer that’s allocated, every mathematical operation that transforms the activations, every update to the KV cache, and every step that eventually produces the next token.
Along the way I kept optimizing the CPU implementation for this one model. Specializing the runtime let me push performance surprisingly far, and it now outperforms llama.cpp on my Ryzen 7 7700.