I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.
A small update: I've also managed to port the engine to iOS.
I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session:
"Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "
I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.
A small update: I've also managed to port the engine to iOS.
I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
Cool, I guess.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session: "Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "