I built a llama.cpp fork that fixes a real problem: with stock llama.cpp, extra GPUs mostly just add memory, not speed,
for MoE models too big for VRAM only one GPU computes at a time.
Mine adds a live expert cache so multiple GPUs actually compute in parallel
with the CPU. Measured on 2x RTX 3090:
12.3 to 27.7 t/s on GLM-5.3-Flash (117GB),
4.7 to 9.9 t/s on MiMo-2.6-Flash (132GB) — 2-2.3x, same output quality. Builds on an existing open PR (csantiago78's GPU expert-cache PR #27861) — credited prominently in the README,
along with GLM-5.3-Flash support from timkhronos's PRs.
Happy to answer questions about the implementation.
Looking for Job abroad too as war is looming in EU
I built a llama.cpp fork that fixes a real problem: with stock llama.cpp, extra GPUs mostly just add memory, not speed, for MoE models too big for VRAM only one GPU computes at a time. Mine adds a live expert cache so multiple GPUs actually compute in parallel with the CPU. Measured on 2x RTX 3090: 12.3 to 27.7 t/s on GLM-5.3-Flash (117GB), 4.7 to 9.9 t/s on MiMo-2.6-Flash (132GB) — 2-2.3x, same output quality. Builds on an existing open PR (csantiago78's GPU expert-cache PR #27861) — credited prominently in the README, along with GLM-5.3-Flash support from timkhronos's PRs. Happy to answer questions about the implementation. Looking for Job abroad too as war is looming in EU
Are you going to / have you tried to contribute this to the main repo? Is the fork for visibility for job hunting?
[dead]