There's, quietly, a llama.cpp WebGPU backend that works *great*.
Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school.
Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really involved after delivering their big refactor that was DOA, other than telling people on issues he's never seen their repro himself. Carefully pick models you can support down to "I can patch around the Jinja template engine". Go through the same llama web server APIs. c.f. github / telosnex / fllama if you need a reference
(n.b. seeing peer comments, its much better than ONNX, ONNX never got within spitting distance of llama.cpp, my understanding from watching the LLM runtime is its used for Windows AI features so their models probably work great on Windows x Qualcomm. ONNX is great for smol models though, like VAD, a god send even.)
Bake me a cake
responds with
> Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.
This seems to be the demo:
https://chat.webllm.ai/
I am getting:
On both, FireFox and Chromium on Linux.You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
What did you switch to?
We use the ONNX runtime for small models in the browser https://github.com/microsoft/onnxruntime
Yeah ONNX runtime or Burn-rs are great because they compile to wasm
And ONNX is what Transformers.js uses as well, at least for the moment.
There's, quietly, a llama.cpp WebGPU backend that works *great*.
Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school.
Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really involved after delivering their big refactor that was DOA, other than telling people on issues he's never seen their repro himself. Carefully pick models you can support down to "I can patch around the Jinja template engine". Go through the same llama web server APIs. c.f. github / telosnex / fllama if you need a reference
(n.b. seeing peer comments, its much better than ONNX, ONNX never got within spitting distance of llama.cpp, my understanding from watching the LLM runtime is its used for Windows AI features so their models probably work great on Windows x Qualcomm. ONNX is great for smol models though, like VAD, a god send even.)
Transformers.js supports a lot of models since v4