LLM continuous batching host scheduler in Rust (191k tok/s on 1 vCPU)

 (github.com)

2 points | by bmartin-systems 4 hours ago ago

1 comments