From what I understand, this project is an attempt to reduce the memory requirements for both LLM weights (using quantization), and also reduce the memory required for the KV-cache. The KV-cache is more or less the "context" of the model, and can be quite a bottleneck in inference speed. When I run local models, the KV-cache for something like Spark-X2.6 can even exceed the size of the model itself, and it takes longer to generate tokens if I am using more of the KV cache. DeepSeek has been working quite a lot on KV-cache efficiency, by sharing the KV-cache of between LLM tensors - so each token of context only requires 890 bytes in the KV-cache of DeepSeek-V4.1-Flash[0]. I am a bit skeptical of the O(1) KV cache performance recorded here - I wish I could understand the math in the project, but I feel that such a lossy (importance-aware compression?) approach to reducing the KV-cache size would harm the model too much to be usable... Though the speed of the approach looks real good!
From what I understand, this project is an attempt to reduce the memory requirements for both LLM weights (using quantization), and also reduce the memory required for the KV-cache. The KV-cache is more or less the "context" of the model, and can be quite a bottleneck in inference speed. When I run local models, the KV-cache for something like Spark-X2.6 can even exceed the size of the model itself, and it takes longer to generate tokens if I am using more of the KV cache. DeepSeek has been working quite a lot on KV-cache efficiency, by sharing the KV-cache of between LLM tensors - so each token of context only requires 890 bytes in the KV-cache of DeepSeek-V4.1-Flash[0]. I am a bit skeptical of the O(1) KV cache performance recorded here - I wish I could understand the math in the project, but I feel that such a lossy (importance-aware compression?) approach to reducing the KV-cache size would harm the model too much to be usable... Though the speed of the approach looks real good!
[0] - https://arxiv.org/abs/2609.19969