In order to run the model, I need to install the Unsloth desktop, on Ubuntu. The link at https://unsloth.ai/download/linux says it should be 44.3MB, but the download itself is 23M.
Is this an oversight on the download page or am I getting the wrong package?
Oh it's correct - that's the installer size - but we will need to download binaries from https://github.com/unslothai/llama.cpp and also PyTorch etc which will take 2-3GB more
Hi! Unsloth Studio is great, and I was wondering where exactly you are with multi-GPU and multi-node training and fine-tuning? I have been checking your documentation and GitHub issues from time to time and there doesn't seem to be as much development as I initially anticipated. Is making this aspect as user friendly as other parts of Studio and the docs on your near term roadmap or are your prioritizing other areas for now?
Oh we have not announced it, but in the past month, Unsloth has FSDP2 support - full finetuning, LoRA and QLoRA all work! We also support pipeline parallel, tensor parallel, and more!
IMO Byteshape is kind of the winner here if you're only looking at 24GB and down targets.
Continually compressing the model down with more computationally complex quantization schemes crosses from memory bandwidth being the limit into compute is the limit, Byteshape tries to keep the models performant while also using techniques appropriately.
Like, for me, their biggest version saves enough space that I can actually fit the dflash2 kernel without digging into context, yet score virtually identical to Unsloth's 3.0 Q4_K_M.
Thats like going from, say, 15-20 tok/s with MTP to 50+ with DFlash2.
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
Yes! One method is KV cache compression to 4-bits - llama.cpp does have hadamard, but it's not very accurate - we were researching if 4-bit KV cache can be used with a sliding window of 1024 for example of 8-bit KV cache
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)
- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
FWIW depending on what exactly you want to review something smaller like Gemma 4 E4B would be much faster. On my 7yo 3700X, llama.cpp runs Gemma 4 E4B UD-Q4_K_XL with a 131k context (context size doesn't seem to make much of a difference in speed) at ~10-15 tps (using MTP draft).
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning.
It’s fun, but it’s just so tough to see the value when the same research report via ChatGPT is done in 5 minutes and likely higher quality, for less than a dollar a day.
This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
When the XGX Spark is the equivalent of 1/10th of the pre tax annual salary of a top 5-8% earner, its far too expensive. Not sure if its cheaper comparatively in the US, but its a nightmare in Australia unless you are on an obscene salary. Throw in an approximate 30% tax rate and high cost of living - you've got to strongly wonder if it's worth it.
Great work as always Dan! I've used Qwen3.8 quite a bit and it's very verbose - its propensity for tool calling is insane and it uses 2x as many tokens as GLM-5.2 (which itself uses huge amount of tokens). I'm wondering if you have any plans to incorporate Bottlecap AI Qwen3.8 [1] checkpoints? They have done post-training where they significantly improve the token efficiency.
Oh yes I think I saw bottecap's work - very interesting!
For Qwen3.8-Flash-Next (I know not 27B haha), we actually cut reasoning by up to 50% esp for the small quants - we found quantizing models naively coincidentally also made them more verbose, as it would "correct" itself during the reasoning process. We managed to make the extra token usage of lower bit quants now only be +5% vs BF16 vs +50% extra from other quants.
27B should have similar benefits, but it's less as it's a smaller model!
Interesting, thanks for the info! Incidentally we've been testing nvfp4 weights vs vanilla 16-bit, GLM-5.3 in this case, and found that the vanilla 16 bit ones use 10x more tokens, but the output is more stable and better quality (less Chinese characters appearing, better coding solutions). So to me it's all just alchemy at this point.
Yes as requested by the community :) We also run all benchmarks multiple times, and provide error bars! For eg Terminal Bench and DeepSWE are run 3x, and LiveCodeBench 6x or more, and other faster benchmarks are run 12x
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Depends on how unique the needles are. If they're fairly unique then even Qwen 3.5 27B @ Q5 with 128k context and Q4 KV cache has perfect recall of at least up to 5 needles, according to my tests.
But I'm sure if you make them less unique they'll start to struggle.
Every local and reasonable model is bad at that. It’s OpenAI’s own benchmark, so take it with a grain of salt, but “GPT-6 Astra achieved 100.0% accuracy on contexts from 256K to 512K tokens and 96.3% accuracy from 512K to 1M tokens on OpenAI's 8-needle long-context benchmark”.
Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio.
If anyone has any questions as well - feel free to ask!
We also updated https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF with UD3.1 quants
And also https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF as well!
We provide our benchmarking methods like what graders we use, infra fixes and issues and more in https://unsloth.ai/docs/basics/dynamic-3.0-ggufs/dynamic-3.1...
In order to run the model, I need to install the Unsloth desktop, on Ubuntu. The link at https://unsloth.ai/download/linux says it should be 44.3MB, but the download itself is 23M. Is this an oversight on the download page or am I getting the wrong package?
Oh it's correct - that's the installer size - but we will need to download binaries from https://github.com/unslothai/llama.cpp and also PyTorch etc which will take 2-3GB more
Hi! Unsloth Studio is great, and I was wondering where exactly you are with multi-GPU and multi-node training and fine-tuning? I have been checking your documentation and GitHub issues from time to time and there doesn't seem to be as much development as I initially anticipated. Is making this aspect as user friendly as other parts of Studio and the docs on your near term roadmap or are your prioritizing other areas for now?
Oh we have not announced it, but in the past month, Unsloth has FSDP2 support - full finetuning, LoRA and QLoRA all work! We also support pipeline parallel, tensor parallel, and more!
See https://github.com/unslothai/notebooks/blob/main/fsdp2_scrip...
how to best run your 27G v3.1 quants on recent mac hardware with 32-64gb of ram. mlx etc. ?
what would you recommend re: software stack for best pp/tg overall performance?
Oh llama.cpp has massively improved so it should function well! We're also making MLX faster soon!
Is there something like artificialanalysis, but for the multitude of versions created for local usage, such as this one?
there is Tiny League (https://tinyleague.nunodonato.com) but it doesnt include quantized models
This is a great area of research with many players. See also ByteShape https://byteshape.com/blogs/Qwen3.8-27B/ (I'm not affiliated)
IMO Byteshape is kind of the winner here if you're only looking at 24GB and down targets.
Continually compressing the model down with more computationally complex quantization schemes crosses from memory bandwidth being the limit into compute is the limit, Byteshape tries to keep the models performant while also using techniques appropriately.
Like, for me, their biggest version saves enough space that I can actually fit the dflash2 kernel without digging into context, yet score virtually identical to Unsloth's 3.0 Q4_K_M.
Thats like going from, say, 15-20 tok/s with MTP to 50+ with DFlash2.
ByteShape was what finally made Qwen3.8 27b actually fast and usably intelligent for me after trying a LOT of different options at 16GB VRAM.
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
Yes! One method is KV cache compression to 4-bits - llama.cpp does have hadamard, but it's not very accurate - we were researching if 4-bit KV cache can be used with a sliding window of 1024 for example of 8-bit KV cache
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
FWIW depending on what exactly you want to review something smaller like Gemma 4 E4B would be much faster. On my 7yo 3700X, llama.cpp runs Gemma 4 E4B UD-Q4_K_XL with a 131k context (context size doesn't seem to make much of a difference in speed) at ~10-15 tps (using MTP draft).
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning.
It’s fun, but it’s just so tough to see the value when the same research report via ChatGPT is done in 5 minutes and likely higher quality, for less than a dollar a day.
This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
Non-local models are sometimes unavailable.
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
I think the last two should have been "Local models" instead of "Non-local models" :-P
Depends. The running costs of the DGX Spark-like’s GB10 are very low; they run at crazy low wattages really. But they’re expensive devices
When the XGX Spark is the equivalent of 1/10th of the pre tax annual salary of a top 5-8% earner, its far too expensive. Not sure if its cheaper comparatively in the US, but its a nightmare in Australia unless you are on an obscene salary. Throw in an approximate 30% tax rate and high cost of living - you've got to strongly wonder if it's worth it.
Doesn't Qwen3.8 do much better than Luna?
You can rent a dedicated server. It's easy and costs 10 times less than a "cloud".
if privacy doesn't matter for you, why use expensive luna instead of deepseek?
>however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token
What is the price of not having your ssh keys uploaded to the NSA?
Great work as always Dan! I've used Qwen3.8 quite a bit and it's very verbose - its propensity for tool calling is insane and it uses 2x as many tokens as GLM-5.2 (which itself uses huge amount of tokens). I'm wondering if you have any plans to incorporate Bottlecap AI Qwen3.8 [1] checkpoints? They have done post-training where they significantly improve the token efficiency.
[1] https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-G...
Oh yes I think I saw bottecap's work - very interesting!
For Qwen3.8-Flash-Next (I know not 27B haha), we actually cut reasoning by up to 50% esp for the small quants - we found quantizing models naively coincidentally also made them more verbose, as it would "correct" itself during the reasoning process. We managed to make the extra token usage of lower bit quants now only be +5% vs BF16 vs +50% extra from other quants.
27B should have similar benefits, but it's less as it's a smaller model!
Interesting, thanks for the info! Incidentally we've been testing nvfp4 weights vs vanilla 16-bit, GLM-5.3 in this case, and found that the vanilla 16 bit ones use 10x more tokens, but the output is more stable and better quality (less Chinese characters appearing, better coding solutions). So to me it's all just alchemy at this point.
Oh yes interesting!
I am happy they are showing charts for actual benchmarks, not just easily measurable proxies like KL-div.
Yes as requested by the community :) We also run all benchmarks multiple times, and provide error bars! For eg Terminal Bench and DeepSWE are run 3x, and LiveCodeBench 6x or more, and other faster benchmarks are run 12x
releasing the Q8 variants anytime soon with the dynamic work ?
Q8 stomps Q6 on personal benchmarks here for me wrt Qwen 3.8 27b.
How does it compare to the 3.8-27B versions from TensorFold? I.e. does it make sense to run unsloth via TensorFold engine?
How does it compare to ik_llama's KT/KS quants?
How do the smallest quants of this compare to Bonsai 2?
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better.
Bonsai's technology would be better suited for 100s of B models.
Bonsai 2 is quite bad, it cannot even reliably retrieve multiple needles. regardless what the benchmark says.
Every model is bad at needle in a haystack. I am constantly shocked at how often they fail my evals.
Depends on how unique the needles are. If they're fairly unique then even Qwen 3.5 27B @ Q5 with 128k context and Q4 KV cache has perfect recall of at least up to 5 needles, according to my tests.
But I'm sure if you make them less unique they'll start to struggle.
Every local and reasonable model is bad at that. It’s OpenAI’s own benchmark, so take it with a grain of salt, but “GPT-6 Astra achieved 100.0% accuracy on contexts from 256K to 512K tokens and 96.3% accuracy from 512K to 1M tokens on OpenAI's 8-needle long-context benchmark”.
Figuring out local llms , but it did built me single page html . On 12 GB VRAM and 16 GB ram windows .
Give us more details. What kind of hardware? What kind of setup?
How do you know which parameters matter?
Less important parameters should get quantized more, seems odd that we would quantize every parameter the same. At least for storage.
Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio.
Awesome! I wonder how it does at Q5+ compared with v3.0
We updated 4, 5 and 6bit as well! For now no agentic / coding benchmarks, but KLD is definitely much better!