I so wanted to be able to use Qwen3.8-27B locally on my fully loaded M1 Max 64GB when spider-mario recommended I use MTPLX[1], but I still found the general MLX MTP acceleration to be so poor, I'd rather just use cheap tokens from OpenCode Zen and Go.
It's just too slow of a model. I know, I know there's new hardware, but my business paid like 4-5 grand for this MBP at the time, and I just don't feel the need to pay 7-8 grand to step up to current hardware when token spend is what it is.
slow as in the output is slow, or slow as in slow token rates?
qwen3.8 has been fantastic as far as token rates are concerned for me, but the overthinking thing with higher reasoning levels takes some coercion to get right.
fwiw pi and hermes both handle that model fairly well. omp required a lot of tuning. I didn't bother figuring out why, I presume it's because qwen3.8 expects reasoning declarations a bit differently. nothing a proxy can't fix.
You should check out our TerminalBench2.1 score for that, the tasks there can run up to 4h! We got almost no loss (we got Base 66.74% vs Swift 65.84%) with -38.7% thinking token reduction. There's also quite a few independent evals on the reddit link as well.
Please let me know when you try the model and if I can help you set it up :)
I will TLDR you on our thought process, research, training and benchmarks.
1. When running our quantized Qwen 3.8 27B instances we were very annoyed by random reasoning loops (in the paper bellow refered to as "overthinking errors". These random loops were persistent throughout medium and low reasoning settings.
2. We found a paper by Meta that's supposed to target this phenomenon in PTQ, but when used straight out of the box got mixed results. https://arxiv.org/abs/2606.00206
3. We figured to try if it's a matter of the targeting the right keywords and tuning the parameters, so we used our 8xH100 box and and generated a large amount of different (ofc out of distribution) domain (coding, language, vision, agentic) traces.
4. We then grouped the ones with overthinking and found "common denominator" tokens between them and targeted the most prominent ones.
5. We then built an inference-time penalizer of those tokens as seen in the paper with the hopes of simply generating traces and doing cross-entropy SFT over them.
6. Did not work at all, but the penalizer seemed to work much better than the tokens provided in the paper and not only for lower precision models but for bf16 as well. Hence we kept experimenting with it. We built a loss function using the tokens we identified and ran LoRa SFT over the traces prev generated and reasoning seemed to be falling off significantly but the accuracy seemed to follow. The reasoning reduction seemed to be generalizing.
7. After a significant amount of tinkering (literally since the day of Qwen 3.8 27B release) we were satisfied with the reasoning reduction. After that we searched for ways of restoring the accuracy. We experimented with several methods, including RL(GSPO), On-Policy Distillation and using the ThinkingCap 3.6 27B adapter chunks until we were satisfied with our accuracy loss. We managed to restore it to <1% loss on almost all of our OOD in house tests
8. We then performed intensive intensive benchmarks, across several reasoning efforts, precision variants etc. We ran into a few problems, one of which is that to get a reliable score we needed to run each benchmark 10x (5x on base + 5x with our adapter, this being the standard procedure on the Qwen 3.6 27B model card on Terminal Bench which we followed). After running it, the performance converged to 40-60% token reduction with <1% accuracy loss across GPQA, MMLU, Terminal Bench 2.1, LiveCodeBench v6, ERQA, C-Eval, IFBench, HMMT25, with an exception being AIME26 with an accuracy loss of 4.6%, which we later linked to a bug during training with a specific token relevant for math-related reasoning being penalized and are planning to fix it in an updated release.
Congrats, seems like a promising concept. I'll give it a go in my local setup. My prior is thinking this will definitely work for speed but also definitely compromise accuracy (I've seen this happen so many times), though benchmark results are encouraging of course.
Nice explanation too. Now that everything reads like a sad blur, it felt refreshing to read your writeup.
I've been using this and it's quite amazing for the amount I've used it. Thanks for the hard work.
Thank you so much!! It would be great if you could share some numbers with the community :)
I so wanted to be able to use Qwen3.8-27B locally on my fully loaded M1 Max 64GB when spider-mario recommended I use MTPLX[1], but I still found the general MLX MTP acceleration to be so poor, I'd rather just use cheap tokens from OpenCode Zen and Go.
It's just too slow of a model. I know, I know there's new hardware, but my business paid like 4-5 grand for this MBP at the time, and I just don't feel the need to pay 7-8 grand to step up to current hardware when token spend is what it is.
[1]: https://news.ycombinator.com/item?id=49611128#49612229
slow as in the output is slow, or slow as in slow token rates?
qwen3.8 has been fantastic as far as token rates are concerned for me, but the overthinking thing with higher reasoning levels takes some coercion to get right.
fwiw pi and hermes both handle that model fairly well. omp required a lot of tuning. I didn't bother figuring out why, I presume it's because qwen3.8 expects reasoning declarations a bit differently. nothing a proxy can't fix.
How does it perform on real long horizon tasks?
You should check out our TerminalBench2.1 score for that, the tasks there can run up to 4h! We got almost no loss (we got Base 66.74% vs Swift 65.84%) with -38.7% thinking token reduction. There's also quite a few independent evals on the reddit link as well.
Please let me know when you try the model and if I can help you set it up :)
I will TLDR you on our thought process, research, training and benchmarks.
1. When running our quantized Qwen 3.8 27B instances we were very annoyed by random reasoning loops (in the paper bellow refered to as "overthinking errors". These random loops were persistent throughout medium and low reasoning settings.
2. We found a paper by Meta that's supposed to target this phenomenon in PTQ, but when used straight out of the box got mixed results. https://arxiv.org/abs/2606.00206
3. We figured to try if it's a matter of the targeting the right keywords and tuning the parameters, so we used our 8xH100 box and and generated a large amount of different (ofc out of distribution) domain (coding, language, vision, agentic) traces.
4. We then grouped the ones with overthinking and found "common denominator" tokens between them and targeted the most prominent ones.
5. We then built an inference-time penalizer of those tokens as seen in the paper with the hopes of simply generating traces and doing cross-entropy SFT over them.
6. Did not work at all, but the penalizer seemed to work much better than the tokens provided in the paper and not only for lower precision models but for bf16 as well. Hence we kept experimenting with it. We built a loss function using the tokens we identified and ran LoRa SFT over the traces prev generated and reasoning seemed to be falling off significantly but the accuracy seemed to follow. The reasoning reduction seemed to be generalizing.
7. After a significant amount of tinkering (literally since the day of Qwen 3.8 27B release) we were satisfied with the reasoning reduction. After that we searched for ways of restoring the accuracy. We experimented with several methods, including RL(GSPO), On-Policy Distillation and using the ThinkingCap 3.6 27B adapter chunks until we were satisfied with our accuracy loss. We managed to restore it to <1% loss on almost all of our OOD in house tests
8. We then performed intensive intensive benchmarks, across several reasoning efforts, precision variants etc. We ran into a few problems, one of which is that to get a reliable score we needed to run each benchmark 10x (5x on base + 5x with our adapter, this being the standard procedure on the Qwen 3.6 27B model card on Terminal Bench which we followed). After running it, the performance converged to 40-60% token reduction with <1% accuracy loss across GPQA, MMLU, Terminal Bench 2.1, LiveCodeBench v6, ERQA, C-Eval, IFBench, HMMT25, with an exception being AIME26 with an accuracy loss of 4.6%, which we later linked to a bug during training with a specific token relevant for math-related reasoning being penalized and are planning to fix it in an updated release.
Congrats, seems like a promising concept. I'll give it a go in my local setup. My prior is thinking this will definitely work for speed but also definitely compromise accuracy (I've seen this happen so many times), though benchmark results are encouraging of course.
Nice explanation too. Now that everything reads like a sad blur, it felt refreshing to read your writeup.
Nice. Going to dl and give it a whirl.