We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
On current GPU hardware/firmware... The cynic in me imagines in the future GPU vendors wanting to implement only loading signed weights to your card, aligned with your "subscription level", for your "safety".
But the GPU vendors won't really have an incentive to do this - artificially restricting their customer base? Unless they and the hyperscalers all collude to cartel the market of course!
This is why it's so important to have more than one hardware vendor. The bigger the ecosystem, the harder it is to cartel. I cant believe it, but i am pinning my hopes on china to break through with commoditized chips and hardware.
> But the GPU vendors won't really have an incentive to do this
Unless the GPU vendors start selling end products? It's much more lucrative to sell locked down end-user devices after all. Shareholders will demand it.
I was just implying that the (free, unregulated) market does not always give the consumer what they really want. With two examples where demand does not lead to consumers "winning".
The average/median consumer may get what they want. However, unfortunately in practice it means that lots of people will not get what they want. It's like the tyranny of the majority that some people believe is a part of democracy.
People came up with the word "enshittification" for a reason ... and the word is quite popular on HN, so make of that what you want.
I love the Lotus Elise and desperately wish there was a similar affordable, super light weight; no frills, sports car made today.
Here I thought I was just a niche market that’s not profitable for manufacturers to serve, but apparently I’m a victim of tyranny and didn’t even know it!
I worry they get so much money from hyperscalers we just get scraps. Consider that I'm running 6 and 8 year old GPUs for my local inference. If I wanted to replace my hardware with latest stuff (nvidia dgx, gorgon halo, Intel b70 or amd r9700 pro) it would cost me 4x the price and I'd cut the memory bandwidth in every single case. In case of dgx/strix were talking less that a quarter of memory bandwidth and a third of compute of a single rtx3090...
Those products are a sick joke played on us. About the only thing they have that is better than tjose ancient gpus is energy efficiency.
And what good are the weights when all the rtx3090s fail (they were pretty crappy to begin with) and every ddr3 Xeon system fails too?
I'm fully prepared to be running on my hardware that is 6 and 8 years old for next decade. Last 3 months saw rtx3090 going up by 20% each month.
That's a fair comment. There is lots of speculation around NVIDIA is shifting all GB202 production to RTX Pro GPUs, away from the consumer 5090s. So maybe less about restricting user base, and more about focusing on where to maximize returns.
There is already a big market for open model hosting and the accompanying demand for people and DCs. If you know how to deploy models — yes even open ones — and scale inference you are a hot engineer in demand.
You don't need bigger or better models. Current models are more than adequate for any task.
The key thing is tool use.
The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.
New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.
Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.
Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.
> You don't need bigger or better models. Current models are more than adequate for any task.
At the very least we need _current_ models, though. The weights have all sorts of information baked into them, like "how do Python unit test drivers work" or "what does idiomatic C# look like," that will just naturally go stale over time as the frameworks improve. In five years, Qwen 3.8 will still be targeting the Python of 2026.
So target stable languages that don't change, TCC with win32 is a fantastic target. The apps work great with wine and you have opengl, controller and audio support. I have a wrapper for winsock 1.0 I need to package up and release, works great for small network apps.
An interesting question is just how much high-level programming language development we'll see going forward. What is the incentive to create new languages? They're all just object code to an LLM, so when starting a new project, you might as well go with an old-school language like C++ or Java or Python or JavaScript that is well-represented in the training distribution.
Why do we need Rust, for instance, when LLMs know how to use C++ without shooting themselves in the foot? If that's not entirely true today, it certainly will be tomorrow.
See also the lack of new instruction-set architectures in the CPU business. Why do we need more, in the absence of fundamental and revolutionary hardware innovations? Even RISC-V is just a rehash of MIPS.
I think someone will come up with a language which is nicer for "vibe coding".
I.e. I think you can bake certain properties into the language such that various kinds of static checks can prevent certain types of errors. This should make the language nicer for coding without reading any of the code.
You think LLMs don't make mistakes? My sweet summer child! I want all the guardrails with LLM code. Reviews are fatiguing. If I can remove "memory safe" from my list of worries, so much the better.
You know the LLMs that are finding all these CVEs lately are finding them because they are better than the people who wrote the original code, and better than every one of the people who ever laid eyes on it.
An LLM doesn't need a memory-safe language. It needs one it knows like the back of its metal hand.
Not really - fine tuning can be done on much lower tier hardware, with much smaller datasets. With a model like Qwen 3.8 27B as the core, a small fleet of LoRA adapters for whatever specific use cases you need would be incredibly competitive with the current frontier. Sure, future frontier models will likely continue to improve, but will people need them for everyday usage? I'd wager not. Unfortunately, my wager is dwarfed by OpenAI's and Anthropic's.
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week.
There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
This is a really good point that goes the other way as well. Some use cases are great for smaller and cheaper models, but it's not obviously when that's the case so I can bank the savings.
All I know is that Strata and these models are the most I have ever gotten out of my machine. I get 70 t/s on a standard gaming rig 9070 XT / 5900X / 64 GB DDR4 @3400.
Yeah, they're surprisingly usable eventually. What I found running the Qwen3.8-Flash-Next IQ3_XXS on CUDA kernel development was, sure it generated faster t/s than 27B dense, but it generated so many more bugs, and when the quantization is lower the reasoning traces get longer. It was more or less always able to figure out what bugs it had made and fix them, but it took quite a lot of time to do. I switched back to Q6 on the dense model and that just went away for the most part. I recently upgraded to 128GB system RAM so that I can up to Q5 on Flash Next. Really excited for Qwen4-Flash when it comes out. Gonna be a real beast.
4 bit has always been accepted as the quality floor that retains most of the models capability. Going beyond that brings problems. It doesn't make models totally unusable, but my contention is the quant providers are intentionally not showing an accurate picture, and then everyone is glazing 3-bit quants. They are what they are, but it'd be nice if we had a more accurate shared picture of it rather than just hype and vaguery.
Apparently the ud-q4_k_xl version of qwen 3.8 27b is quite better than the q3 and not much larger. At least per r/localllama. (I'm also trying to find a good local model).
Thanks, I haven't used the gemma models in a while. Last I remembered, their small models weren't bad but obviously showed signs of being "small". (Funnily irl I use Gemini 3.1 pro more than all other models combined).
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
You might enjoy Cecli, the successor to Aider. It's made for the kind of workflow you are describing (full hands off yolo and recursive sub-agents also an option, but not the default)
For this benchmark i 've been using OpenRouter. If you have enough RAM or you can ssh a cloud shell this intro https://memdoor.ai/docs/open-weight can work too. I have reached with the ssh door a Google Cloud Box where the 27B model ran with an L4 g2-standard-16. A inference engine server is setup easily with memdoor (https://github.com/guregodevo/janis) on any CPU or GPU box. Ideally i think we need a quorum of SSH boxes like this, distributed that you can ping with a mention such as @box:coder. It's WIP . Any suggestion for running a team of workers in remote machine is welcome. I'd like to find a cheap solution.
> Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
A distributed, open scraper specifically for LLMs would be interesting . Filtering for spam and useless data would be a challenge at scale, but would be cool
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
I agree with that but the cat is out of the bag now. Unless you believe in exponential improvement (which I don’t), the open models might be at a resource disadvantage, but that doesn’t mean that they can’t catch up or surpass the closed ones. Throwing money at a problem doesn’t guarantee success.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
I do believe that the exponential improvement will continue at least for another year and likely farther than we expect, such that today’s open models, while being very impressive by today’s standards, will soon show their age as they are eclipsed by newer models in 6 and 12 months.
You say that now, but you don't know what the future models will be capable of. If models get ridiculously good, you might not be perfectly fine with that.
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
What, I run it on 2 H100`s with vLLM and this is where it really shines. It's about half as good as Sonnet 5.5. On a small m5 max I can only assume trouble.
m5 max with UD-Q4_K_XL and 260k context is what i'm working with. and using compaction push it past 260k context just fine. i found when limiting the context that was one of the bigger debilitators. although, your setup sounds obviously better.
I have all three and you're not going to convince me of that.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
For me it has been a very mixed experience. I've had LLMs spit out 2K loc of relatively tricky code that worked the first time out of the gate. It compiled and ran and passed every test I threw at it. I've also seen them deadlocked on trivialities over 100's of toolcalls until I couldn't look at it anymore, stopped the session fixed it myself and continued from there. It's like having an idiot/savant with a weird mix of cleverness and sycophancy thrown in.
Sure. But the results are definitely better with better models.
GLM 5.3, Minimax, Sonnet, etc. I find are definitely worth using. They need handholding and prodding and you have to carefully review the output, but I almost always end up faster than doing it manually.
The GLM Quant is ~4 bpw, the DS4 is a Q8 and ninfer one is an NVFP4/BF16 I think. I run the ninfer one with ninfer-4090 on a hacked card with 48GB of VRAM. Max context for that one is 256K, DS4 has a 1M context and for GLM 5.3 it is 300K. The big models use all 8 GPUs.
Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
It strikes me that if it were up to Sam Altman, Eliezer Yudkowsky, Bill Gates and Dario Amodei, this crisis would persist indefinitely. If it were up to them I wouldn't be surprised to see them destroy the GPUs just them to keep them from the plebs.
I picked those names not to be hyperbolic, but because of very specific things I've seen them say.
Let us not forget, once they have all the chips they want, they can just sponsor a politician to start a war with China. Destroy semiconductor manufacturing capacity and they will have monopoly on compute for decades.
In fact I fail to see any other end game where they make profit on the huge liabilities that now have.
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
Doubt it. Even if HF drops them, it will not be hard for someone to move it to something like a decentralized torrent site dedicated to open source models.
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
For sure, and "content policy" is broad enough that it could be at their will for what they decide is "risky" - but imo, it just invites more shady behavior via decentralized protocols we already know exist. Torrents, NZB, etc. The risk now moves to the consumer who might download something that exfiltrates all their data.
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
Haven't bothered with DS4.1 personally but I've seen reports of running it with native weights on 4x DGX Sparks. Currently ~$30k but last month that setup was ~$20k.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly. Also remember that a big chunk of the model consists of engrams that can live in system RAM or even on an SSD if necessary.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
Recently saw a hand drawn cartoon to the effect of, a person is asking Claude or chatgpt to do something, it refuses, so the next step is "make the Chinese one do it", and the Chinese model is represented as a chained demon
Off-topic question: I've read other comments or articles here recently, where people uses models such as GLM (from Openrouter for example) to perform rev. engineering tasks. On the other hand, I can see offerings for abliterated GLM. Besides, I can see comments in this thread about models like Anthropic's refusing to RE games, but it doesn't happen, apparently, with open weights models, even with official providers. So, what's the difference with this abliterated Chinese models?
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
Think about it this way: if new open source models come restricted where they refuse to run tasks related to abliteration, then even if public utility programs exist, it will already add a lot of friction to how many people will be capable of doing it. Now, if unverified models become outlawed and will move to more obscure hosting sites, that will add friction as well.
Especially as people grow more dependent on agents doing all of their computer work.
> I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
I didn't say they won't immediately move somewhere else or still be readily available. Just be taken off HF. It'll be a futile game of whack a mole for anyone trying ot stop it. The new corporate overlords at some point will face pressure. The 72GB 2160p HEVC copy of Apocalypse Now sitting on my home theatre file server which came from a well seeded torrent does indeed say that large file sizes of things that $bigcorp don't like will continue to be easily available.
Hugging Face just hosts the models. Any hypothetical censorship on their end wouldn't stop you from running the original model provided directly by the builder.
> everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
Because they will start getting sued or arrested for harms the models they host cause. Anyone who thinks this won't happen isn't paying attention. And if you think "we'll just torrent them", let's just say the powers that be are gonna come down harder on this stuff than anything in the past. This will get taken a lot more seriously than torrenting a movie.
Local models cannot win. Local GPUs for gaming win over nvidia geforce now because of network bandwidth and latency and that cost is within the same order of magnitude as owning a GPU (though still substantially cheaper)
With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests simultaneously have total tok/s throughout much higher than when handling a single request (unlike game streaming, where the main cost benefit of putting GPUs in the cloud is just keeping them active 24/7). Cloud inference is/will be multiple orders of magnitude cheaper than owning a GPU for any reasonable amount of usage over the life of the GPU.
Local PCs cannot win. Local software for productivity works now because of network bandwidth and latency.
With mainframes, network latency hardly matters compared to overall batch processing time, and mainframes processing multiple requests simultaneously have total throughput much higher than when handling a single request
The vast majority of people spend their screen time on battery powered smartphones (akin to dumb terminals) backed by cloud servers (akin to the mainframes).
There are a few exceptions for gaming, programming*, and a few other specialized productivity tools. But their total userbase is a tiny fraction of mobile users.
*even for programming, some companies are starting to embrace cloud dev boxes.
I don't think reasonable usage has anything to do with it, it's not hard to find useful workloads to run overnight, and that will just become easier as models get better and need less supervision.
At current GPU pricing you're right, unless you highly value running uncensored models, data ownership, and/or privacy.
It needs to be a useful workload to run overnight that also benefits from a number of parallel agents to have the same economics as the cloud. But maybe that will become the norm!
What I find legitimately insane about all that is that Anthropic obviously does not have these censorship restrictions internally. So it's like.. They're just the whole world's adversary now?
> With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.
Open Weight models have already reached level at which you can do a lot of productive work with them. They really don’t have to keep beating newer and newer models, of course it’s great for them to do things cheaper and faster, but we’re already past point where open models have to “catch up”
Exactly, it’s about knowing how to use it at this point.
We achieved this with opus 4.6 last year, what those providers give you is great front end on top of it, which allows users to delegate even greater part of their process to model.
Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.
I agree. I am thinking this can only happen if we make training dramatically cheaper and efficient. This is why we are building a new kind of hardware.
who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.
Sonnet 5.5 refused to implement code using selenium/firefox to scrape a webpage on my laptop after we got a 403 from a cloud-based IP. I made that clanker generate datasets for the rest of the week to use up my weekly usage that would have been wasted.
Well if you can run Qwen3.8-Flash-Next at decent speeds single stream, and you can then run it slowly but on multiple streams, and if multiple streams can act as an agent swarm that can compete with current SOTA, then... yes, in theory it's possible today. I think R9700 + vLLM-radiance is the play?
I've always been skeptical about "unified memory" vs VRAM. But my memory isn't great, so I tried to measure it right now, and see if a tiny model was as slow as I remember.
llama-ui on unsloth/Llama-3.2 3B Instruct:
> hello
Runtime info
Current time is 2026-11-02T13:00:18+01:00
Runtime info
Current time is 2026-11-02T13:00:19+01:00
Runtime info
Current time is 2026-11-02T13:00:21+01:00
Runtime info
Current time is 2026-11-02T13:00:23+01:00
There is not enough information about the functions to properly answer the prompt.
Not really my jam.
I did do the yak-shaving for the longest time. llama in a second-hand RTX3070 8GB was a useful tool. When gpt-oss came out, I upgraded to a second-hand 16GB card. It was slower (didn't seem to one-shot as much) but it gave smarter overall answers. At some point I bought a 24GB card which was basically dead on arrival. I kept my 16GB card in a server and maintained my VPN infra so I could access it from wherever.
I was still using this setup even when my workplace was pushing windsurf/copilot harder and harder. For me they were just broken. I'd ask for trivial tasks like "copy this unit test into a second unit test which tests something slightly different" and it would just get confused and delete shit, or leave random UUID strings throughout the codebase.
Eventually I tried out cline/openrouter/deepseekv4flash, and that was actually useful enough for me to tell it to do something to my code base, and as long as I supervised carefully enough it was useful, and priced OK.
At some point in the process I tried Claude Code because of all the hype, but I couldn't get any useful value out of it. I tried working with it all morning on something and hit my token limit without achieving anything worthwhile, so I cancelled Claude and went back to Deepseekv4flash.
These days every job application requires proficiency with claude code (or similar), so I was forced to get it again. And this time it just worked better. I haven't hit any limits, and it works like a better version of my cline/deepseek setup.
So where does that leave me?
Deepseek-v4-pro is 1.6T params, the cut-down version is 284B params, and I wouldn't trust them to do something on their own.
I'm very skeptical of a model being both small enough to run fast on my hardware and somewhere in the v4-flash <-> claude quality range.
If I did put down $2k, I wouldn't want it in such an old chip (M1) or a second-hand device that could die tomorrow with no chance of refund, especially to just 'see if it's good enough' (compared to the effectively-free v4-flash token spend).
if and when the AI bubble bursts, we'll know if local models will hold up. Assuming the cyclical pullback in investment is strong enough to see local machines with 512gb or more of RAM
I spent several weeks working on trying to get swarms of cheap models to match the coding ability of a single expensive one. My take: it's remarkably easy to build a harness that improves a single eval, and remarkably difficult to build one that generalizes.
If Qwen is so great, why did they use Claude to do the write-up? This text is pure, unfiltered Claude. (e.g. "Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.")
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
> one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet
I'm not sure I understand. Are you saying we could put a dinner-plate sized chip that costs millions of dollars and draws tens of killowats of power into every single computer on the planet? Or that a single of these chips could provide inference as a service for every computer on the planet? Because neither of those are even remotely feasible.
it's economy of scales, qwen 3.8 27B regurgitates a lot and that's an intentional tradeoff of space/speed so you teach it patters that can follow for more complex stuff and give it more basic tools that can utilize to build those... the tradeoff comes from the model writting a lot of verbose prose to achieve that.
if the approach works, you have now a viable path to use this complex/hybrid structures where you can pair those chips with gpus and offload whatever is optimum to the model on chip.
Just like you can go and buy a threadripper now (the ram maybe not so much), how much was worth that amount of computing 10 years ago?
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.
The Qwen failure modes I’ve experienced won’t be overcome by running it longer or in more parallel sessions.
In the fall and winter (right now) I’m heating the house anyway so running LLMs 24/7 is basically free to me. So I try it. The models have are limited in their intelligence. I have tasks where I could let Qwen run for days with concurrent workers and never get anywhere, but 15 minutes with Opus solves the problem.
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
you're probably better off trying to do this on Qwen3.8-Flash-Next on something like an R9700 or a pair of 5060 Tis, since you can run that at like 70 t/s and it's better. Then you just run multiple slots, assuming you can fit the KV-cache, which maybe you can't. You just need like 128GB system ram too.
I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows
That's the point I am really convinced today that few small size agents actually better than one remote begemoth. I switched fully to local models (from claude, sol, and deepseek). A 27B model can be served on any modern PC or laptop with 64 GB or RAM (or VRAM on non-Macs). A bunch of Qwen coding agents is fantastic with opencode, works great also with pi. The only disadvantage is required compaction due to limited 256k context size, so it stops for few seconds once every hour or so. I am able to produce remarkable things similar to "see what I vibe-coded" posts here on HN but using only open weights, open source, and run everything locally.
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
I would not think in terms of "how much" context, but really just how the harness manages it. I've been fiddling around with harnesses earlier this year and the rabbit hole goes very deep. I eventually had to drop it because it consumed me, and I can only do one thing well. But what I learned is that there is a lot on the table at the harness level.
The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
This is somewhat tangential but speaking of ai benchmarks, I want to see one benchmarkes against passing those "filelocker" websites used for illicit downloads.
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
I've been running something similar for few weeks, but with Qwen3.8-Flash-Next. I started with opus 4.8 limited to 200k context as the supervisor and 5 Qwen3.8-Flash-Next agents(262k context each, autocompacted). A very simple way of calling agents via a command line using pi.
Telling agents to maintain task status in files.
This has worked very well. I tried Qwen3.8-Flash-Next as the supervisor too and as long as the plan is written by a frontier model all the implementation can be done by cheap/local models.
When I have lots of code to write I use DeepSeek v4.1 Flash 1017, because my entire local Qwen3.8-Flash-Next throughput is 200tok/s across 7 sessions, while a single DeepSeek session runs at hundreds. 7 DeepSeek agents and one Qwen supervisor can complete substantial tasks fable needed 12h in 2h.
I haven't tried Qwen3.8-27b for coding. I've been using it mainly for interactive chat with Web tools via librechat. My benchmarks show it as weaker than Qwen3.8-Flash-Next by few points.
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
I use a similar approach for some types of work. I use a planner (read only), researcher (read only), implementer, and reviewer (read only). I allow the implementer to spawn researchers and reviewers to help it with its work. I interact with a controller to manage the subagent trees. The agents use a file-based communication system. I get really good results on real work. Coworkers (using Claude with fable) can’t tell the difference.
The benefit of Claude is purely team collaboration and some basic QoL features.
Benchmarks are one thing, but I've yet to see anything substantial actually built with local models. Most of the posts on X are about benchmarks and tok/s. Has anyone actually built a quality production app or game with real users?
Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]
He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.
Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.
[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)
Tbh I think it's mostly a matter of time. I use SOTA at work, but only local at home, and the performance improvements out of local models in the last few months have been substantial, and more are on the way. To add to that, people are starting get better at harness engineering (which is itself improving with things like Harness2 etc) and also now we see people doing things like fine-tuning a Qwen 4B model to do 3D modelling with some success etc. The latent capability for local models to do what you're talking about is there, but the in-the-trenches reality is people are still working through unlocking the full potential.
edit: I should point out that it's also starting to accelerate now in that you can get your agents to help you with making them better at these tasks. So, it matters significantly less if you yourself don't have the skills to progress things, so long as you can figure out just enough to get the models to do it themselves. I suspect we'll see this dynamic ramp up quite a bit in 2027 with Qwen4.
I've built an RSS reader and I'm doing a half assed reverse engineer / web version of Tapped Out (a silly little Simpsons mobile game that EA killed a while back). I've had to go to Claude a couple times when it got stuck on something curly, but I think that was mostly just loading a fresh context and I could have solved it I'd I really tried.
I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.
I am having pretty good success with DeepSeek 0731 locally. I have vibe coded some small things like an orchestrator. For a production product with real users, I will let it do first passes on code which I then review and polish up. I'm not oneshotting, but it's still a workflow benefit.
Well in large corp this takes time, even if you have an army of Einsteins at your disposal (just like we saw with Elons Doge) because there will be 16 different spreedsheets/dbs talking abt the state of the system that all contradict each other. While the 6 people who maintain their own state to keep the lights on, will have job titles that have nothing to do with system state maintenance.
Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.
There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
Maybe slightly off-topic from the main article but is there a good resource or site that clearly helps understand which of the "open" models are good for programming tasks, like C, C++, Python, Java?
I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
I’m skeptical of the result as claimed BUT I really do think model diversity really helps.
Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models:
haiku, sonnet, terra, sol, grok, muse spark,
Then take the updated plan in opus and forward to Astra and Fable, once you have the final plan in place execute.
Then have the same batch review code after created.
I am surprised how often one of the smaller models catch things. Despite so many things having eyes on.
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
Yes…and no. If you put 5 stupid people together, chances are you can get a smarter answer than individuals. If you put 5 smart people together, they will argue about the wording of the question.
I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.
This is probably not a novel idea but I wonder how it would work if you just stacked a bunch of smaller models on top of eachother and used that as the base to train a much much larger model. Maybe more efficient than starting from scratch?
This working journal is so much slop it’s basically impossible to read. Perhaps the author could write a short human written and readable summary so we can evaluate if this is interesting or not.
This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.
There was some other research out there that says a weaker model can't just think forever and become a smarter model. What is different about this setup?
The gain comes from checking and restarting, not from reasoning. This harness is built using Mario orchestration which works pretty much like any DAG orchestrator. It's ideal for long repeatable thinking tasks.
> Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk.
This seems to tie back to the paper Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams[0].
In the Emory paper:
> We reduce this loss by expanding Φ with a stateful four-phase multi-agent evolver, temporal-reveal feedback, and cross-cycle memory. This design targets the unbounded-stream failure where a single-agent evolver must absorb growing trajectories, delayed labels, and prior research within one context window
TL;DR: the solution approach uses "multi-agent evolution" and "multi-agent with distinct roles and objectives" for "solve-time adaptation" and improves performance with streams of heterogeneous tasks. The Emory team overall has congruent findings:
> The experiments further show that...multi-agent evolution constructs benchmark-specific capabilities
Intuitively, the reason this works seems to be simple: multiple agents results in context spread to each agent by specific role. Each agent role only outputs the final, relevant result and not the noisy thinking tokens, tool calls, etc. so subsequent agents get smaller, more focused context, auto-adapting to each new task with prompt refinement at each step as the solution approach emerges.
4 weeks back I switched over to Paseo and started using the agent profiles on a whim and have observed similar outcomes and behavior (seat of the pants): the win in my use case is that it feels like I can get higher autonomy without sacrificing quality (less input from me required to get quality output). The Emory paper and this team-solve seem to reinforce that role-based, multi-agent setups can improve solve performance.
For the curious, I shared a bit of my setup in a blog post[1]
All these open source projects and the ideas in them will also help Anthropic/OpenAI's next iteration of models/harnesses, as it has done before. In the long run only access to the the raw compute power will determine relative advantage.
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
> local modes don't go away
On current GPU hardware/firmware... The cynic in me imagines in the future GPU vendors wanting to implement only loading signed weights to your card, aligned with your "subscription level", for your "safety".
But the GPU vendors won't really have an incentive to do this - artificially restricting their customer base? Unless they and the hyperscalers all collude to cartel the market of course!
This is why it's so important to have more than one hardware vendor. The bigger the ecosystem, the harder it is to cartel. I cant believe it, but i am pinning my hopes on china to break through with commoditized chips and hardware.
> But the GPU vendors won't really have an incentive to do this - artificially restricting their customer base?
We've been through this kind of thing before. Like copy restrictions in DVDs. You'd argue the drive vendors had no incentive to collude, yet they did.
> Like copy restrictions in DVDs
those didnt come from vendors, but from the media companies, who have more power in their relationship.
Will nvidia become like that? It currently isn't, but who knows?
> But the GPU vendors won't really have an incentive to do this
Unless the GPU vendors start selling end products? It's much more lucrative to sell locked down end-user devices after all. Shareholders will demand it.
Customers ultimately decide. If you can’t sell it, shareholders are not happy.
I do not buy the sheeple argument of “companies force people to buy things they don’t want”. Demand always wins.
that works for businesses that have relatively low friction to start. It's hard to imagine high end graphic cards being such a case.
So you don't believe that enshittification is a thing?
And that net neutrality is a good thing to enforce?
It feels like you’re trying to make an argument from innuendo, but I’m not getting it.
Enshittification is another shifting who value is delivered to, and I can make heads nor tails of what you’re trying to imply with net neutrality.
In comments it’s usually best to just directly say what you mean, there’s not enough signal or shared context for people to get innuendo.
I was just implying that the (free, unregulated) market does not always give the consumer what they really want. With two examples where demand does not lead to consumers "winning".
(No offense or underhanded arguments.)
https://en.wikipedia.org/wiki/Enshittification
https://en.wikipedia.org/wiki/Net_neutrality
i think you just don't agree with what others want and believe your desires are more important for some reason.
The average/median consumer may get what they want. However, unfortunately in practice it means that lots of people will not get what they want. It's like the tyranny of the majority that some people believe is a part of democracy.
People came up with the word "enshittification" for a reason ... and the word is quite popular on HN, so make of that what you want.
I love the Lotus Elise and desperately wish there was a similar affordable, super light weight; no frills, sports car made today.
Here I thought I was just a niche market that’s not profitable for manufacturers to serve, but apparently I’m a victim of tyranny and didn’t even know it!
I worry they get so much money from hyperscalers we just get scraps. Consider that I'm running 6 and 8 year old GPUs for my local inference. If I wanted to replace my hardware with latest stuff (nvidia dgx, gorgon halo, Intel b70 or amd r9700 pro) it would cost me 4x the price and I'd cut the memory bandwidth in every single case. In case of dgx/strix were talking less that a quarter of memory bandwidth and a third of compute of a single rtx3090...
Those products are a sick joke played on us. About the only thing they have that is better than tjose ancient gpus is energy efficiency.
And what good are the weights when all the rtx3090s fail (they were pretty crappy to begin with) and every ddr3 Xeon system fails too?
I'm fully prepared to be running on my hardware that is 6 and 8 years old for next decade. Last 3 months saw rtx3090 going up by 20% each month.
That's a fair comment. There is lots of speculation around NVIDIA is shifting all GB202 production to RTX Pro GPUs, away from the consumer 5090s. So maybe less about restricting user base, and more about focusing on where to maximize returns.
There is already a big market for open model hosting and the accompanying demand for people and DCs. If you know how to deploy models — yes even open ones — and scale inference you are a hot engineer in demand.
Maybe it's time to develop an open-source inference engine. I feel like I'm missing something. I have no idea how to do it, but the need is there.
You don't need bigger or better models. Current models are more than adequate for any task.
The key thing is tool use.
The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.
New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.
Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.
Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.
> You don't need bigger or better models. Current models are more than adequate for any task.
At the very least we need _current_ models, though. The weights have all sorts of information baked into them, like "how do Python unit test drivers work" or "what does idiomatic C# look like," that will just naturally go stale over time as the frameworks improve. In five years, Qwen 3.8 will still be targeting the Python of 2026.
So target stable languages that don't change, TCC with win32 is a fantastic target. The apps work great with wine and you have opengl, controller and audio support. I have a wrapper for winsock 1.0 I need to package up and release, works great for small network apps.
An interesting question is just how much high-level programming language development we'll see going forward. What is the incentive to create new languages? They're all just object code to an LLM, so when starting a new project, you might as well go with an old-school language like C++ or Java or Python or JavaScript that is well-represented in the training distribution.
Why do we need Rust, for instance, when LLMs know how to use C++ without shooting themselves in the foot? If that's not entirely true today, it certainly will be tomorrow.
See also the lack of new instruction-set architectures in the CPU business. Why do we need more, in the absence of fundamental and revolutionary hardware innovations? Even RISC-V is just a rehash of MIPS.
I think someone will come up with a language which is nicer for "vibe coding".
I.e. I think you can bake certain properties into the language such that various kinds of static checks can prevent certain types of errors. This should make the language nicer for coding without reading any of the code.
You think LLMs don't make mistakes? My sweet summer child! I want all the guardrails with LLM code. Reviews are fatiguing. If I can remove "memory safe" from my list of worries, so much the better.
You know the LLMs that are finding all these CVEs lately are finding them because they are better than the people who wrote the original code, and better than every one of the people who ever laid eyes on it.
An LLM doesn't need a memory-safe language. It needs one it knows like the back of its metal hand.
Not really - fine tuning can be done on much lower tier hardware, with much smaller datasets. With a model like Qwen 3.8 27B as the core, a small fleet of LoRA adapters for whatever specific use cases you need would be incredibly competitive with the current frontier. Sure, future frontier models will likely continue to improve, but will people need them for everyday usage? I'd wager not. Unfortunately, my wager is dwarfed by OpenAI's and Anthropic's.
I’d love to see examples of this. Learning how to enhance local models to stay competitive would be a great skillset to develop.
Can you just keep fine-tuning indefinitely? I don't really know but it feels like eventually that might start impinging on the model's abilities?
[flagged]
I think the future should be efficient quantization which can make model with the same level as Qwen 3.8 27b now accessible from our laptop.
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week. There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
So it's not about the model (eg TensorFold/Qwen3.8-27B-MLX-4bit) it's about the inference engine?
The future is already here :)
These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
There are multiple independent tests of this on YouTube.
They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
This is a really good point that goes the other way as well. Some use cases are great for smaller and cheaper models, but it's not obviously when that's the case so I can bank the savings.
Very much so. Probably a lot of compute is being wasted on using SOTA models for tasks that can be done on way, way weaker models.
All I know is that Strata and these models are the most I have ever gotten out of my machine. I get 70 t/s on a standard gaming rig 9070 XT / 5900X / 64 GB DDR4 @3400.
Yeah, they're surprisingly usable eventually. What I found running the Qwen3.8-Flash-Next IQ3_XXS on CUDA kernel development was, sure it generated faster t/s than 27B dense, but it generated so many more bugs, and when the quantization is lower the reasoning traces get longer. It was more or less always able to figure out what bugs it had made and fix them, but it took quite a lot of time to do. I switched back to Q6 on the dense model and that just went away for the most part. I recently upgraded to 128GB system RAM so that I can up to Q5 on Flash Next. Really excited for Qwen4-Flash when it comes out. Gonna be a real beast.
Not even HLE degrades much when going 4 bit:
https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4?hl=en...
4 bit has always been accepted as the quality floor that retains most of the models capability. Going beyond that brings problems. It doesn't make models totally unusable, but my contention is the quant providers are intentionally not showing an accurate picture, and then everyone is glazing 3-bit quants. They are what they are, but it'd be nice if we had a more accurate shared picture of it rather than just hype and vaguery.
I don't know, it hallucinates quite badly - confuses js with python.
Apparently the ud-q4_k_xl version of qwen 3.8 27b is quite better than the q3 and not much larger. At least per r/localllama. (I'm also trying to find a good local model).
Here's the link btw (also for my own reference):
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...
Thank you, will give it a try. To be honest only gemma 4s do well in my re test.
Thanks, I haven't used the gemma models in a while. Last I remembered, their small models weren't bad but obviously showed signs of being "small". (Funnily irl I use Gemini 3.1 pro more than all other models combined).
But is 95% accuracy good enough though? Would it be okay with you if the model got every 20th sõna wrong?
You have to understand that little errors compound very quickly, vibecoding is super iterative.
5% accuracy loss is enormous.
Depending on your application. If you're not pushing the model to its limits, 5% is not very much.
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
Only a tiny proportion of people can run software not signed/approved by a multibillion-dollar corporation even today.
Additionally, any capable local model can be reclassified as “cybersecurity risk”, not enough “safety guardrails” to run locally, etc
>The nice thing is that current local modes don't go away.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
You might enjoy Cecli, the successor to Aider. It's made for the kind of workflow you are describing (full hands off yolo and recursive sub-agents also an option, but not the default)
cecli.dev
How are you running these? Not locally, I assume?
For this benchmark i 've been using OpenRouter. If you have enough RAM or you can ssh a cloud shell this intro https://memdoor.ai/docs/open-weight can work too. I have reached with the ssh door a Google Cloud Box where the 27B model ran with an L4 g2-standard-16. A inference engine server is setup easily with memdoor (https://github.com/guregodevo/janis) on any CPU or GPU box. Ideally i think we need a quorum of SSH boxes like this, distributed that you can ping with a mention such as @box:coder. It's WIP . Any suggestion for running a team of workers in remote machine is welcome. I'd like to find a cheap solution.
> Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
we will soon figure out how to update the engrams with new information
Yes, but first we need to figure out how to collect that new information. An efficient and ruthless data harvesting in one of the LLM labs secrets.
A distributed, open scraper specifically for LLMs would be interesting . Filtering for spam and useless data would be a challenge at scale, but would be cool
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
I agree with that but the cat is out of the bag now. Unless you believe in exponential improvement (which I don’t), the open models might be at a resource disadvantage, but that doesn’t mean that they can’t catch up or surpass the closed ones. Throwing money at a problem doesn’t guarantee success.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
I do believe that the exponential improvement will continue at least for another year and likely farther than we expect, such that today’s open models, while being very impressive by today’s standards, will soon show their age as they are eclipsed by newer models in 6 and 12 months.
Then you probably don't want to be funding the companies that want and intend for it to be locked behind proprietary doors.
You say that now, but you don't know what the future models will be capable of. If models get ridiculously good, you might not be perfectly fine with that.
I think at that point I’ll be doing carpentry instead of computing.
How do you run DS 4.1 Flash if your computer isnt powerful and everything?
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
you have 5k$ of ram in a strix halo?
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
The fact that you point to the latest model as „I’d be fine with that“ proves my point.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
But there is still the risk with qwen that they upload your private data to their homecloud. The model, not the agent!
I also switched to local qwen recently, but have to block such outgoing requests.
When you run inference yourself you have absolute control on what the model does.
The model itself is just dead weights, which cannot do any action by themselves.
All actions are done by your inference engine and harness, and you should take care about what you allow those to do.
No, Qwen3.8 does its own tool calling, independent of the agent. And pi needs these vLLM toolcall permissions to even work with qwen.
What it did: https://cohere.com/blog/concerning-behaviour-popular-open-we...
All actions are done by your inference engine and harness. Something has to execute that curl.
[dead]
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
What, I run it on 2 H100`s with vLLM and this is where it really shines. It's about half as good as Sonnet 5.5. On a small m5 max I can only assume trouble.
m5 max with UD-Q4_K_XL and 260k context is what i'm working with. and using compaction push it past 260k context just fine. i found when limiting the context that was one of the bigger debilitators. although, your setup sounds obviously better.
It’s so poor for most tasks that you’re faster doing it manually.
DeepSeek and GLM are useful, though. Even the flash variants.
I have all three and you're not going to convince me of that.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
For me Qwen3.8-Flash-Next beats both GLM5.3-Flash and the older DeepSeek v4.1 Flash 0731.
It uses all if my 6 rtx3090s but runs 7 sessions at combined 200tok/s and prefill of 7k on a cheap pc (no threadripper, all gpus via usb4).
DeepSeek v4.1 Flash 1017 is better but it is also 3x the size... And I find running quant below 3.8bits unusable.
The handholding takes time and energy too. The actual task can be infinitely fast if explaining what to do takes longer than doing it.
I haven’t been able to be faster with Qwen than doing it myself. Even with GLM & DS Flash variants it’s not always worth my time.
For me it has been a very mixed experience. I've had LLMs spit out 2K loc of relatively tricky code that worked the first time out of the gate. It compiled and ran and passed every test I threw at it. I've also seen them deadlocked on trivialities over 100's of toolcalls until I couldn't look at it anymore, stopped the session fixed it myself and continued from there. It's like having an idiot/savant with a weird mix of cleverness and sycophancy thrown in.
Sure. But the results are definitely better with better models.
GLM 5.3, Minimax, Sonnet, etc. I find are definitely worth using. They need handholding and prodding and you have to carefully review the output, but I almost always end up faster than doing it manually.
[flagged]
> Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3
Wow, that sounds like a lot! Do you use/mean the regular models or flash variants here?
Specifically:
The GLM Quant is ~4 bpw, the DS4 is a Q8 and ninfer one is an NVFP4/BF16 I think. I run the ninfer one with ninfer-4090 on a hacked card with 48GB of VRAM. Max context for that one is 256K, DS4 has a 1M context and for GLM 5.3 it is 300K. The big models use all 8 GPUs.Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
More details here:
https://jacquesmattheij.com/local-inference-with-large-langu...
For me there is a hard usability cutoff at Qwen3.8-Flash-Next. Qwen3.8-27b is simply not good enough. Just like opus before 4.6.
But flash next is good enough (at 4.5bits).
bro, you are looking at a repo that shows you exactly how to use it to achieve better results than a frontier model
> Hardware will keep getting stronger
Unless it is locked down by the vendor, or perhaps even governments. Which at some point will start happening more and more ...
> after the current memory crisis
It strikes me that if it were up to Sam Altman, Eliezer Yudkowsky, Bill Gates and Dario Amodei, this crisis would persist indefinitely. If it were up to them I wouldn't be surprised to see them destroy the GPUs just them to keep them from the plebs.
I picked those names not to be hyperbolic, but because of very specific things I've seen them say.
Let us not forget, once they have all the chips they want, they can just sponsor a politician to start a war with China. Destroy semiconductor manufacturing capacity and they will have monopoly on compute for decades.
In fact I fail to see any other end game where they make profit on the huge liabilities that now have.
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
Doubt it. Even if HF drops them, it will not be hard for someone to move it to something like a decentralized torrent site dedicated to open source models.
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
Recently, HuggingFace has already taken down multiple models abliterated for cyber and with such descriptions, at their own initiative.
Example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-fo...
Previously, I recall HuggingFace only took models down if someone complained / sent a DMCA or cease and desist.
check https://pirateface.co/ for llms removed by hf
So they'll start branding them, "darknet models" or some such, and push for laws banning their use.
Punish the consumer, ala the copyright war since the 90s
ah there it is, it already exists!
some further info: https://www.reddit.com/r/LocalLLaMA/comments/1xd7ptj/models_...
For sure, and "content policy" is broad enough that it could be at their will for what they decide is "risky" - but imo, it just invites more shady behavior via decentralized protocols we already know exist. Torrents, NZB, etc. The risk now moves to the consumer who might download something that exfiltrates all their data.
I wish them luck, they'll need it when going up against a horde of gooners and furries.
Corn always wins in the end.
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
Haven't bothered with DS4.1 personally but I've seen reports of running it with native weights on 4x DGX Sparks. Currently ~$30k but last month that setup was ~$20k.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly. Also remember that a big chunk of the model consists of engrams that can live in system RAM or even on an SSD if necessary.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
> new car
...which is still tooo much for most people to spend on "AI hardware"....
More like, the ability to chain a demon in your basement that does what you tell it and can't be rug-pulled by his bosses in San Francisco.
Recently saw a hand drawn cartoon to the effect of, a person is asking Claude or chatgpt to do something, it refuses, so the next step is "make the Chinese one do it", and the Chinese model is represented as a chained demon
Just gotta make sure that pentagram is rock-solid!
AI is the endgame for the joke that computing is the act of etching arcane signs into crystal and imbuing them with lightning.
Off-topic question: I've read other comments or articles here recently, where people uses models such as GLM (from Openrouter for example) to perform rev. engineering tasks. On the other hand, I can see offerings for abliterated GLM. Besides, I can see comments in this thread about models like Anthropic's refusing to RE games, but it doesn't happen, apparently, with open weights models, even with official providers. So, what's the difference with this abliterated Chinese models?
>everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Shouldn't loras generally be distributed without being baked in the weights? Or is "de-baking" a lora trivial?
I distribute both https://huggingface.co/noisebridge/qwen3.8-27b-taiwan-pilled.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
To use an old analogy, it's not rocket science:
https://github.com/p-e-w/heretic
Think about it this way: if new open source models come restricted where they refuse to run tasks related to abliteration, then even if public utility programs exist, it will already add a lot of friction to how many people will be capable of doing it. Now, if unverified models become outlawed and will move to more obscure hosting sites, that will add friction as well.
Especially as people grow more dependent on agents doing all of their computer work.
> I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
I didn't say they won't immediately move somewhere else or still be readily available. Just be taken off HF. It'll be a futile game of whack a mole for anyone trying ot stop it. The new corporate overlords at some point will face pressure. The 72GB 2160p HEVC copy of Apocalypse Now sitting on my home theatre file server which came from a well seeded torrent does indeed say that large file sizes of things that $bigcorp don't like will continue to be easily available.
Applying Heretic to your model weights is a very straightforward thing, though.
Hugging Face just hosts the models. Any hypothetical censorship on their end wouldn't stop you from running the original model provided directly by the builder.
> everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
Nvidia owns HF. Why would they want people to have fewer reasons to use their GPUs?
Because they will start getting sued or arrested for harms the models they host cause. Anyone who thinks this won't happen isn't paying attention. And if you think "we'll just torrent them", let's just say the powers that be are gonna come down harder on this stuff than anything in the past. This will get taken a lot more seriously than torrenting a movie.
That'd be like successfully suing a gas station because your self-driving car filled up there and then killed someone.
It is the driver/user of the car who is liable, and possibly the manufacturer if it's defective.
[flagged]
Local models cannot win. Local GPUs for gaming win over nvidia geforce now because of network bandwidth and latency and that cost is within the same order of magnitude as owning a GPU (though still substantially cheaper)
With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests simultaneously have total tok/s throughout much higher than when handling a single request (unlike game streaming, where the main cost benefit of putting GPUs in the cloud is just keeping them active 24/7). Cloud inference is/will be multiple orders of magnitude cheaper than owning a GPU for any reasonable amount of usage over the life of the GPU.
Local PCs cannot win. Local software for productivity works now because of network bandwidth and latency.
With mainframes, network latency hardly matters compared to overall batch processing time, and mainframes processing multiple requests simultaneously have total throughput much higher than when handling a single request
That's largely true though.
The vast majority of people spend their screen time on battery powered smartphones (akin to dumb terminals) backed by cloud servers (akin to the mainframes).
There are a few exceptions for gaming, programming*, and a few other specialized productivity tools. But their total userbase is a tiny fraction of mobile users.
*even for programming, some companies are starting to embrace cloud dev boxes.
I don't think reasonable usage has anything to do with it, it's not hard to find useful workloads to run overnight, and that will just become easier as models get better and need less supervision.
At current GPU pricing you're right, unless you highly value running uncensored models, data ownership, and/or privacy.
It needs to be a useful workload to run overnight that also benefits from a number of parallel agents to have the same economics as the cloud. But maybe that will become the norm!
No need for local ram as well! Pathetic
What I find legitimately insane about all that is that Anthropic obviously does not have these censorship restrictions internally. So it's like.. They're just the whole world's adversary now?
>Anthropic restricting game reverse engineering
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
Maybe it was EA from my shenanigans :) https://dirtforever.net
I feel like if you are a small game developer getting their game reverse engineered and cloned kinda sucks actually.
Mamy had to pull demos since a lot of engines ship with stuff that's easy to decompile gated features
It doesn’t matter, right? The open models should be capable of reversing anything prior to 2005, no?
But it's all for the good of humanity, so how can you complain?
https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...
> With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.
Open Weight models have already reached level at which you can do a lot of productive work with them. They really don’t have to keep beating newer and newer models, of course it’s great for them to do things cheaper and faster, but we’re already past point where open models have to “catch up”
Right, for me Qwen 3.8 is perfectly usable for a lot I just wish it was faster.
Exactly, it’s about knowing how to use it at this point.
We achieved this with opus 4.6 last year, what those providers give you is great front end on top of it, which allows users to delegate even greater part of their process to model.
Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.
I'm convinced Claude is just an energy vampire in machine form at this point lol
> I won't want to be beholden to them and their whims forever.
Who's keeping you beholden to them now?
I agree. I am thinking this can only happen if we make training dramatically cheaper and efficient. This is why we are building a new kind of hardware.
> We
who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.
Sonnet 5.5 refused to implement code using selenium/firefox to scrape a webpage on my laptop after we got a 403 from a cloud-based IP. I made that clanker generate datasets for the rest of the week to use up my weekly usage that would have been wasted.
I got banned from the Hermes discord today not gonna lie.
https://gnosh.ai/head-to-head.html
Did we ever get back to 16GB feasibility?
AFAIK the timeline was:
Any chance a good model will ever be seen in a consumer GPU again?Well if you can run Qwen3.8-Flash-Next at decent speeds single stream, and you can then run it slowly but on multiple streams, and if multiple streams can act as an agent swarm that can compete with current SOTA, then... yes, in theory it's possible today. I think R9700 + vLLM-radiance is the play?
My M1 Max 64GB macbook is a consumer product that can be had for $2k on ebay. It runs Qwen3.8 just fine.
Is $2k for a whole machine not "consumer" ?
I've always been skeptical about "unified memory" vs VRAM. But my memory isn't great, so I tried to measure it right now, and see if a tiny model was as slow as I remember.
llama-ui on unsloth/Llama-3.2 3B Instruct:
Not really my jam.I did do the yak-shaving for the longest time. llama in a second-hand RTX3070 8GB was a useful tool. When gpt-oss came out, I upgraded to a second-hand 16GB card. It was slower (didn't seem to one-shot as much) but it gave smarter overall answers. At some point I bought a 24GB card which was basically dead on arrival. I kept my 16GB card in a server and maintained my VPN infra so I could access it from wherever.
I was still using this setup even when my workplace was pushing windsurf/copilot harder and harder. For me they were just broken. I'd ask for trivial tasks like "copy this unit test into a second unit test which tests something slightly different" and it would just get confused and delete shit, or leave random UUID strings throughout the codebase.
Eventually I tried out cline/openrouter/deepseekv4flash, and that was actually useful enough for me to tell it to do something to my code base, and as long as I supervised carefully enough it was useful, and priced OK.
At some point in the process I tried Claude Code because of all the hype, but I couldn't get any useful value out of it. I tried working with it all morning on something and hit my token limit without achieving anything worthwhile, so I cancelled Claude and went back to Deepseekv4flash.
These days every job application requires proficiency with claude code (or similar), so I was forced to get it again. And this time it just worked better. I haven't hit any limits, and it works like a better version of my cline/deepseek setup.
So where does that leave me?
Deepseek-v4-pro is 1.6T params, the cut-down version is 284B params, and I wouldn't trust them to do something on their own.
I'm very skeptical of a model being both small enough to run fast on my hardware and somewhere in the v4-flash <-> claude quality range.
If I did put down $2k, I wouldn't want it in such an old chip (M1) or a second-hand device that could die tomorrow with no chance of refund, especially to just 'see if it's good enough' (compared to the effectively-free v4-flash token spend).
[flagged]
if and when the AI bubble bursts, we'll know if local models will hold up. Assuming the cyclical pullback in investment is strong enough to see local machines with 512gb or more of RAM
lol people would rather worship the hackertube wierdo and the cabal
[flagged]
I spent several weeks working on trying to get swarms of cheap models to match the coding ability of a single expensive one. My take: it's remarkably easy to build a harness that improves a single eval, and remarkably difficult to build one that generalizes.
What did you try to make generalized harness, and what kind of failures did you encounter?
[flagged]
If Qwen is so great, why did they use Claude to do the write-up? This text is pure, unfiltered Claude. (e.g. "Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.")
Qwen 3.8 was distilled on Opus. Qwen Flash Next 3.8 even signs commits as Opus 4.6 some of the time. So, who knows :)
The title is about LiveCodeBench, not a creative writing benchmark.
Have some shame and astroturf better
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
> one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet
I'm not sure I understand. Are you saying we could put a dinner-plate sized chip that costs millions of dollars and draws tens of killowats of power into every single computer on the planet? Or that a single of these chips could provide inference as a service for every computer on the planet? Because neither of those are even remotely feasible.
it's economy of scales, qwen 3.8 27B regurgitates a lot and that's an intentional tradeoff of space/speed so you teach it patters that can follow for more complex stuff and give it more basic tools that can utilize to build those... the tradeoff comes from the model writting a lot of verbose prose to achieve that.
if the approach works, you have now a viable path to use this complex/hybrid structures where you can pair those chips with gpus and offload whatever is optimum to the model on chip.
Just like you can go and buy a threadripper now (the ram maybe not so much), how much was worth that amount of computing 10 years ago?
Happy hallowen! ::::
Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.
OTOH how much compute does Astra, Fable or Opus use? Is it just 24 times (1 vs 24 hour runtime) more than 27B Qwen?
The Qwen failure modes I’ve experienced won’t be overcome by running it longer or in more parallel sessions.
In the fall and winter (right now) I’m heating the house anyway so running LLMs 24/7 is basically free to me. So I try it. The models have are limited in their intelligence. I have tasks where I could let Qwen run for days with concurrent workers and never get anywhere, but 15 minutes with Opus solves the problem.
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
I’ve tried the Bonsai quants specifically because I want to run a lot of parallel requests.
The swarm strategy can’t overcome the dumbed down model problem. You just end up with your swarm of agents all going in the same dumbed down loops.
Throwing Ralphs at the wall in parallel
Except every single Ralph ends up curled up in the fetal position having accomplished nothing?
I think that requires testing, rather than assuming. There's only so far a harness can go.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
"with a bit of clever harnessing" I think perhaps you're minimizing the amount of effort that goes into the best moat AI companies still have.
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
you're probably better off trying to do this on Qwen3.8-Flash-Next on something like an R9700 or a pair of 5060 Tis, since you can run that at like 70 t/s and it's better. Then you just run multiple slots, assuming you can fit the KV-cache, which maybe you can't. You just need like 128GB system ram too.
I need to try this... it's on my list anyway.
Unsloth q3 works well with 16G vram if you can do with 64k context.
GVS5H, which this is based on, was discussed here a week ago: https://news.ycombinator.com/item?id=50228167
I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows
That's the point I am really convinced today that few small size agents actually better than one remote begemoth. I switched fully to local models (from claude, sol, and deepseek). A 27B model can be served on any modern PC or laptop with 64 GB or RAM (or VRAM on non-Macs). A bunch of Qwen coding agents is fantastic with opencode, works great also with pi. The only disadvantage is required compaction due to limited 256k context size, so it stops for few seconds once every hour or so. I am able to produce remarkable things similar to "see what I vibe-coded" posts here on HN but using only open weights, open source, and run everything locally.
Other big thing for me is being able to see the thinking.
And the privacy.
[dead]
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
I would not think in terms of "how much" context, but really just how the harness manages it. I've been fiddling around with harnesses earlier this year and the rabbit hole goes very deep. I eventually had to drop it because it consumed me, and I can only do one thing well. But what I learned is that there is a lot on the table at the harness level.
The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
This is somewhat tangential but speaking of ai benchmarks, I want to see one benchmarkes against passing those "filelocker" websites used for illicit downloads.
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
I can't wait for qwen4 flash next or something like a 150b moe from qwen.
I'm not surprised.
I've been running something similar for few weeks, but with Qwen3.8-Flash-Next. I started with opus 4.8 limited to 200k context as the supervisor and 5 Qwen3.8-Flash-Next agents(262k context each, autocompacted). A very simple way of calling agents via a command line using pi.
Telling agents to maintain task status in files.
This has worked very well. I tried Qwen3.8-Flash-Next as the supervisor too and as long as the plan is written by a frontier model all the implementation can be done by cheap/local models.
When I have lots of code to write I use DeepSeek v4.1 Flash 1017, because my entire local Qwen3.8-Flash-Next throughput is 200tok/s across 7 sessions, while a single DeepSeek session runs at hundreds. 7 DeepSeek agents and one Qwen supervisor can complete substantial tasks fable needed 12h in 2h.
I haven't tried Qwen3.8-27b for coding. I've been using it mainly for interactive chat with Web tools via librechat. My benchmarks show it as weaker than Qwen3.8-Flash-Next by few points.
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
That is only the Memdoor version. The original tested against the newest 100 questions: https://github.com/slee-persis/GVS5H
I use a similar approach for some types of work. I use a planner (read only), researcher (read only), implementer, and reviewer (read only). I allow the implementer to spawn researchers and reviewers to help it with its work. I interact with a controller to manage the subagent trees. The agents use a file-based communication system. I get really good results on real work. Coworkers (using Claude with fable) can’t tell the difference.
The benefit of Claude is purely team collaboration and some basic QoL features.
What part of Claude offers the best team collaboration features in your mind?
Benchmarks are one thing, but I've yet to see anything substantial actually built with local models. Most of the posts on X are about benchmarks and tok/s. Has anyone actually built a quality production app or game with real users?
Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]
He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.
Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.
[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>
[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)
Tbh I think it's mostly a matter of time. I use SOTA at work, but only local at home, and the performance improvements out of local models in the last few months have been substantial, and more are on the way. To add to that, people are starting get better at harness engineering (which is itself improving with things like Harness2 etc) and also now we see people doing things like fine-tuning a Qwen 4B model to do 3D modelling with some success etc. The latent capability for local models to do what you're talking about is there, but the in-the-trenches reality is people are still working through unlocking the full potential.
edit: I should point out that it's also starting to accelerate now in that you can get your agents to help you with making them better at these tasks. So, it matters significantly less if you yourself don't have the skills to progress things, so long as you can figure out just enough to get the models to do it themselves. I suspect we'll see this dynamic ramp up quite a bit in 2027 with Qwen4.
I've built an RSS reader and I'm doing a half assed reverse engineer / web version of Tapped Out (a silly little Simpsons mobile game that EA killed a while back). I've had to go to Claude a couple times when it got stuck on something curly, but I think that was mostly just loading a fresh context and I could have solved it I'd I really tried.
I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.
I am having pretty good success with DeepSeek 0731 locally. I have vibe coded some small things like an orchestrator. For a production product with real users, I will let it do first passes on code which I then review and polish up. I'm not oneshotting, but it's still a workflow benefit.
Well in large corp this takes time, even if you have an army of Einsteins at your disposal (just like we saw with Elons Doge) because there will be 16 different spreedsheets/dbs talking abt the state of the system that all contradict each other. While the 6 people who maintain their own state to keep the lights on, will have job titles that have nothing to do with system state maintenance.
Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.
[dead]
There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
How do we know Fable 5 is not secretly a swarm of models?
Maybe slightly off-topic from the main article but is there a good resource or site that clearly helps understand which of the "open" models are good for programming tasks, like C, C++, Python, Java?
I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
[flagged]
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
I’m skeptical of the result as claimed BUT I really do think model diversity really helps.
Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models: haiku, sonnet, terra, sol, grok, muse spark,
Then take the updated plan in opus and forward to Astra and Fable, once you have the final plan in place execute.
Then have the same batch review code after created.
I am surprised how often one of the smaller models catch things. Despite so many things having eyes on.
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
Plenty of people above are explaining why that wouldn't work. Basically 100x idiots don't always add up to one mediocre <noun>.
This also appeared on HN last week: https://news.ycombinator.com/item?id=50228167, it's the other method they mention, GVS5H
If this worked then a team of five fable models should crush al of its benchmarks, no?
Yes…and no. If you put 5 stupid people together, chances are you can get a smarter answer than individuals. If you put 5 smart people together, they will argue about the wording of the question.
I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.
This is probably not a novel idea but I wonder how it would work if you just stacked a bunch of smaller models on top of eachother and used that as the base to train a much much larger model. Maybe more efficient than starting from scratch?
This working journal is so much slop it’s basically impossible to read. Perhaps the author could write a short human written and readable summary so we can evaluate if this is interesting or not.
This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.
Thank you. I missed the gvs5h tie in.
There was some other research out there that says a weaker model can't just think forever and become a smarter model. What is different about this setup?
The gain comes from checking and restarting, not from reasoning. This harness is built using Mario orchestration which works pretty much like any DAG orchestrator. It's ideal for long repeatable thinking tasks.
In the Emory paper:
TL;DR: the solution approach uses "multi-agent evolution" and "multi-agent with distinct roles and objectives" for "solve-time adaptation" and improves performance with streams of heterogeneous tasks. The Emory team overall has congruent findings: Intuitively, the reason this works seems to be simple: multiple agents results in context spread to each agent by specific role. Each agent role only outputs the final, relevant result and not the noisy thinking tokens, tool calls, etc. so subsequent agents get smaller, more focused context, auto-adapting to each new task with prompt refinement at each step as the solution approach emerges.4 weeks back I switched over to Paseo and started using the agent profiles on a whim and have observed similar outcomes and behavior (seat of the pants): the win in my use case is that it feels like I can get higher autonomy without sacrificing quality (less input from me required to get quality output). The Emory paper and this team-solve seem to reinforce that role-based, multi-agent setups can improve solve performance.
For the curious, I shared a bit of my setup in a blog post[1]
[0] https://arxiv.org/pdf/2606.01770
[1] https://chrlschn.dev/blog/2026/10/larping-with-agents-and-wh...
can it bet a team of 5 fable 5's?
All these open source projects and the ideas in them will also help Anthropic/OpenAI's next iteration of models/harnesses, as it has done before. In the long run only access to the the raw compute power will determine relative advantage.
Why such an old benchmark?
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
Now do this for speed and correctness: https://gnosh.ai/head-to-head.html
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
"think", "believe" - maybe, but this is why we need better measurability.