As someone who is searching for a new programming contract right now, reading all of the incredible abilities here is pretty intimidating. Especially since I get almost all of my projects from Upwork which is an outsourcing site.
I believe I am competing directly with these frontier models in some circumstances. Like there are a ton of programmers who previously would be outsourcing work to that site, but now they assign that same work to AI agents.
Ever since November 2022 when ChatGPT blew up, I have been focusing on agents in order to try to get ahead of the curve. But I haven't managed to get an agent business off the ground and have been doing poorly paid agentic projects from that site instead.
But now everyone is building agents, and this crazy list of accomplishments makes it look like we are close to the point where the agents are building agents.
In fact the next time I get an Upwork contract for another agent, I actually should run it through my agent and see how far it can get. What I'm seeing a lot of now is requests to automate as much of a business as possible.
Anyway the point is these models are just about capable of doing the entire job of analyzing a small business and building out all the agents and iterating on them with the business owner.
That's actually what I should build is a SaaS that does that. Which I would if I wasn't basically desperate to get another contract this week.
And I know Upwork is bad but I have not had much success with other options on short notice.
> Anyway the point is these models are just about capable of doing the entire job of analyzing a small business and building out all the agents and iterating on them with the business owner.
If you search for LLM benchmarks focused on real-world tasks, you'll quickly learn this ain't the case. No point in telling business owners about that though, they need to see/learn by themselves.
What does the benchmark even mean when people are using AI to make real world things that solve real world problems?
I see people, and my self making amazing things with AI and fixing old projects and having real world impact at the fraction of the cost it would take me to hire people, or hours spent on my own coding.
I have built tools and systems with AI that have allowed me to build windows drivers, android apps, web apps, iOS apps, vm occultation, custom block drivers, custom file systems and more. To the point where entire products have been created.
Not trying to be a doomsday, but yes. It seems as though with the right infrastructure we are at the point where businesses owners can go from idea to product very fast and not need or hire much external talent.
Idk about sold. But it’s loaded on all the windows machines in a fairly big company that solves a real world problem.
It allows us to apply custom ACLs to AI agents and the child process spawned by AI agents. Giving us the ability to control what files an AI agent can read or write to, while still being in the calling users context. It allows us to force all ai derived processes to use a transparent MITM proxy so we can then also apply robust access rules to remote host allow or deny access to specific urls and not others. It also allows us to monitor access to windows Credential Manager with rules ti allow specific singed binaries to access some credentials but not others. It give us complete control of what AI agents on windows can see or not see or access.
Windows native sandboxing is lacking. You have some stuff in WSL that completely are broken once you call a windows native app. Or you have app containers which are too restrictive and result in applying expensive file system ACL to all files the app containers would access, which can take hours when dealing with million of files, and would be required to be applied every time you chains your app container (there are some workarounds, for them but they still have a one time cost a long with a fairly flaky maintenance process). You can get the network part done by running commands as a different user but that would result in the same file system ACL nightmare that app containers has.
Result is we get seatbelt level sandboxing in windows native, and can apply dynamic rules like preventing access to .aws folders regardless of the OS level ACLs, using glob rules like */.aws, so we don’t have to be aware of the exact path ahead of time.
It also has registry tree ACLs and, can prevent process and process trees from gaining administrative access, the list of features goes on and on.
According to a few tasks from my little personal coding benchmark it's very good at coding and kinda bad at web design. (Also excellent at "draw me a picture" one-shot prompts, for whatever that's worth)
On a sneaky one that involved parsing MIME headers and dealing with character encodings it did better than Kimi K3 at Max and for 38% lower cost.
Interestingly it seems noticeably better than the qwen3.8-max-preview model they offered just a few weeks ago.
Im sorry to hear about your situation. Have you blogged about it? I’m curious about how the volume of your type of work has been changing over the past 2-3 years.
That working is as not common as college dropouts becoming successful entrepreneurs. But both become stereotypes. But I know you might also just want to read about it.
PS. Not everyone (in fact the rare few) write as if no one is reading. For most, literally no one ever reads :D
You need to run away from any client who thinks your primary purpose is to write code. You must run away from the business if YOU think your primary purpose is to write code.
You should be able to write code and do it well, But the AIs, and I'm not even talking about the ones on the frontier, have been able to write code faster than I have for quite some time.
What you need to explain to your customers is all the things around writing code: software architecture, performance, and so on. Also make sure you have some understanding of the customers business, so you can suggest additional ways to make their lives better/make more money.
Software developers have a chance. Code monkeys? None.
First, I am aware they have been able to write code for some time. I made my first LLM coding agent experiment the day after ChatGPT first blew up in November 2022.
I haven't been a "code monkey" in the last say 15 or 20 years of my programming. I would argue also that code money was never really a thing, it was just an excuse that pretentious overpaid developers used for the existence of underpaid exploited programmers. 95% of programmers have to do requirements analysis and design regardless of their pay grade. I learned a long time ago how important requirements analysis and good design is. And the jobs I am applying to largely specify architecture and requirements analysis in the description. Employers are aware of code generation tools.
But also, the premise that writing the code is not the hard part is just nonsense. Sure requirements analysis is harder than people realize, good design is hard and so is understanding things about proper and closed iteration. But that stuff is not, in the aggregate, harder than the code.
And also, frontier models can absolutely do requirements analysis and architecture. And the sheer speed of implementation means that there is a huge built-in advantage for iterating more effectively.
"Code monkey" absolutely was and still is a thing. I was hired by Big Name Co originally to be staff but a reorg had me do typical senior position duties for around a year, and it was hell. Every decision was made over our heads, feedback was unwelcome because the feature had already gone through design and gotten layers of sign-offs from different areas of the org, etc. My sole purpose was to write code and review peer code. Staying there a full year was way too much.
Second point, requirements analysis is hard because:
* Different stakeholders have conflicting goals, sometimes mutually exclusive
* Some stakeholders have ideas that are actively harmful to users, and you must flex political capital to nip it in the bud
* Some stakeholders think they know what they want with a far greater degree of clarity than they actually do, and lack the time to sit down and suss out the nuances, because you're a side-show in the grander scheme of their own priorities.
Writing code is a heck of a lot easier than dealing with people at the edges.
Those are good points, but that mostly comes down to lack of organizational status or office politics etc. which is independent of capability level and largely orthogonal to whether it's a human programmer or an AI agent. Unless you are redefining "software engineer" to be more like a political operator. But that goes for any human group and any job. And anyway after a certain level, not providing the senior engineers and or project managers with sufficient political sway to realize projects goals is a management failure. Unless again you are defining software engineer as someone who is a master social manipulator in the context of their job.
To say that you're building agents, is based on a way of viewing things that isn't at all pervasive. Some might say you're building a customer support chatbot. To talk of building agents as a common activity makes sense if you use LangChain I guess, where its title is "Open Source AI Agent Framework | Build Agents Faster". But for instance, YC just released qm, and in its README it talks about customizing the agent, which is quite a bit different from just casually dropping the phrase "I'm building an agent". https://github.com/yc-software/qm
Not sure what you are trying to say but thanks for reminding me about qm, it has some cool features.
My MindRoot framework had some of qm's main features awhile ago though. I do try to use it to build solutions by customizing rather than from scratch when possible. But a lot of clients or potential clients don't like that idea even though it's MIT. They want some that has clearly been invented by them as a new programming project.
I actually turned down a project last year from someone who wanted me to start over in LangGraph -- he had already decided on it before he knew about my thing.
The most recent project I have been using MindRoot and building up the voice capabilities so I can now handle voice agents end to end including SIP.
But on that website now I try not to emphasize my framework at all for most proposals. You don't have leverage on Upwork. And also with code generation it isn't a critical point necessarily.
They've also announced Qwen3.8-27B being released open-weight next week. Qwen3.6-27B is widely regarded as one of the best local models, especially since nothing else comes close to it, that isn't benchmaxxed, without being significantly larger. If 3.8 truly improves upon it that would be awesome.
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
Qwen-3.6-35B-A3B was our "gateway drug" into switching our organisation to agent/harness-first coding.
Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode.
It continues to be a workhorse to this day, running on both my local Mac for various types of jobs, an AMD R9700 at the office, and said teammember still uses it on his 5090, although in practical terms we do a lot more with DS-V4-Flash-0731 these days.
I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Let it run overnight and wake up to millions of output tokens worth of results without data having left my house, all for the price of electricity.
I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though.
DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models.
For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs.
But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.
I started with Karpathy's LLM wiki, and did everything he said not to do - downgraded the model to mere tool usage and summarization, and it works great.
I am a data hoarder, and finally I can just dump all the content I remotely like, and get something interesting to browse for the price of electricity.
Agentic long-running tasks, as others have mentioned:
- Groom and triage tickets for agentic SWE workflows
- bug hunt — the probability of Qwen fixing a complex bug is 50/50 but often it is capable of identifying the root cause or at least laying the ground work for a more capable model to pick it up.
SmartConnections but heads up, Connections Pro asks $300/year, or, more for the plugin than for an LLM subscription, more than Microsoft Office for that matter.
Not OP, but I use it for a ton of smaller things. I have it hooked into Hermes and have been using it to help bulk rename my media folders so they all follow a common format, add titles that sort of thing which wouldn't be easy to 'script'. Another thing I use it for is comparing data sets, looking at my exported Spotify artists and compare to what I have locally, and letting me know where there are missing artists, or albums, and recommendations based on similar artists that I may not have locally.
Sure a lot of this could be done without AI, but it's certainly quicker and easier, and since my AI box is on solar, it's just the power of the sun to keep it going.
Not OP, but I use it for personal tasks that are just not worth the claude tokens -- rooting through historical medical records to unify prescription history, super-OCR'ing thousands of PDF pages (i.e. beyond PDF dumping -- vision means it can look at tables, understand tricky things like a continuation of a block quote or aside on the next page, etc.), and when power is cheap I'll just let it noodle on little projects on my data. During my agent's "free time" I give it with a tremendously open ended prompt last week, it did a linguistic analysis of how my texting changes in the lead up to, initiation, maintenance, and ending of romantic relationships.
The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something.
I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.
What!? You are skeptical of AI but will go through the manual process of hosting a model that’s less than frontier intelligence (talking about Qwen 3.6)? Anti-AI folks are always odd to me
A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
- You at least need a capable machine, so that's not 0 monetary investment.
- You need to spend at least an hour decicding between ollama, llamacp, mlx, etc.
- You need to find the correct quantized version of the model that works for you based on the architecture.
- You need to figure out the correct context window size to get reasonable performance.
- You need to setup a harness that works against your model
- You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI.
Or, you could sign up for Opencode for $10 and just be productive.
I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
install LM Studio, download the automatically selected quant based on your hardware, start a conversation with the automatic context size. 10 minutes at best and zero effort
Maybe not '0 investment and 0 commitment', but incredibly little depending on what you have laying around. It takes less than 5 minutes to download say LM Studio and an Open Model and as long as you have the hardware to support it, you start moving along. If you are on AMD in some ways it's even 'easier', you can download Lemonade and it will tell you exactly what will fit and best options based on what you are trying to do.
For me at least the local AI stuff, powered with solar has been pretty great. Would that scale to a large business? Goodness no, but for my tinkering and learning, it works great.
All of your objections have already been addressed by the previous comments.
The original comment states that the person in question already had a suitable graphics card to hand, so it did not require a monetary investment.
GP clearly states that "someone who is into that that stuff" was guiding the process, so it did not require a significant time investment.
> I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
I see no hypocrisy in the original comment.
You've also assumed the skeptic in question doubts the capabilities of AI. That may be the case (like you, I have no idea), but they may also have privacy concerns, in which case a local model is the appropriate choice.
There are plenty of reasons to be skeptical of AI.
I'm running Qwen 27B no problem with an AMD 9070XT + 24gb DDR5 ram. Does basic web search for me (tool call with tavily, costs nothing I get 1000 searches a month) and is great for creative writing (primarily breaking writer's block). Until the recent surge in ram costs, that wouldn't be hard to do. I built the computer for ~$1600 a year ago.
I am getting ~13-15 tps with my 9070XT for the 27B (~35tps for the 35B-A3B), but I think for me the main bottleneck is the 64gb of DDR4 3600 memory. What kinda speeds are you getting with what speed of DDR5?
Back in “the day” nerds just bought the hardware to fuck with. Some of us still do. Claiming that compute is the barrier to entry just means you’re not a nerd. That’s ok.
> You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model
How is that hypocrisy? Self hosting is somehow anti AI? Its not anti AI. Its literally using AI!
…and honestly, at a higher technical level than slapping your wallet against a token provider and running prompts in a hosted sandbox you can't even see the prompts in.
Look man I am incredibly skeptical of how LLM’s have been rolled out and all the promises people make (it’s so much snake oil and pipedreams), but I also found it very trivial to hop on LM studio and start tinkering with models. If you’ve already got a decent midtier computer on hand, which I imagine a lot of us already do, then it’s really not hard to get started and get immediate results.
Local models on regular hardware aren't really capable of anything. Whatever you're testing is nowhere near a measly $20/mo subscription, so it's of limited use.
i really like the idea of running local models but i'm always in the position of wanting the best model(s) available and i don't have any severe privacy concerns. as such i have yet to justify ever using local models.
This is the diametric opposite of the rent-vs-buy scenario that this entails.
Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware.
Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine.
> takes literal minutes to get started
Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers.
Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute.
> you can just `rm -fr` it and forget the whole thing existed.
I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it.
> I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument.
I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
It took me about three hours total to set up a local model. I already have a GPU and I have fiber for the download. llama.cpp is not difficult to compile and has many backends. It can run parts of the model on different backends, like in the common case that the GPU doesn't have enough VRAM for everything. There are many step-by-step guides available.
Takes even less depending on your system. LM Studio or Lemonade and you are set up in minutes and now they can even tell you what models will fit with the memory you have.
> "I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!"
I used to deal with these kinds of frustrations too.
fd --unrestricted --size +1G
fd --help
-u, --unrestricted...
Perform an unrestricted search, including ignored and hidden files. This is an alias for
'--no-ignore --hidden'.
-S, --size size
Limit results based on the size of files using the format
<+-><NUM><UNIT>
Full model or a 4-bit quant? I have a 5090 and I'm not sure whether I should use a quant that fits within the VRAM or a much bigger version where I'd have to offload a lot to 64GB RAM and a beefy CPU (but still a CPU)
I personally run the Q6 quant on my RX 9070 XT (16GB VRAM). On r/LocalLlama there was a post recently as well, which talked about the degradation of different quants (for the 27B version)[0]
A lot of people already have 12GB+ GPUs lying around for playing games, doing video editing, etc. I would not get a GPU or mac just to run LLMs personally, but if one wants to get such a device for other tasks too, it may make sense to eg choose a slightly higher (v)RAM variant if they want to run some bigger models. Then what you pay for the local llms is just the difference.
This largely describes me. I'm skeptical of AI in that it's capabilities, while very impressive, are vastly oversold and overblown. Being skeptical of AI is not being "Anti-AI". That's largely the AI data centers are using up all the water and electricity types.
Maybe you're anti-AI because you're really anti-outsourcing your thinking to some remote corporation you don't control?
That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime.
I'm not against AI. I'm calling out the hypocrisy in the comment. I'm anti-AI, but will spend hours trying to setup a local model, instead of just getting access to frontier intelligence in 15 mins, and actually getting useful work done.
If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
I think learning how to set up a local AI now requires less time than learning about potential pitfalls of token-plans and processing payments in corporate environments.
It's not that billing is complicated, but learning to set up a local AI is a lot more useful and more rewarding.
35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter, so I daily drive that. I wish there was a ~100B MoE with maybe 10B active. It would be super smart and fast!
I tried it for a bit, and It was not really worth its size. It got swept up in all the other AI news recently, but laguna s 2.1 I think is the best ~100B moe model right now
I didn't mention it above, but Laguna S is my other favorite model. I use Qwen a lot more, it's smaller and faster, but I like to switch to Laguna when I feel like I need a "heavy hitter" for certain huge or complex tasks.
What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
Strix Halo as well. Bought it for $1,800 new on sale and shoved an extra 4tb drive into it. Been amazing for local AI. Maybe not the absolute fastest thing (usually around 30t/s depending on the task) but has been awesome for a local AI box that I can solar power.
Couple of rack mount batteries and roughly 5kw of solar panels. Feeds into a subpanel so I can flip it when I want a couple rooms of solar on the house, or hook a generator up if needed. Can't power the entire house, but works well for thinks like computers, lighting, etc. And if I want to expand, just throw on more panels, or realistically, just throw on more batteries to store the juice.
Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have.
Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.
Yeah, here I am sitting deeply deeply deeply regretting not buying couple CMP 170HX at $200 or $350, knowing I could just flip them ethically at purchase price if nothing came of it... I could have just casually built a 128GB dual A100 local AI monster
I'm working with a lab that has a few Ampere GPUs on infiniband and they are just not compatible with the latest quants and vLLM updates. FP8 is about as low as you can go.
I have an RX 6700 XT with 12gb vram and 64gb system ram. running dense models like 27b is difficult, but i can run IQ4/IQ5 qwen 122b-a10b or 35b-a3b at ~20tok/s
IQ4 qwen 122b-a10b would mean 61GB total size and 5GB active, so about 5GB of the model loaded into GPURAM plus any generated context, and 61GB of weights loaded into system RAM? I don't know if that math is correct, but does that run well? Wouldn't that only leave 3GB of system RAM?
Not badly so because MoE models(identifiable by "CoolName-xxxB-AxxB" naming scheme) have bunch of branches in the middle that only one out of all gets non-zero values. Each of branches aka "Experts" as well as top/bottom parts are significantly smaller than the whole, and so CPU emulation of CUDA operations mixed with GPU taking as much as possible become not so out of question, unlike for dense models("CoolName-xxxB" without "-AxxB")
I use 27B in plan mode and 35B MoE in act mode. I noticed that is the best balance for me for consistent tool calls and intelligent planning. Takes some time to switch, but it's worth it for me.
Honestly the 27b dense one punches way above its weight in a lot of domains, especially coding in my testing, so I think you will probably be disappointed.
Can you elaborate on DeepSeek (deepseek-v4-flash, i assume?). What does your typical usage pattern look like and what is your weekly/monthly spend?
I gave it a try for a few days (pi + openrouter + deepseek-v4-flash via deepinfra) and ended up paying ~$18 for rather light usage. Yes it's still cheap, yes it's fast, but i feel i would still get a better deal with a Claude subscription plan.
I agree with parent. OpenRouter might be cheaper list-price, but i have been using 10$ on DS platform since April/May, still have 2$ left. Using OpenRouter i depleted the same dollar-amount in a 1-2 weeks with same usage pattern. No idea why.
I would recommend looking into Ornith1.0 - it's using Qwen3.6 35B-A3B and excels in coding, at least for my coding needs, Python, web-dev, SQL scripting and some C#. Using Pi harness.
Apple Silicon. But: there's no need to use the FP16 version. At 8-bit precision the quality loss is almost imperceptible. That cuts the footprint to 36GB. Which is great for a 64GB Mac, because you have room for plenty of context. 6-bit also works nicely at 26GB + context.
You want to use the newer quantization formats like Unsloth's UD quants or oQe, where the weights are selectively quantized using a calibration dataset so that important weights are left at/closer to full precision.
use a quantized version. since it's MoE, what matters is that the 3b parameters that are used for every token fit in gpu vram, the rest can stay in system ram. really great if you don't have unified memory.
I host the models on my Mac Studio, an M1 Ultra with 64gb ram (I bought it when it came out, just happens to be good at LLMs). So when I work on my laptop, I have my oh-my-pi setup configured to use the models on my Mac over my local "bonjour" network or whatever Apple calls it. That way I have a nice cool lap, while using models that my M4 MacBook Air with its 16gb ram couldn't possibly run.
Strix Halo for me. If I am running something on my laptop, it's a much smaller usually around 12b model, but those are a bit less functional. I mean I think there is a a ROG FLow Z that has the Strix Halo setup, but that thing was super expensive.
On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.
Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
While the laptop option is nice, for an inference server you're probably going to want the desktop form factor as it has significantly more thermal overhead and thus better performance. In the desktop models most of the internal volume is a gigantic heatsink
I have a framework desktop, but depending on your need, DGX spark might be better. The prefill and NVFP4 is a significant advantage. But framework desktop is a better general computer. I expect to be able to use it for years to come. Where as DGX Spark you’re at the mercy of NVIDIA BSP.
On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama with completely default settings (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)
Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).
It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).
Depends on how you are doing it. LM Studio and tool calling models can use the web, or you could go for something like Perplexica, or if you want to go real crazy, something like Hermes or OpenClaw.
I recommend trying pi.dev as your agent harness for local models. In my experience it has been the sweet spot of functionality (which you can and should extend with plugins) vs performance (OpenCode just swamps local models on my hardware).
you can try ollama, omlx or llama.cpp for instance to download a model and get an inference server running locally. They expose „open ai compatible“ endpoints, so you can configure almost any harness to use them.
Not the GP, but I run this model as daily driver too. It runs great on a Macbook Pro 64GB (M3 Max). Token generation speed can be about 100 tokens/sec with multi-token prediction, although it depends on the context. Worst case speed is around 50 tokens/sec.
The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.
I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.
It's just a Macbook Air with the base M4 and 16gb ram, but I'm hosting the models on a Mac Studio with M1 Ultra and 64gb ram that I had purchased when it came out. I get about 45-55 tokens per second with this setup. I think I could get more if I spent some time fiddling with the parameters, but I don't really know what I'm doing there so I've just left most of it on oMLX's defaults.
I'm on the verge over here, the new Anthropic models have been a disappointment. I've tried the A3B variant, but had mixed results. What do you use as the coding agent, and have you heavily customized your workflows?
It's just a MacBook Air with an M4, cheap and nothing special. I host Qwen on my Mac Studio, an M1 with 64gb ram. The model uses around 20-25gb ram depending on what it's doing.
Kimi K3 is more like "weights available" in that you can download and use them but it is under a custom license that has a bunch of limitations where you have to pay Moonshot for doing some stuff. GLM 5.2 on the other hand is plain old MIT.
Not sure how Qwen3.8-Max is going to be licensed, hopefully it'll be Apache like the smaller ones.
You can do whatever you want with the model within your own organization. If you use it commercially—either as a model-as-a-service business or in a very large-scale product—you should check the additional license terms, which go beyond MIT. My interpretation is that Moonshot cares about the exact inference behavior and accurate representation of their model or derivatives, and perhaps also about capturing some additional value despite their own GPU limitations, so the extra license terms focus on those large-scale commercial deployments.
Considering that very few orgs are going to be able to host a 3T parameter model internally, chances are most deployments would be subject to these restrictions and require a separate license from Moonshot.
US AI companies are already sweating and 100% pressuring the Trump administration for more anti-Chinese regulation, since there have already been talk of Trump considering banning Chinese models. There's however another push back from the startup industry urging them not to ban it, since it will stifle the innovation. In other recent news OpenAI also greatly cut their model prices, 20% for 5.6 Terra and 80% for 5.6 Luna, to stay competitive.
> In other recent news OpenAI also greatly cut their model prices, 20% for 5.6 Terra and 80% for 5.6 Luna, to stay competitive.
I’ve seen comments on HN saying how bad this is for the Chinese model developers since the cheaper option like Deepseek Flash are not longer as price competitive to justify the hassle/risk/lack of multimodal… but isn’t this a gigantic red flag for OpenAI/Anthropic at their current valuations?
Sure, it’s just the lowest end for now, and the enterprise money is at the top of the market. And there’s protectionism/enterprise lock-in/etc that complicate things somewhat.
But still, if the US AI labs ever tap the training brakes for a millisecond, the “inference is still a money maker” argument seems to evaporate when they’ll immediately have to fight a race to the bottom until margins are virtually nothing.
Or if the benchmaxing “line goes up” FOMO mindset starts to lose its luster and companies find their individual niches for productive use of AI and stop bothering with all the latest and greatest churn for top dollar.
Which might be even worse if it means the training arms race is still ongoing but neither Anthropic or OpenAI want to be the first to “lose”. While the marginal value of each new model training run keeps decreasing and enterprises signal they’re more concerned with cost reductions than solving ARC-AGI-7 puzzles.
Huh? V4Flash is still incredibly price competitive. The new version is right around GLM 5.2 and maybe slightly worse than opus 4.8 depending on which benchmark you use while being much cheaper(even factoring how most US zdr providers charge 10x Deepseek’s api caching price). K3 is also a tad behind fable/sol while being alot cheaper
Some advice I got from another HN Mac user was to run local models in energy saver mode. You'll get slightly reduced tokens, but the laptop won't overheat and the fans won't go wild.
Oh. I've been using an icepack under my laptop to keep mine cool. I'm watching it with llamatop to see if the GPU is actually active or not, aw activity monitor wasn't showing me what I wanted.
This is what I've been waiting for. We are still using fine-tuned deployments of Qwen3.6-27B with a lot of success but could use a bump in intelligence. Here's hoping.
I also "evolved" into 27b (q8 unsloth) and pi.dev (tried many combinations) feels for me the same as opus 4.5 that i use at work, faster even (using 2x 3080 20GB gives me 60-80tk/s). Though you do need to feed it more details up front (about what exactly you are planning to do and a good written skill.md) but I work that way anyways, im hyped for 3.8
Having invested in a machine with 128GB of RAM, I would love seeing something a bit larger than 27B / 35B, possibly a 54B dense model or 70B MoE would be much closer to the Qwen 3.8 Max experience.
All of us with a 96gb rtx 6000 would love to see a 70b moe. Maybe they are waiting for OpenAI and Anthropic to IPO so they can short their stock and release. Local LLM is going to get very interesting in the next 2 years.
The 27B have many more active parameters than much bigger models such as DS4Flash, MiniMax etc, which makes it punch above its tiny weight. A great fit for a 5090 in a closet for meat-and-potatoes, kind of work.
I've been running Qwen3.6-27B-IQ4 (4-bit quantized) locally and it's been great. I can't run the non-quantized version as I only have a 4090 w/24 GB of VRAM and it won't fit and leave any context room, but the quantized version only uses 18GB.
Yeah, Qwen3.8-Max is the new Flagship model for coding and harness system and many other benchmarks are reaching equal performance as Claude and other close models. That's gonna drop the price of LLM in agent landscape a lot.
the bonsai 27B 1bit quant version of Qwen3.6 27B is even more nuts, model fits in 4GB, and with 100k of content model+kv cache fits in 8GB. I’ve been running it locally on my mac mini 16GB. it gets around 4-6 tok/s, so not quite real-time ready, but good enough to let it run on task async for 20 min and come back. The 1bit model struggles a bit with multi-turn conversations though (e.g. when switching from plan to act mode it will still keep trying to make a plan) but that’s easy enough to reformat prompts into multiple one shot sessions of smaller work.
For those of us who don't have the time to follow closely, Qwen3.6-27B being Open Source and Open Weight, what level is this compared to other Western paid version?
Just so that we know what 3.8 would be like.
I currently have about 150 Tabs of Antirez posting on AI and running local model I haven't had the time to read. And there are probably some prerequisite reading or other research in between as well. I just wish there are some very high level overview and news coverage on all these.
> what level is this compared to other Western paid version?
IMHO this is a difficult question to answer. Part of the power of paid models comes from the software supporting it. With local models, you have tons of workflows that can severely influence the quality of the result.
In my personal experience, the SOTA models are way more consistent and can handle more complex questions. Part of that is (probably) because I don't let my local model access the internet, while paid models do use the internet to look at docs etc.
You absolutely need to let models access the Internet if you want consistently good results. Pretty much any non-trivial task requires the model to do things like look up APIs, code examples, or existing discussions of a given topic.
You don't really need the Internet. Tons of documentation is available for download (either as a zip, or with the documentation site as its own git repo). Wikipedia is available for download. You can get reddit dumps, HN dumps, stack exchange dumps, etc. This can all easily fit on one hard drive.
Reading the actual code is also always a better source of truth than docs anyway (this is true for people and LLMs). Just clone whatever libraries you use.
If they trained it well, and can do computer use, it will be a new era. Companies can keep PCs, put Qwen 3.8 27b on it and get rid of the employees, lol...
I am surprised that they keep going with it, seeing how fast it improves and basically soon running themselves too out of business.
What's even their end goal? Open source models make sense, if profit is not the target, but for OpenAI and the rest, once they achieve "AGI", don't they basically become useless?
There was an interesting interview by MLST with a team doing well on ARC AGI 3 who are using Qwen 3.6 27B, and said that it's actually better at coding than the larger 3.6 35B.
I guess which of the smaller 3.8 models is best for coding will depend on which one they put the training effort into.
The larger 3.6 35B is actually a mixture of experts (MOE). This means a small proportion of those B's are actually active. It's fast and suitable for agentic tasks but nowhere near good as the dense 27B model, which has all of its parameters loaded.
This makes me wonder if AI companies even have a MOAT in the first place.
All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days.
And that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars. Or all of them are worth a lot less. One of those statements is true.
Also this makes me wonder if the next iteration of LLMs would be based on fine-tuning, where LLMs actually learn from your past behaviour so that it would grant some amount of stickiness to the product. OpenAI used to offer fine tuning runs for GPT-3.5, but they don't seem to do that anymore.
While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow.
The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow?
If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
As a heavy pi user - this doesn’t help, for codex. The thinking traces are encrypted and you cannot move that session over to eg your local ds4. It will lack those precious thinking traces that you absolutely need to continue a session productively.
They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time.
This is harder to do on other architectures that themselves aren't fully deterministic though.
Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245...
My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences
Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction.
You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".
Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.
I appreciate the replies on the determinism point and I’ve learned some new things here. In any case I probably should not have tagged that on, as my main point was to share that the sort of property that parent is talking about (whether true for all LLMs/providers/harnesses or not) is statelessness, not idempotency.
It's a more or less solved problem (e.g. DeepSeek has batch-invariant kernels, vLLM also has reproducible outputs etc), with an obvious caveat that you need to lock your hardware setup, model snapshot, and inference stack if you want any useful reproducibility across longer time periods. It's up to specific providers to actually implement it to any extent, of course commercial providers are incentivized to do the opposite.
>LLM responses are certainly not idempotent, as they are not even deterministic.
Isn't that more due to an optimization and not how the LLM itself runs?
Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.
For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.
Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.
It is much more subtle than your specific example, which is strictly speaking a bug, though ofc it has been used during pretraining for efficiency purposes. Sglang and miles have been working towards full determinism in open source codebases, so the LLMs can help explain the subtleties encountered in actual projects if you point them to these repos.
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.
Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks.
Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down.
Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.
Agree. Harness can not be a moat. There are many open harnesses and they are at least on par with the providers ones. It looks like Anthropic/OpenAI's approach to vendor lock-in is not so much the inference or the harness it is functional integration across the individuals and teams in a company. I don't think this will be a moat either, but I think it's all they have outside of compute.
The training data, at least up to now, is very abundant and basically every lab has the same data from scraping the Internet. RLHF data is what's now valuable.
One possibility is that they're counting on another bailout package like the banks and others got in 2008.
The bets are absolutely huge and have been spread throughout the financial system via securitization. If the number does not continue to go up, a lot of people are going to find themselves backing their numbers with money that simply isn't there anymore. Therefore, the number will continue to go up and we (meaning American taxpayers) will back whatever it takes for that to happen.
Now, were the bets stupid to begin with? Of course. Who wants a machine that potentially unemploys major percentages of the population? No one. But the bets were made regardless, and then more bets were made on the hypothetical returns of the first round of bets, and so on and so on.
> that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified.
They aren't, not even if we forget about the capable Chinese models.
I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.
The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are.
The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor.
If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.
compute is not a moat, it's a rapidly depreciating physical asset. buying up all the shovels in a gold rush does not give you a moat, it gives you a slight advantage for the time being. someone else will just start making shovels. and the data is clearly available, hence the number of open-weight models.
well the boards have been laid across the two shores, and the enemies are now climbing the walls, i guess is what i'm saying. if they had a moat, it's been breached
your post helped me realize a change Meta is pursuing on Instagram that is to give more weight to captions and long text posts so they can have more data to training that would usually go to websites/Google. Even AI slop is good for this.
These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through.
But all the money in the world cannot move government too quickly.
Other moats exist too. OS or browser can undermine performance and availability of alternatives.
> This makes me wonder if AI companies even have a MOAT in the first place.
Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style.
Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.
With their current API approach they're essentially a commodity. They need to start moving parts of the harness behind the API, otherwise they'll remain a commodity.
Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.
> Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars
It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.
This isn’t technically true. Most model providers don’t send the thinking tokens anymore, so if you switch from one provider to another, you will be missing large parts of the conversation.
fine tuning runs of models the size of gpt 5.6 are absurdly expensive. I'd guess at least $100k in cloud gpu time for a single run, and you have to do a few iterations to get things right
I think you're right and I think it's why Google have taken their pedal off the metal for model releases to focus on integrations and tools. And why Microsoft have backed off from the OpenAI partnership to do the same. Anthropic and OpenAI are going to massively struggle to maintain their pace and reach profitability just selling commodity tokens.
Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.
From my experience these open source models are nowhere near the performance offered by Fable/Opus/GPT-5.6. Whenever I tried Qwen, Kimi, Deepseek, the results were much worse and it just took much more time to get something usable. When you consider that, the frontier offerings are still much cheaper.
That might be true right now, but how long until you have to move the goalposts? In my experience with DeepSeek and Kimi, they're as capable as the frontier was four months ago, which already solves a big chunk of the coding tasks that I'm interested in.
Same prompt for both for the conversion. I used OpenCode for the qwen version, but I encountered a significant amount of errors / timeouts while it was running. Claude finished in around 16 min, but I spent close to 2 hours shepherding the Qwen build. For the implementation, there were signs it had good vision, but the timeouts make this very hard to use in a production setting.
Update: Tried using Qwen Desktop. It told me it succeeded, and linked me to the folder it created it in.
That folder didn't exist, I asked Qwen where the files were, and it attempted to create them again.
Stopped it, and asked what happened to the original files. Qwen Desktop apparently doesn't have access to the local file system, but continued merrily along without warning about that.
I've had nothing but trouble with Qwen on opencode. GLM too. I know a few people who use them on Ollama could without issue though, so I don't think it's a model issue.
Both horrific (from a user's perspective, perhaps not the underlying code).
I've noticed another type of AI slop that's prevalent in all the popular models; font sizes and variations like you wouldn't believe. It's very hard to instruct LLM's not to do this.
I thought it was pretty cool, its the kind of stuff that I wish there was more of on the internet. I guess some people's imagination runs a little more than others.
Whilst these coding models improve, they seem to cater for multiple languages, and for many, a trimmed-down LLM that supports just one language would be fantastic. This would be smaller, more able to run on the hardware people have at home, realistically (even on the CPU). Having one LLM that knows assembler, java, bain, C etc is neat, but when you only work and use one language at a time, it would be kinda neat to have those broken out into dedicated, smaller LLMs. After all Qwen3.8-Max handles over 90 programming languages - ask any programmer to name a list of computer languages and if they get over 20, they are doing well.
I believe the knowledge of multiple languages brings something onto the table that having narrow language knowledge does not. The LLM will be able to better generalize a problem, apply universal concepts and avoid mistakes that would otherwise be too "echo chambery". Even as a human, knowing multiple languages has made me better at my primary language.
That's not how LLMs work. If you're talking about number of parameters, you wouldn't be able to reduce the size much by "removing" support for other languages.
Only a tiny, tiny fraction of the parameters are encoding information that's specific to a particular programming language. Even if you could remove those without degrading performance, it would have a negligible effect on the model size.
Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):
0.728201017
-1.9293839303
2.93020202929
5.82290101728
-0.2844920917
…
And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).
Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:
0.428201017
-1.6293839303
2.63020202929
5.52290101728
-0.5844920917
…
(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)
Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.
Please do correct me if this simplified too much or I had a mistake in my understanding.
This is oversimplified. The proposed question is whether a model with fewer parameters could achieve performance on one language similar to that of a larger model that’s been trained more broadly, which isn’t straightforward to do.
Oh, I didn’t interpret the above question as asking in that direction; but yeah, that’s of course something I didn’t attempt to answer with my comment.
Although I’d be intrigued in the answer to that small-narrow vs. large-broad model question, too!
The exception being the token embeddings and lm head (which scale with the number of tokens the model knows and presumably you need a smaller number in the tokenizer for only English and python). But those are a pretty small % of the total model weights on most LLM sizes
For a set parameter size, if you benchmark an LLM on a single language, you'd get better results by training it on multiple languages rather than just the one. Scaling quality training data almost always increases performance as the neural network picks up general rules and an understanding of code that are language agnostic.
I think you'd want to remove e.g. knowledge of harry potter universe and ancient egypt. Training on a bunch of high quality java code bases is still likely to improve your python model.
Actually, it's shown that even general knowledge helps coding models because their input is natural language itself so they need to understand it well enough to even turn into code.
Teaching model to code in say python does indeed increase its effectiveness in other seemingly unrelated areas. On the other hand I remember that early models of ChatGPT were great in chess notation and later models aren’t as good as OpenAI doesn’t prioritize this now
It was a matter of time for China to catch up with the US. In terms of infrastructure, manufacturing, and engineering workforce, China has the upperhand and I foresee them becoming the SOTA leaders. Maybe if the US wasn't so busy gatekeeping and keeping things proprietary, they would've had more trust from the open source community.
First and foremost, China isn't energy-limited like the US is. A modern grid, lost of cheap power coming online every day. The worlds most advance ultra-high voltage transport links are tying the deserts in the west to the cities in the south.
It probably sucks for you if the proposed powerline goes through your house, but the need of the many come first.
Having cheap and plentiful power is a huge market advantage. It was one of the factors driving Norway out of poverty 100 years ago.
I think this is really a rosy image of China’s electrify situation. They have huge needs in the east and green energy in the west, they can’t build their UHV links fast enough. They are doing ok at keeping up, but there isn’t a surplus of cheap electricity lying around in the same places where you have cheap water as well.
What is stopping them from just building their data centers in the west where they have energy surplus? Not like you need to locate a data center inside a city.
Water. The west is really dry. We could throw in data lines as well but it’s really bottlenecked by water. Actually the only place in China with water in abundance is southeast China.
Also they really don’t have the chips to supply those data centers, and the government isn’t so crazy to go that aggressive on AI, they have lots of other priorities as well.
Chinese data centers like to use air cooled chillers for cooling, not evaporative coolers. It's simple because power for the chillers is cheaper than water for the evaporative coolers. The opposite is true in the US
Evaporative cooling is a matter of convenience and cost, not a necessity. Other cooling technologies can be used. Intel has already proven a closed-loop system can work just as well.
One of China's big sources of power is hydroelectric such as the Three Gorges Dam, so if you want electricity and water in one place, then there it is.
The US also isn't energy-limited. They'll just price out all humans living there from buying any electricity.
EDIT: This was obviously meant to be facetious with the intent to highlight the negative effects on the general population that datacenter buildout has in the US.
That graph features a completely-made-up superexponential AI growth curve coincidentally reaching 100% of US electricity generation in 2030, but even if you take the prediction up to 2026 at face value, it also agrees that AI is currently a small fraction of overall demand. Of course any individual data center site might struggle with insufficient local supply, but China would have the same problem if they wanted to build bigger data centers. (Production capacity is actually north of 25 000 TWh, but poorly utilized because of a mismatch in place and time between producers and consumers.)
Also give it to Chinese labs for the vision of going open source and open models to compete. Apart from their great models, I quite like how they publish papers of their research too.
I think the window for a ban of open weight models is closing fast so let's hope US administration is going to miss it and we get Fable-level models (at least in some aspects) with open weights without infringing any newly introduced law as a long-term local baseline.
Even if the US does ban it, Europe due to its lack of European competitors, has no current similar protectionist incentives. Even if there is regulatory capture in the US, the rest of the world isn't going to follow suit until the current market leaders outright are replaced with other competitors that Europe would want to favor. The genie is out of the bottle in the West already.
They are profitable and active on the enterprise local model territory. You can RL a model with them for your own purposes and I heard good things about it.
If China keeps releasing LLMs with such permissive licenses it probably works better since the pretraining rnd is subsidized and de-risked - but that’s a big if.
Would be hard for them to act alone due to free movement in EU. Would be hard for them to outlaw a German business from offering the models from a German datacenter to French customers.
Theres a pretty big chance imho that europe will follow suit , or at least the segmenets that are fully aligned with the US on all things china even to their own detriment.
Remeber rip and replace Huawei 5G because of national security , leading to some parts of europe having worse coverage than some tiny african markets.
Best bet would be for europe to look at it like what it is an open source commodity that is useful to build on rather than a moat to extract rents with.
Doubt it now when US declared that Europe is an enemy, Europe is banned from the best models and that US is considering an invasion of Europe (Greenland). The days of US influence in Europe is limited.
If it was 2021, probably. But Trump has soured relations in the last 18 months enough that the soft power about technology isn't there like it once was assumed to be. What's he going to do if Europe doesn't step in? Jack up tariffs? Is it that day of the week already?
Open weights is more beneficial to Europe spring-boarding a catch up than defending Claude/Anthropic.
If CADA passes in its current state I doubt we'll be banning Chinese models any time soon. I think most people misunderstand the extent of the permanent damage caused by Trump
He threatened to go to war with Denmark over Greenland. In the US, that might just be a side story, a little joke from insane grandpa president. But it's not in Europe. And it's only one of the crazy adversarial things he did.
well, the EU just brought down the regulatory hammer which complicates things a bit. we will have to see if it operates as a proxy ban on open models depending on how they go about enforcing all the safety, bias, transparency and data provenance requirements.
>Europe due to its lack of European competitors, has no current similar protectionist incentives [...] The genie is out of the bottle in the West already.
I agree, but you are underestimating EU regulatory incompetence
You can send armed men to your house, or restrict companies in America from using these models.
They can also restrict US providers from serving these models. Looking at OpenRouter, there is currently 11 providers for 'DeepSeek V4 Flash 0731' with the only non-US one being Deepseek themselves, which aren't even the cheapest.
> the only non-US one being Deepseek themselves, which aren't even the cheapest.
Interesting.
If I were a western lab, might I be tempted to subsidize at least one faster & cheaper provider for Chinese models to keep even more data from leaving the US?
(I know in this case there can just be more efficient providers, am asking more than speculating)
The NSA will run Mythos against the entire US population and it will find the patterns. Powerful computer purchases. Energy consumption suddenly spiking. HN comments demonstrating intent and/or contempt for the glorious self-evident freedom preserving laws of the United States.
Then they call your local law enforcement with an anonymous tip about this suspicious dude who's long overdue for a totally random home inspection.
When has enforceability ever been a concern of the government? This is the same government that had anti-sodomy laws on the books until the Supreme Court ruled it unconstitutional in 2003.
Making it illegal even without having a realistic means of enforcing it still chills behavior, and so just because it isn't practically enforceable in the general case doesn't mean it isn't without consequences.
Same way they detect if you do business with Russia, North Korea, Iran. They don’t detect 100% of the cases but the punishment are severe enough to deter most companies.
They can't enforce a ban. Companies can just download the models and run them on MacBooks or GPUs (or the cloud) and USG has no idea. It would also hurt the administration's funding from corporate tech (companies don't like to fund politicians who restrict them). I'm not saying the administration won't do it, it would just be very dumb.
> Today, we are officially releasing Qwen 3.8-Max, the most capable model in the Qwen family to date. This also marks the first time we will open-source the weights of a Qwen-Max-class model — the open weights will be released next week.
> Qwen3.8 is launching and going open-weight soon! [...] You don't have to wait to test it. Just now, the Qwen3.8-Max-Preview made its debut on Alibaba’s Token Plan, Qoder, and QoderWork.
July 19th post mentions “Max-Preview” vs. today’s post dropping the “Preview”. Unclear what changed if anything though.. Maybe broader availability or it’s a slightly improved checkpoint
My understanding is that these "preview" models are usually earlier RL checkpoints, and that "official release" happens when they're happy with the training run?
I believe they mentioned around the preview announcement that they'd be releasing improvements to capability, which I assume means continued training.
K3 also forgot the wheels when I tried it, but it turned out to be a bug in llama-server's built-in SVG viewer. I know you have your own rasterizer but just FYI, sometimes it isn't the model's fault.
Agree, I don't necessarily see a strong argument favoring OpenAI or Anthropic here. In the interest of perspective, can anyone (perhaps playing devil's advocate) give one?
The open models are now good enough for what I want to do with them, let alone any future improvements. And factoring in efficiency gains, a model in the ~70b range starting to satisfy my needs would completely obviate the need to pay others for inference. This does not seem far-fetched to me, comparing with where open models were at this time last year. What am I missing?
Seems to me OpenAI and Anthropic are kinda following the Apple business strategy. Those two offer a premium service that gets better results and works more seamlessly. I.e. the integration between Anthropic models and Claude Code is apparently nice and gets better results, and I've heard anecdotally that Codex is currently the best.
So just like in IPhone vs Android, you could end up with a situation where Chinese firms compete and get most of the revenue and usage with low profit margins but OpenAI and Anthropic capture a premium side of the market and still get a lot of profits.
> In the interest of perspective, can anyone (perhaps playing devil's advocate) give one?
I have numerous cases where Sol failed and only Fable could solve a problem. For example yesterday I was merging a Q2 curved with a Bezier curved face in 3D using OpenSCAD. I tried for over 2 hours with Sol 5.6 high and x-high.
Fable two-shotted it in about 30 minutes.
In my experience open models (or GLM, DS and Kimi) are radically worse than either of Claude or ChatGPT at these tasks.
I think there is a huge "long tail" of tasks like this where the frontier labs are ahead, and I think this long tail is quite important.
Hardware and electrical costs including power usage and electric wiring/outlet costs of such machine.
Unless you are spending more than a max subscription (200 a month+) its cheaper to use the cloud.
But things are priced cheaper in the cloud now to lock you in and restrictions around hosted models are getting worse.
If you only have a $300 dollar laptop its probably not worth the upgrade.
I'm personally excited by local AI but the experience for the average isn't the same. I'm willing to get .5/s running on 10-15 years old machines but what I can do with it is limited.
Their marketing department :-) . I'm only half-joking; those guys are hard at work finding the best product-market fit for ChatGPT/Claude. "Product market fit" means "strongest revenue", which is not necessarily going to bring the best tool for you or me, but the one that can either get more consumers to shell off money, or more enterprises to cough money for licenses, and in both cases those consumer basis will be narrowed down to what legal and geopolitical circumstances allow OpenAI and Anthropic (and this is why they want to ban the competition!). It also means dark patterns and enshitification, of which I'm already seeing some both in the Codex interface (it was just renamed "ChatGPT"!!!) and in Claude Code (which also is just "Claude" now and can't '@' properly any longer). So in the medium run most people will be better off running an open source harness that can use any model.
you can't create a new drug by running a model on a laptop. You can't serve a customer support bot running on a laptop. You can't generate video in bulk for many users on a laptop. So there is still a case for paying others for inference.
Does it justify the valuations? No idea, but some major use cases are still there. That's why they are rushing to implement, OpenAI creating a "deployment company", Anthropic having some pharma rumors, etc.
yes, but at that point even open weight models are not free. Can someone set up the best open-weight model for free and charge pretty much just for GPU usage+operations+margin for inference? yes. But they would have to keep up with how good the private models are. The bet is private models would sell in a similar way at a premium and invest in staying ahead. Is this bet going to work? no idea, don't shoot the messenger.
> yes, but at that point even open weight models are not free. Can someone set up the best open-weight model for free and charge pretty much just for GPU usage+operations+margin for inference? yes.
Or, the company itself can rent the GPU and host the model of their choice on it.
> But they would have to keep up with how good the private models are.
Why? The whole point of the thread is that the marginal gains of using private models is a fraction of the additional cost of using those private models.
IOW, if you really need that improvement over the open models, you can go ahead and pay it. What we are seeing though is that the improvement in general is barely noticeable but the cost is 4x as much.
Can they still go public ? MiniMax M3 Pro is also coming, then DeepSeek-v4-Pro GA, then GLM5.5. There will only be bad news for them in the coming few weeks/months.
US AI labs really rub me the wrong way, especially with the doom and scare tactics they use. Both Altman and Dario keep talking about how AI will replace workers and how we should regulate LLMs for national security, Dario’s main point.
LLMs are useful. We can all see that in agentic coding. But replacing everyone’s job? Hardly. And what’s with the scare tactic of trying to get the US government to ban foreign models?
LLMs are useful, and dare I say they’re on par with the internet. Making them cheaper and affordable is good for everyone. The fear mongering from Anthropic and OpenAI looks like an attempt to corner the US market into using only US models so they can keep the profits, especially since China has proven that LLMs are a commodity. US AI labs should work on making LLMs cheaper or better harness. Altman and Dario are not trustworthy.
> "Anthropic believes that they are the ones who should have final say over how Anthropic is used; given that they think only they should be developing leading edge AI, they by extension think that only they should have final say over AI generally. When you further combine this realization with the company’s pronouncements about AI’s ability to conduct all economic activity, you realize that Anthropic’s leadership effectively wants to have power over everything and everyone."
To be fair, we're simultaneously mocking anthropic for believing in safety so much and also for them thinking they're the only ones that care enough about it. It's true that no one else seems to care as much. Judging by reactions from everyone, all their safety talk is very bad PR.
The criticism isn't that they believe in safety too much. It's that they patently using safety as a red herring with the actual goal of regulating away competition that they know the cannot beat.
The people involved have been talking about safety long before they had any users or a company.
"Concrete Problems in AI Safety" by Dario and another founder was published 2016, anthropic was founded 2021. Among a bunch of other examples, including other founders.
If safety is seen as a joke now, it was really seen as a joke back then. The reason they keep shooting themselves in the foot with bad PR is because they truly believe in risks.
I’m sorry, but this do-gooder concept doesn’t jibe with their other actions. For example, swindling their business partners (such as Figma). Filing an unnecessary trademark lawsuit against a customer whose brand predates Anthropic.
And for safety? Oh yeah, they literally sell the removal of guardrails in exchange for minimum spend commitments.
For example, swindling their business partners (such as Figma). Filing an unnecessary trademark lawsuit against a customer whose brand predates Anthropic.
Completely unrelated to AI safety.
they literally sell the removal of guardrails in exchange for minimum spend commitments
It seems blindingly obvious to me that you'd want to let trusted organizations use the models defensively, which means without guardrails, and that you'd probably gate that behind some kind of enterprise sales process, for multiple reasons.
Honestly, it just seems like you have an axe to grind and aren't really particularly knowledgeable (or curious) about AI safety.
What they believe now or care about is fundamentally not very relevant: allowing them a monopoly would be a big mistake. Google was supposed to do no evil and it probably started that way but nothing guarantees the future other than having alternatives.
If there are genuine society risks in a tech I don't want to discourage CEOs from talking about them. I feel like we've spent decades talking about how evil chemical companies (etc.) were about covering up issues in the 20th century. But yes, that's different to being a reason to ban external models.
I agree re: Dario quietly pushing for government control. He also said LLMs would replace a lot of entry-level information jobs, doubling the unemployment rate from 4-5% to 10%.
but this crap may take forever to play out even if the outcome is well-known. Self-driving is "here", it's obvious that once it's cheap enough having a human behind a car wheel or a freight truck wheel is an absurd waste of human life (kinda like digging canals with bare hands instead of an excavator), yet truckers and uber drivers are still employed. But everyone knows the writing is on the wall for them.
It's not so simple, if such a headline can get them closer to the regulatory capture they want to lock in American businesses and forbid them from using Chinese AI.
AWS Bedrock still lacks the support for the latest open weights models like GLM 5.2, DSV4 Flash 0731, Kimi K3 etc. Even they only support Qwen 3 which is a very old model. Any specific reasons they are reluctant to support open weights models?
AWS Bedrock do support many, older, open weight models. Perhaps there is a reluctance to support anything more powerful that would compete with their partner Anthropic?
In practice I'm not sure how big a deal this is - Bedrock is not the cheapest or best provider (try Fireworks AI or DeepInfra who do support more recent models), and would seem to be of more interest to corporate users who already have an AWS account.
Perhaps also of relevance the US government is hassling US companies that are using Chinese models (currently DoorDash, previously AirBnB & Cursor), which may limit demand for corporate use.
I was wondering the same thing. Maybe it’s a licensing issue. Or the models are too big and there isn’t enough demand. But size is not a solid argument because they also don’t support Qwen3.6 27B and 35B-A3B, especially since they just added Gemma 31B and 26B-A4B.
> How Qwen Cloud handles your data during inference?
> Qwen Cloud does not use your API inputs or outputs to train or improve models. > Learn more about how your data is handled during the inference process from Zero data retention.
If this is true, this is a big deal for me, but unfortunately I cannot find anything in their legal agreements, so this marketing sentence is worthless.
2.4 Trillion parameters with open weights releasing next week? The open-source community is going to need a collective GoFundMe just to buy enough VRAM to host this monster.
I like that Alibaba are emphasizing more challenging real-world "coding" tasks as well as just benchmarks.
The "Reproduce a research paper — then improve it" seems particularly impressive as well as actually useful - perhaps as close as we're going to get to "recursive self-improvement" given that these are data-driven not code-driven models.
I'm not sure how much real world use of AI is 100% hands off agentic coding, but I guess for objective evaluation purposes it needs to be something without a human in the loop.
It'd be interesting to see a comparison of each company's best models on a suite of real-world challenging tasks like this, but I guess difficult since each group of tasks would need to be one-time-use only to prevent subsequent benchmaxxing which makes comparisons useless.
Qwen3.6 35B-A3B Is my current backup model after Ornith—1.0, which has performed slightly better and faster at Python, db and some backend tasks. About 75 tokens/sec on a 5060ti 16gb depending on context size (usually 64k)
At $2/$6, it can be much more "approachable" than K3. Wondering what have they done differently to be able to afford this price (that K3 couldn't, apparently - most other providers offer similar prices to K3 itself).
Moonshot is printing money on k3. It likely costs the same to serve as qwen3.8. The license requires all major inference providers to sign an extra (secret) licensing agreement with moonshot that almost certainly requires them to agree to this price and pay royalties to moonshot. Watch as the k3 price plummets over the next 1-2 weeks.
If k3 is open-weight (and it's available on hugging face [1]), how could they force providers to sing an extra (secret) licensing agreement?
Edit to answer my own question:
License file [2] states:
> If the Licensee or any of its affiliates operates a Model as a Service business,
and the aggregate revenue of the Licensee and its affiliates exceeds 20 million
US dollars (or the equivalent in other currencies) in total over any consecutive
12 months, the Licensee must enter into a separate agreement with Moonshot AI
before using the Software or its derivative works for any commercial purpose.
Licenses are not copyright law, they are contract law. Copyright is automatic and does not require any terms, as long as the work is covered by copyright. Contract law can then override the innate copyright rights (for example to permit people to use your copyrighted works).
If you create something that isn't copyright-able, you can still create a contract which people must agree to in order to use your work. The enforcement/validity of that contract then depends on making sure the user knew about the contract and agreed to it. The way they did that here (just publishing a LICENSE file) could be interpreted as a judge as no confirmation of agreement of contract - depending on if they find a "reasonable" user would have known to look for the file.
Copyright is an automatic protection (meaning you don’t need to register it, like registered trademarks), for certain types of creative works. Contracts are one way to dish out these rights (eg. for a fee).
If you have a work that is not copyrightable, and you give it to people lawfully, then you cannot protect it via copyright.
Qwen is Alibaba. Alibaba rents hardware to Moonshot. So Qwen always has the option of cheaper hardware. Qwen might also break even on inference as competitive advantage since Alibaba has wider pockets. Alibaba also owns a 36% stake in Moonshot, which must make pricing discussions interesting... but then again Google rents hardware to OpenAI and Anthropic.
I have a 5080 Super RTX but its still not enough to run these big models. I use Gemma 4 right now on a Debian Linux with no GUI that I access remotely. Its works pretty well, but still not as good as any of the big models like Opus 5... Can't wait to be able to buy a personal home server that can run much bigger models. I heard some companies have started building AI PCs only used to have a personal AI model at home, have any of you tried one yet?
Their base models and architecture has quickly become the go-to for local inference and fine-tuning, even when they introduced some tricky things like GDN, so many people use it, that it was matter of days/weeks until lots of OSS frameworks adopted it.
I used Qwen3.8 Max Preview for 2 weeks and can't tell if I truly miss Fable.
Qwen doesn't overuse em-dashes, weird sentence structures with bold text: normal text.
In my private GitHub Repo Qwen finished as much tasks as Fable did. Without hitting 5-hour session limits. Qwen's token-per-second performance fluctuated greatly. From 20 tps up to 80 tps.
Qwen failed on some "reading between the lines", but so did Fable.
The only noticeable difference is skill loading. Until "Must use" in a skills front matter, Qwen tends to be very lightly loading skills. Superpowers works, because of the same Must use wording. All my local skills use the same wording now. For installed Skills, it's more or less my responsibility to remind Qwen using a certain skill.
I used Qwen together with Qwen Code. I didn't compare Qwen+Claude Code.
With my Owl code scanner, Claude and Qwen produced similar results. Qwen repeatedly used the same output format, despite no format provided and disabling memory. Fable varied greatly. Both followed the file format output and produced valid files.
With Qwen 3.8 being now a visual understanding model, I don't really miss Fable/Opus/Sonnet for my private projects.
At work we are still exclusively using Anthropic models with Claude Code. No change incoming on that front.
Privately, I'm sold on Qwen and Qwen Token plan. No session limits, many open weights models available via a single API.
Since it's horrifying thinking about what US companies do with my data, it's no worse or better thinking about what Chinese companies do. It's a choice between the Plague and Cholera.
The models are good even by skeptics standard, it's just that evangelists are overselling the capabilities. If you understand the limits of LLMs not using them as a business is shooting yourself in the foot.
However, they are not at the point where they can effectively train themselves, nor did they are capable of researching their own method of learning. SWEs in mid-corps on my country are right now relegated to reviews and sanity check, basically babysitting the LLMs and making sure they're not spouting nonsense. If you think about it, that's basically QA and can also be delegated to another AI. If Bun's rust rewrite that they tout as fully LLM-led can pass the test of time in a year or so I think that's it.
I believe all that is now constrained by compute and capital, not tech.
I mean its pretty obvious right? This models are not flawless and sometimes reach stupid conclusions so there needs to be some one who watches it. Thought i must say u are right. Every one of them pretends that this new model is gonna finally take ur jobs lol
There's infinite work to be done, so higher productivity makes people worth more. (Obviously this doesn't apply if AI can do everything but we're not there yet.)
The percentage of the population that needs to work will continue to go down because of aging and automation although you might not see it as a reduction of hours for a given individual employed person.
Already 38% of Americans aged 16 or older do not work and are not looking for work and yet are not dying from hunger or exposure. This would have seemed like a utopian dream to someone from the 1800s.
Being 16, 17 and 18 and not working is expected, given they are still at school. A good percentage of people aged 18 to 24 are studying full time.
This stat seems a bit misleading and doesn't support the "you can live and not work argument". How many of those people are retired? How many of those are under 18? How many of those simply cannot work? How many of those live at home or are on food stamp?
Maybe you should ask AI to review your statistics and reasoning. A large amount of 16-22 year old kids are in school. And retirees 80+ are retired. I don't see how any of this is relevant to the idea of what would happen to the economy if there is an employment collapse. I don't know that this would also seem like a utopian dream to someone in the 1800s; I'm not sure what that means. I think someone from the 1800s would be surprised by a great many things and that statistic would probably be low on their list.
> I love their video with the idea that people will be able to do their hobbies while ai does their job...
Are you not already experiencing this? I think this is fairly common for people using AI now, though the time may not always go into hobbies or sports. It's common for me to setup Claude with an hour+ task while I catch up on housework, or while I'm getting ready in the morning.
In the last couple of weeks I've unfortunately had multiple family illnesses - it has been helpful to have Claude keep up with much of my product development programming work while I visit my mother in hospital and check on my father's recovery. I'm able to give more time to family without worrying that business progress isn't keeping up. The overnight Claude sessions while I'm asleep have been particularly helpful.
We are in the golden period where this IS possible. Once it becomes the norm to "do something else while your agent works", we'll be asked to do more WORK while the agent works, rather than do hobbies/housework/nap/etc.
In a 30M+ LOC repo we blew out Qwen3.6’s context thinking about a code review. This was via direct call to model. Do harnesses facilitate better context management, or is there something else to accommodate its smaller context window?
What was it reviewing? Was it just a "take a look at this pile of code over here"? Because I think asking any model to review 30M lines of code is a stretch...
in general with automated code reviews I've found it fruitful to tell the model what specifically to look for and where. I usually don't get over 10% of the claude opus context window for code reviews, but it of course depends on how your code is structured, how much the agent has to explore etc.
Does this desideratum make any sense? The whole point is that you write to it in English or Urdu, and it writes the specified code. If you cut off everything but the code writing, you cut off everything.
Domain modelling as well: we have clear programming concepts, but the kind of autocomplete that can autocomplete a customer sales process needs to be able to refer/leverage ontological connections on words like ‘bass’, ‘season’, ‘rebate’, and ‘jamboree’.
Intuitively it feels like focused models should be better models, but human programmers are ‘better’ knowing programming alongside general stuff. We’re not reduced by reading economics or Tolkien, removing such knowledge would be premature optimization.
Thanks..non AI expert here just wanted to understand..cos one of the most talked about topics is running models locally on cheaper hardware. Helps many esp in poorer places.
Would be nice if a genuinely useful coding model can fit into an old laptop, give decent performance and know nothing but (say) Go or c language.
no, apparently, otherwise we'd already have specialized models. every bit of meaningful human-generated data appears to improve the overall capability of the model.
Kind of yes, but this be get you are pretty stupid autocomplete yuo can probably avhieve without an LLM at all.
>we have professors with specialisation
Yes, but any professor with a specialisation is an erudite with vast knowledge across the board.
I'd argue we can have a meaningful BIG model with a specialisation but not vice versa.
PS: all of this is about a model that is expected to be able to execute development tasks on a human level. Obviously we can have small models that are very capable in things like TTS or STT.
I guess it can but it will be useless. After all the model superpower is awareness and ability to guess and infer some stuff. Right now a model saves you time not only by coding faster, but that it can figure out some stuff about the shape of the data and its purpose.
If you throw general purpose model at a codebase - it will look at the table and data logical connections beyond what is explicitly declared. It will figure out on its own that Salaries should be displayed on SalariesTable.php and it will "know" that your prices should include vat and so on.
A human knows that VAT and price go together and are related, full size LLM does too, stripped one - doesn't.
I'm trying and failing to find value running a potential Qwen 3.8 27b dense model on a 16 core, 128 GB of ram, 2080ti box. Yes, the GPU yells for help, but the problem is that no math works to upgrade this machine even when pouring $200 in rent every month into the large model providers...
How are you all justifying economical use of these local models right now? What's the cost efficient way to do this and do better (even with models evolving over time and losing now vs later) than the big labs?
You will simply not get more value out of running a local model vs paying for a subscription/API from the cloud in 2026. There is no math that will make local models come out ahead in $/intelligence/token.*
The point of local models is privacy, offline use, and maybe no guard rails.
* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.
Points are starting to be made in favor of value, to the contrary of what you are affirming. Specifically because the new open weights models lower the TCO of hardware in an environment where new open weights were previously thought to be a thing of the past.
There is for $/creativity/token. LLM sampling settings are poorly supported even in open source serverless providers but are the single best lever you have for getting better outputs in regards to creativity (and quality for long context or highly quantized models).
There's no point, except if you want privacy and independence. I am playing with a personal assistant that checks my emails, calendars, sends me an agenda and maintains my TODO list. I am not sending such data outside.
Running locally for me is mainly about learning, maintaining control+privacy, and helping shift my coding+design process to leverage LLMs. I guess if you made me boil it down to a single word to justify the cost I would just say: tuition.
Sounds like we have similar boxes - mine has a 10 core CPU, 64 GB of ram, and a 2070 Super. My motherboard had two unused PCIe3x8 slots and doesn't support Blackwell GPUs. I bought a couple of brand new Ada generation RTX 2000s with 16GB of memory for under $1400 to get to 40 GB of VRAM. That will easily run Qwen3.6-27b at a 6-bit quantization and 80,000 token context size. It isn't fast (19-21 t/s), but using pi-coding-agent is fine.
Now, my instinct is that I am giving up SOTA performance on agentic coding with this setup and LLM. But the gap between my setup and SOTA commercial models is small enough that it doesn't matter to me.
The only scenario is if you have enough work to do batch inference. Using a tiny fraction of GPU capacity to decode a single request at a time just doesn't make sense, as you say.
> $200 in rent every month into the large model providers
We all know that is hugely subsidized, and I guarantee that OpenAI and Anthropic are looking to enshittify that ASAP.
The enterprise users, however, are not subsidized like that. They pay per token. And some developers in those companies are chewing down a lot of tokens. Self-hosting an open weight model could be a massive savings very quickly. It also gives them negotiation leverage when talking to OpenAI and Anthropic.
Well, it is always possible that OpenAI and Anthropic have some secret sauce that they are hiding that nobody knows about. However, the pricing of tokens on places like HuggingFace and OpenRouter suggests that probably isn't true. In addition, the lack of reliability and general slowness of Opus and Fable at various times also suggests that.
Now, the one wildcard in all of this could be Google. They are on the eighth generation of their TPU and have been holding their cards extremely close to the vest. I don't think anybody has a good read on exactly how much capacity they have. Most things you can kind of figure out the overall business numbers and what's going on in Google--the TPU area is one of the exceptions. I know a couple of big customers and even they don't have any visibility on that front.
Yes, I don't. I have my own harness where the agent is only able to do a set of predefined things and none of them involves the internet. However, I imagine the model could be a lot smaller if it does not have the baggage of coding and programming in general.
I mostly just need it to mostly be a very advanced NLP model that is able to figure the set of tools that it needs to call with what argument. Then it will just follow a predefined decision tree.
AI is a commodity. This is proven now. And valuations will have to drop 90%.
There's a lot of AI models that each 'win' one week over the next. The pure definition of a commodity market and not a 'winner takes all market' as valuations would imply
> In this case, Qwen3.8-Max was asked to create the oh-my-cli project from scratch and, over a 10+ day long-horizon autonomous coding run, build a self-evolving harness.
They don't explain how successful that went but it's a bit hilarious seen that an Anthropic dev explained that it's been 15 days Claude was hard at work --with nothing to show yet-- trying to rewrite itself in another language.
"You rewrite Claude Code, we rewrite oh-my-pi."
"You're nowhere after 15 days, we do it in 10."
Sure, it's apples to oranges and all that. But part of me thinks they know fully well what they did there.
It's meaningless. Models have always been able to do this and this capability is strengthened during RL since being able to explore the solution space to figure something out will give it a reward.
What is important is how long it can go without requiring human intervention. Not just that it's possible to run on its own for a time.
The second sentence of the page: "This also marks the first time we will open-source the weights of a Qwen-Max-class model — the open weights will be released next week."
Reddit has become a fairly worthless site because people just ask questions instead of taking anytime at all to figure out the answer. He just needed to wait for the page to load instead of adding noise to the discussion
As someone who is searching for a new programming contract right now, reading all of the incredible abilities here is pretty intimidating. Especially since I get almost all of my projects from Upwork which is an outsourcing site.
I believe I am competing directly with these frontier models in some circumstances. Like there are a ton of programmers who previously would be outsourcing work to that site, but now they assign that same work to AI agents.
Ever since November 2022 when ChatGPT blew up, I have been focusing on agents in order to try to get ahead of the curve. But I haven't managed to get an agent business off the ground and have been doing poorly paid agentic projects from that site instead.
But now everyone is building agents, and this crazy list of accomplishments makes it look like we are close to the point where the agents are building agents.
In fact the next time I get an Upwork contract for another agent, I actually should run it through my agent and see how far it can get. What I'm seeing a lot of now is requests to automate as much of a business as possible.
Anyway the point is these models are just about capable of doing the entire job of analyzing a small business and building out all the agents and iterating on them with the business owner.
That's actually what I should build is a SaaS that does that. Which I would if I wasn't basically desperate to get another contract this week.
And I know Upwork is bad but I have not had much success with other options on short notice.
> Anyway the point is these models are just about capable of doing the entire job of analyzing a small business and building out all the agents and iterating on them with the business owner.
If you search for LLM benchmarks focused on real-world tasks, you'll quickly learn this ain't the case. No point in telling business owners about that though, they need to see/learn by themselves.
> they need to see/learn by themselves
The cleanup pay is mighty generous as well.
I just finished two solid months of my life cleaning up a vibe coded app.
Has anyone gotten a position as a post-vibe janitor yet?
You got problems there too, but I don't do shitters. That's Engineering.
What does the benchmark even mean when people are using AI to make real world things that solve real world problems?
I see people, and my self making amazing things with AI and fixing old projects and having real world impact at the fraction of the cost it would take me to hire people, or hours spent on my own coding.
I have built tools and systems with AI that have allowed me to build windows drivers, android apps, web apps, iOS apps, vm occultation, custom block drivers, custom file systems and more. To the point where entire products have been created.
Not trying to be a doomsday, but yes. It seems as though with the right infrastructure we are at the point where businesses owners can go from idea to product very fast and not need or hire much external talent.
What amazing windows drivers have you sold?
Idk about sold. But it’s loaded on all the windows machines in a fairly big company that solves a real world problem.
It allows us to apply custom ACLs to AI agents and the child process spawned by AI agents. Giving us the ability to control what files an AI agent can read or write to, while still being in the calling users context. It allows us to force all ai derived processes to use a transparent MITM proxy so we can then also apply robust access rules to remote host allow or deny access to specific urls and not others. It also allows us to monitor access to windows Credential Manager with rules ti allow specific singed binaries to access some credentials but not others. It give us complete control of what AI agents on windows can see or not see or access.
Windows native sandboxing is lacking. You have some stuff in WSL that completely are broken once you call a windows native app. Or you have app containers which are too restrictive and result in applying expensive file system ACL to all files the app containers would access, which can take hours when dealing with million of files, and would be required to be applied every time you chains your app container (there are some workarounds, for them but they still have a one time cost a long with a fairly flaky maintenance process). You can get the network part done by running commands as a different user but that would result in the same file system ACL nightmare that app containers has.
Result is we get seatbelt level sandboxing in windows native, and can apply dynamic rules like preventing access to .aws folders regardless of the OS level ACLs, using glob rules like */.aws, so we don’t have to be aware of the exact path ahead of time.
It also has registry tree ACLs and, can prevent process and process trees from gaining administrative access, the list of features goes on and on.
so you decided to vibe code a sandbox because windows sandboxing is lacking, instead of moving to linux where sandboxing is robust?
Some products and companies require windows.
It might please you to know that this driver was vibe coded from a linux vm based sandbox.(that was also vibe coded)
A better question is which products/services have you avoided purchasing.
If only LLM benchmarks could benchmark it in the first day!
Still no Artificial Analysis benchmark yet. Or benchmark for Laguna S 2.1 or Meituan models or lots of other models.
According to a few tasks from my little personal coding benchmark it's very good at coding and kinda bad at web design. (Also excellent at "draw me a picture" one-shot prompts, for whatever that's worth)
On a sneaky one that involved parsing MIME headers and dealing with character encodings it did better than Kimi K3 at Max and for 38% lower cost.
Interestingly it seems noticeably better than the qwen3.8-max-preview model they offered just a few weeks ago.
Im sorry to hear about your situation. Have you blogged about it? I’m curious about how the volume of your type of work has been changing over the past 2-3 years.
> Im sorry to hear about your situation. Have you blogged about it?
I don't know why but this made me laugh out loud. I know you're trying to help but just a funny jump
I stopped blogging years ago when I realized that only bots were reading the posts.
I think the volume for custom agents is probably higher than ever but so is the competition for that work.
That working is as not common as college dropouts becoming successful entrepreneurs. But both become stereotypes. But I know you might also just want to read about it.
PS. Not everyone (in fact the rare few) write as if no one is reading. For most, literally no one ever reads :D
As someone else who is an IT consultant.
You need to run away from any client who thinks your primary purpose is to write code. You must run away from the business if YOU think your primary purpose is to write code.
You should be able to write code and do it well, But the AIs, and I'm not even talking about the ones on the frontier, have been able to write code faster than I have for quite some time.
What you need to explain to your customers is all the things around writing code: software architecture, performance, and so on. Also make sure you have some understanding of the customers business, so you can suggest additional ways to make their lives better/make more money.
Software developers have a chance. Code monkeys? None.
The problem is Claude Fable is now better than most programmers I know at software architecture and performance optimization as well.
That's a popular type of AI cope.
First, I am aware they have been able to write code for some time. I made my first LLM coding agent experiment the day after ChatGPT first blew up in November 2022.
I haven't been a "code monkey" in the last say 15 or 20 years of my programming. I would argue also that code money was never really a thing, it was just an excuse that pretentious overpaid developers used for the existence of underpaid exploited programmers. 95% of programmers have to do requirements analysis and design regardless of their pay grade. I learned a long time ago how important requirements analysis and good design is. And the jobs I am applying to largely specify architecture and requirements analysis in the description. Employers are aware of code generation tools.
But also, the premise that writing the code is not the hard part is just nonsense. Sure requirements analysis is harder than people realize, good design is hard and so is understanding things about proper and closed iteration. But that stuff is not, in the aggregate, harder than the code.
And also, frontier models can absolutely do requirements analysis and architecture. And the sheer speed of implementation means that there is a huge built-in advantage for iterating more effectively.
Two points of disagreement:
"Code monkey" absolutely was and still is a thing. I was hired by Big Name Co originally to be staff but a reorg had me do typical senior position duties for around a year, and it was hell. Every decision was made over our heads, feedback was unwelcome because the feature had already gone through design and gotten layers of sign-offs from different areas of the org, etc. My sole purpose was to write code and review peer code. Staying there a full year was way too much.
Second point, requirements analysis is hard because:
* Different stakeholders have conflicting goals, sometimes mutually exclusive * Some stakeholders have ideas that are actively harmful to users, and you must flex political capital to nip it in the bud * Some stakeholders think they know what they want with a far greater degree of clarity than they actually do, and lack the time to sit down and suss out the nuances, because you're a side-show in the grander scheme of their own priorities.
Writing code is a heck of a lot easier than dealing with people at the edges.
Those are good points, but that mostly comes down to lack of organizational status or office politics etc. which is independent of capability level and largely orthogonal to whether it's a human programmer or an AI agent. Unless you are redefining "software engineer" to be more like a political operator. But that goes for any human group and any job. And anyway after a certain level, not providing the senior engineers and or project managers with sufficient political sway to realize projects goals is a management failure. Unless again you are defining software engineer as someone who is a master social manipulator in the context of their job.
I thought this comment was going to be about leveraging agents in ways your existing clientele demographic cannot or will not (yet|ever),
but then you totally jumped the shark!
Can you give me a hint about wtf you are talking about?
To say that you're building agents, is based on a way of viewing things that isn't at all pervasive. Some might say you're building a customer support chatbot. To talk of building agents as a common activity makes sense if you use LangChain I guess, where its title is "Open Source AI Agent Framework | Build Agents Faster". But for instance, YC just released qm, and in its README it talks about customizing the agent, which is quite a bit different from just casually dropping the phrase "I'm building an agent". https://github.com/yc-software/qm
Not sure what you are trying to say but thanks for reminding me about qm, it has some cool features.
My MindRoot framework had some of qm's main features awhile ago though. I do try to use it to build solutions by customizing rather than from scratch when possible. But a lot of clients or potential clients don't like that idea even though it's MIT. They want some that has clearly been invented by them as a new programming project.
I actually turned down a project last year from someone who wanted me to start over in LangGraph -- he had already decided on it before he knew about my thing.
The most recent project I have been using MindRoot and building up the voice capabilities so I can now handle voice agents end to end including SIP.
But on that website now I try not to emphasize my framework at all for most proposals. You don't have leverage on Upwork. And also with code generation it isn't a critical point necessarily.
They've also announced Qwen3.8-27B being released open-weight next week. Qwen3.6-27B is widely regarded as one of the best local models, especially since nothing else comes close to it, that isn't benchmaxxed, without being significantly larger. If 3.8 truly improves upon it that would be awesome.
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
Qwen-3.6-35B-A3B was our "gateway drug" into switching our organisation to agent/harness-first coding.
Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode.
It continues to be a workhorse to this day, running on both my local Mac for various types of jobs, an AMD R9700 at the office, and said teammember still uses it on his 5090, although in practical terms we do a lot more with DS-V4-Flash-0731 these days.
I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks. Let it run overnight and wake up to millions of output tokens worth of results without data having left my house, all for the price of electricity.
I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though.
DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models.
For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs.
But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.
> I’ve run 3.6-27B and 3.6-35B on 32GB locally for a lot of bulk non-code tasks.
Do you mind sharing your use cases?
Not OP, but driving knowledge bases is the poster child use case for me https://github.com/aka-rider/llm-wiki
I started with Karpathy's LLM wiki, and did everything he said not to do - downgraded the model to mere tool usage and summarization, and it works great.
I am a data hoarder, and finally I can just dump all the content I remotely like, and get something interesting to browse for the price of electricity.
Agentic long-running tasks, as others have mentioned:
- Groom and triage tickets for agentic SWE workflows
- bug hunt — the probability of Qwen fixing a complex bug is 50/50 but often it is capable of identifying the root cause or at least laying the ground work for a more capable model to pick it up.
Love the workflow. Is there an existing way to do this inside Obsidian -- where I have all my knowledge?
SmartConnections but heads up, Connections Pro asks $300/year, or, more for the plugin than for an LLM subscription, more than Microsoft Office for that matter.
https://smartconnections.app
Keep in mind Obsidian is open standard JS plugins... You know what can write open standard JS plugins?
Classic which comes first, LLM or the plugin, though. :-)
Not OP, but I use it for a ton of smaller things. I have it hooked into Hermes and have been using it to help bulk rename my media folders so they all follow a common format, add titles that sort of thing which wouldn't be easy to 'script'. Another thing I use it for is comparing data sets, looking at my exported Spotify artists and compare to what I have locally, and letting me know where there are missing artists, or albums, and recommendations based on similar artists that I may not have locally.
Sure a lot of this could be done without AI, but it's certainly quicker and easier, and since my AI box is on solar, it's just the power of the sun to keep it going.
Not OP, but I use it for personal tasks that are just not worth the claude tokens -- rooting through historical medical records to unify prescription history, super-OCR'ing thousands of PDF pages (i.e. beyond PDF dumping -- vision means it can look at tables, understand tricky things like a continuation of a block quote or aside on the next page, etc.), and when power is cheap I'll just let it noodle on little projects on my data. During my agent's "free time" I give it with a tremendously open ended prompt last week, it did a linguistic analysis of how my texting changes in the lead up to, initiation, maintenance, and ending of romantic relationships.
That is exactly what got me past just enough of my cynicism to get started. I am still cynical but now I have meaningful knowledge.
> Qwen-3.6-35B-A3B
The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something.
I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.
We’re using 6bit quants since we have 32GB cards.
Gemma QAT is an honourable mention.
What!? You are skeptical of AI but will go through the manual process of hosting a model that’s less than frontier intelligence (talking about Qwen 3.6)? Anti-AI folks are always odd to me
A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
Needs 0 investment and 0 committment?
- You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI.
Or, you could sign up for Opencode for $10 and just be productive.
I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
> Or, you could sign up for Opencode for $10 and just be productive.
You forgot the step before where you spend months waiting for security to vet it, legal to sign off and procurement to approve it.
Or you could use hardware your team has lying around. Everyone isn’t working on cloud-hosted CRUD APIs.
You might be anti-ai in the sense you aren’t comfortable with all your data being shipped back and forth to a third party.
Im anti AI in the sense of VC backed global warming far right accelerationists.
Local AI is almost perfect. But its like all democracy: its history is marred with lots of crap.
install LM Studio, download the automatically selected quant based on your hardware, start a conversation with the automatic context size. 10 minutes at best and zero effort
Maybe not '0 investment and 0 commitment', but incredibly little depending on what you have laying around. It takes less than 5 minutes to download say LM Studio and an Open Model and as long as you have the hardware to support it, you start moving along. If you are on AMD in some ways it's even 'easier', you can download Lemonade and it will tell you exactly what will fit and best options based on what you are trying to do.
For me at least the local AI stuff, powered with solar has been pretty great. Would that scale to a large business? Goodness no, but for my tinkering and learning, it works great.
All of your objections have already been addressed by the previous comments.
The original comment states that the person in question already had a suitable graphics card to hand, so it did not require a monetary investment.
GP clearly states that "someone who is into that that stuff" was guiding the process, so it did not require a significant time investment.
> I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
I see no hypocrisy in the original comment.
You've also assumed the skeptic in question doubts the capabilities of AI. That may be the case (like you, I have no idea), but they may also have privacy concerns, in which case a local model is the appropriate choice.
There are plenty of reasons to be skeptical of AI.
> You at least need a capable machine, so that's not 0 monetary investment
It is 0 monetary investment if I already have said machine lying around doing nothing.
Which is exactly the story OP talked about.
But most people don't have an RTX 5090 lying around, so the story doesn't apply to them, right?
You don't need a 5090 to run local AI. A whole lot of people out there are doing it with Macs. Unified ram is the biggest thing.
I'm running Qwen 27B no problem with an AMD 9070XT + 24gb DDR5 ram. Does basic web search for me (tool call with tavily, costs nothing I get 1000 searches a month) and is great for creative writing (primarily breaking writer's block). Until the recent surge in ram costs, that wouldn't be hard to do. I built the computer for ~$1600 a year ago.
I am getting ~13-15 tps with my 9070XT for the 27B (~35tps for the 35B-A3B), but I think for me the main bottleneck is the 64gb of DDR4 3600 memory. What kinda speeds are you getting with what speed of DDR5?
Correct. If the premise doesn't hold, it's ex falso quodlibet for anyone.
Back in “the day” nerds just bought the hardware to fuck with. Some of us still do. Claiming that compute is the barrier to entry just means you’re not a nerd. That’s ok.
> You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model
Pi has a nice guide on it (https://pi.dev/docs/latest/llama-cpp) and it is really not that hard.
How is that hypocrisy? Self hosting is somehow anti AI? Its not anti AI. Its literally using AI!
…and honestly, at a higher technical level than slapping your wallet against a token provider and running prompts in a hosted sandbox you can't even see the prompts in.
Look man I am incredibly skeptical of how LLM’s have been rolled out and all the promises people make (it’s so much snake oil and pipedreams), but I also found it very trivial to hop on LM studio and start tinkering with models. If you’ve already got a decent midtier computer on hand, which I imagine a lot of us already do, then it’s really not hard to get started and get immediate results.
Local models on regular hardware aren't really capable of anything. Whatever you're testing is nowhere near a measly $20/mo subscription, so it's of limited use.
i really like the idea of running local models but i'm always in the position of wanting the best model(s) available and i don't have any severe privacy concerns. as such i have yet to justify ever using local models.
This is the diametric opposite of the rent-vs-buy scenario that this entails.
Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware.
Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine.
> takes literal minutes to get started
Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers.
Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute.
> you can just `rm -fr` it and forget the whole thing existed.
I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it.
You have it totally backwards.
> I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument.
I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
> you're having issues handling files properly?
I guess they were using ollama, which does not tell you where it puts the models it downloads.
Filelight / ncdu are my friends for finding random 30GB directories containing cached models.
Personally, I prefer QDirStat. I just tried to use FileLight to compare, but the package seems to be broken on Lubuntu.
It took me about three hours total to set up a local model. I already have a GPU and I have fiber for the download. llama.cpp is not difficult to compile and has many backends. It can run parts of the model on different backends, like in the common case that the GPU doesn't have enough VRAM for everything. There are many step-by-step guides available.
Takes even less depending on your system. LM Studio or Lemonade and you are set up in minutes and now they can even tell you what models will fit with the memory you have.
> "I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!"
I used to deal with these kinds of frustrations too.
Huh what? Qwen3.5-35B-A3B runs just fine with maximum context, on an RTX SUPER 12 GB, with offloading of some expert layers to DDR4-3200.
Same story on an RTX 4060 Ti 16 GB. MTP is a serious boost to tg.
Downloading the model is a simple hf command that HuggingFace's web UI even gives you.
llama.cpp is trivial to use, and so is llama-swap, if you want to use other models too.
If you don't know what arguments to run it with, you download ggrun and use that.
Local LLMs are incredibly capable and don't need expensive hardware. A $500 GPU will do. Or even cheaper.
This is all trivial.
Full model or a 4-bit quant? I have a 5090 and I'm not sure whether I should use a quant that fits within the VRAM or a much bigger version where I'd have to offload a lot to 64GB RAM and a beefy CPU (but still a CPU)
I personally run the Q6 quant on my RX 9070 XT (16GB VRAM). On r/LocalLlama there was a post recently as well, which talked about the degradation of different quants (for the 27B version)[0]
[0]: https://www.reddit.com/r/LocalLLaMA/comments/1vef79c/quantiz...
RTX Super 12GB costs $700. An openrouter account costs nothing.
A lot of people already have 12GB+ GPUs lying around for playing games, doing video editing, etc. I would not get a GPU or mac just to run LLMs personally, but if one wants to get such a device for other tasks too, it may make sense to eg choose a slightly higher (v)RAM variant if they want to run some bigger models. Then what you pay for the local llms is just the difference.
> Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers.
Download LM studio, search models, click download, wait minutes, prompt and have fun
This largely describes me. I'm skeptical of AI in that it's capabilities, while very impressive, are vastly oversold and overblown. Being skeptical of AI is not being "Anti-AI". That's largely the AI data centers are using up all the water and electricity types.
Maybe you're anti-AI because you're really anti-outsourcing your thinking to some remote corporation you don't control?
That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime.
Local AI solves for all of those.
I'm not against AI. I'm calling out the hypocrisy in the comment. I'm anti-AI, but will spend hours trying to setup a local model, instead of just getting access to frontier intelligence in 15 mins, and actually getting useful work done.
If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
I think learning how to set up a local AI now requires less time than learning about potential pitfalls of token-plans and processing payments in corporate environments.
It's not that billing is complicated, but learning to set up a local AI is a lot more useful and more rewarding.
35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter, so I daily drive that. I wish there was a ~100B MoE with maybe 10B active. It would be super smart and fast!
There was a 3.5 122B 10A release -
https://huggingface.co/Qwen/Qwen3.5-122B-A10B
I tried it for a bit, and It was not really worth its size. It got swept up in all the other AI news recently, but laguna s 2.1 I think is the best ~100B moe model right now
I didn't mention it above, but Laguna S is my other favorite model. I use Qwen a lot more, it's smaller and faster, but I like to switch to Laguna when I feel like I need a "heavy hitter" for certain huge or complex tasks.
What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
Strix Halo, 128GB RAM. I got a refurbished Corsair AI Workstation for a smoking price ($2100) about two months ago. Lucky timing that it was in stock.
Strix Halo as well. Bought it for $1,800 new on sale and shoved an extra 4tb drive into it. Been amazing for local AI. Maybe not the absolute fastest thing (usually around 30t/s depending on the task) but has been awesome for a local AI box that I can solar power.
Nice. Mind sharing the solar side of your setup?
Couple of rack mount batteries and roughly 5kw of solar panels. Feeds into a subpanel so I can flip it when I want a couple rooms of solar on the house, or hook a generator up if needed. Can't power the entire house, but works well for thinks like computers, lighting, etc. And if I want to expand, just throw on more panels, or realistically, just throw on more batteries to store the juice.
Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have.
brb, going to see if 2nd hand mac studios are available!
Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.
I don't know if that's, well, a rule of thumb, it might be, well, straight multiplication.
Oh, that is a useful rule to know! Thanks!
That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.
And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.
Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.
Yeah, here I am sitting deeply deeply deeply regretting not buying couple CMP 170HX at $200 or $350, knowing I could just flip them ethically at purchase price if nothing came of it... I could have just casually built a 128GB dual A100 local AI monster
I'm working with a lab that has a few Ampere GPUs on infiniband and they are just not compatible with the latest quants and vLLM updates. FP8 is about as low as you can go.
But they're reportedly a soft nerfed GA100 64GB/40GB at $1200, that's not more expensive and certainly can't be slower than a Mac Studio.
Usually theyre quantized. Also, there was a window where AMD 395+ W/128GB was just a high end $2500 hardware with unified gpu memory.
quantized + offload
I have an RX 6700 XT with 12gb vram and 64gb system ram. running dense models like 27b is difficult, but i can run IQ4/IQ5 qwen 122b-a10b or 35b-a3b at ~20tok/s
IQ4 qwen 122b-a10b would mean 61GB total size and 5GB active, so about 5GB of the model loaded into GPURAM plus any generated context, and 61GB of weights loaded into system RAM? I don't know if that math is correct, but does that run well? Wouldn't that only leave 3GB of system RAM?
dgx spark, nvfp4 so I have spare room for KV cache (context)
MoE models can use system memory along with a GPU.
and get high token bandwidth?
Not badly so because MoE models(identifiable by "CoolName-xxxB-AxxB" naming scheme) have bunch of branches in the middle that only one out of all gets non-zero values. Each of branches aka "Experts" as well as top/bottom parts are significantly smaller than the whole, and so CPU emulation of CUDA operations mixed with GPU taking as much as possible become not so out of question, unlike for dense models("CoolName-xxxB" without "-AxxB")
Similar to a spark, which isn't blazing fast but usable.
Using qwen 3.6 27b for local coding as well and downloaded Laguna s 2.1 but haven't had time to give it a full spin yet.
Curious for any more experiences
I agree. 27b dense really did seem like the sweet spot.
A 35 A3B as smart as previous gen 27B would be a sweet point
> 35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter
Isn't that just the definition of MoE vs dense ?
Full name is 35B-A3B. 3B Is the token generator thats selected out of the 35b available in the model, which is some layered jazz.
So it can be dumber but its quite capablr.
I use 27B in plan mode and 35B MoE in act mode. I noticed that is the best balance for me for consistent tool calls and intelligent planning. Takes some time to switch, but it's worth it for me.
Wonder if it's possible to share a common cache via cachyllama between 27B and 35B-A3B
I've heard 27B is smarter! I tried it some time ago but couldn't get it working with my oMLX. I need to try it again.
Honestly the 27b dense one punches way above its weight in a lot of domains, especially coding in my testing, so I think you will probably be disappointed.
In my case I would say they are comparable but moe models are looping and getting lost a lot more than dense models.
On the other hand having 90t/s with any local model is nice and Pi with loop police extension can prevent looping a lot.
Looping seems related to quantization and not the model itself. If youre digging deep into quants to get working context then yeah.
It was for me too but the new deepseek pricing is too good to ignore for now.
I honestly think that with my electricity prices running qwen 36B myself is more expensive than hitting the cache rate at deepseek.
Can you elaborate on DeepSeek (deepseek-v4-flash, i assume?). What does your typical usage pattern look like and what is your weekly/monthly spend?
I gave it a try for a few days (pi + openrouter + deepseek-v4-flash via deepinfra) and ended up paying ~$18 for rather light usage. Yes it's still cheap, yes it's fast, but i feel i would still get a better deal with a Claude subscription plan.
DeepSeek without OpenRouter is wayyyy cheaper
Is it? OpenRouter shows DeepInfra being cheaper than DeepSeek directly
https://openrouter.ai/deepseek/deepseek-v4-flash-20260731#pr...
I agree with parent. OpenRouter might be cheaper list-price, but i have been using 10$ on DS platform since April/May, still have 2$ left. Using OpenRouter i depleted the same dollar-amount in a 1-2 weeks with same usage pattern. No idea why.
What does your usage look like?
DeepInfra's Cache Read is 6.5 times more expensive than DeepSeek.
And in many scenarios, cache hits are 99% of tokens, so the price difference in caching really adds up quickly.
Interesting. My anecdotal experience is that it is.
Going directly though DeepSeek's API and hitting all day long with light/medium tasks I'm at $5/month. That's with pretty vanilla ohmypi.
I would recommend looking into Ornith1.0 - it's using Qwen3.6 35B-A3B and excels in coding, at least for my coding needs, Python, web-dev, SQL scripting and some C#. Using Pi harness.
I have been using Qwen3.6-35B-A3B as my daily driver as well and its been phenomenal when it comes to coding
How do you use a 72GB model as your daily driver locally?
Apple Silicon. But: there's no need to use the FP16 version. At 8-bit precision the quality loss is almost imperceptible. That cuts the footprint to 36GB. Which is great for a 64GB Mac, because you have room for plenty of context. 6-bit also works nicely at 26GB + context.
You want to use the newer quantization formats like Unsloth's UD quants or oQe, where the weights are selectively quantized using a calibration dataset so that important weights are left at/closer to full precision.
use a quantized version. since it's MoE, what matters is that the 3b parameters that are used for every token fit in gpu vram, the rest can stay in system ram. really great if you don't have unified memory.
At 4 bit it easily fits in 32GB. That's what most people use.
What kind of machine do you have running that? My attempts at local have always resulted in a very hot lap
I host the models on my Mac Studio, an M1 Ultra with 64gb ram (I bought it when it came out, just happens to be good at LLMs). So when I work on my laptop, I have my oh-my-pi setup configured to use the models on my Mac over my local "bonjour" network or whatever Apple calls it. That way I have a nice cool lap, while using models that my M4 MacBook Air with its 16gb ram couldn't possibly run.
Cool yeah. I got a M3 with 32GB ram and it’s a little iffy. I’ve considered getting a MacMini to act as an in-house
Strix Halo for me. If I am running something on my laptop, it's a much smaller usually around 12b model, but those are a bit less functional. I mean I think there is a a ROG FLow Z that has the Strix Halo setup, but that thing was super expensive.
With laptop being ...?
Is your Qwen3.6 locally run on your laptop? What kind of tokens/s are you getting from your laptop GPU?
I have Qwen3.6 35B-A3B on my laptop and it does 60 tokens/s
Could you share the specs of your laptop?
M5 Pro/64 GB RAM
compared to claude - how 'fast' is it in terms of throughput on your laptop?
On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.
Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
what hardware do you use or recommend for this? never heard of it until today.
Strix Halo is the unified memory platform from AMD. Similar to the DGX Spark from NVIDIA or the M series Macs.
I personally have the Framework Desktop, but there's also systems from other brands like Bosgame
You can also get it in a laptop form factor that feels like a MBP with a nicer keyboard if you get an HP Zbook G1A!
Huge fan of that thing, it's th e Linux MBP I've always wanted.
While the laptop option is nice, for an inference server you're probably going to want the desktop form factor as it has significantly more thermal overhead and thus better performance. In the desktop models most of the internal volume is a gigantic heatsink
I have a framework desktop, but depending on your need, DGX spark might be better. The prefill and NVFP4 is a significant advantage. But framework desktop is a better general computer. I expect to be able to use it for years to come. Where as DGX Spark you’re at the mercy of NVIDIA BSP.
RTX 4060 and above. Ideally RTX 50 Series, because you can run NVFP4-quantized GGUFs that give you better prefill AND better quality.
On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama with completely default settings (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)
Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).
It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).
What do you use to pair it with web search?
Depends on how you are doing it. LM Studio and tool calling models can use the web, or you could go for something like Perplexica, or if you want to go real crazy, something like Hermes or OpenClaw.
I use searxng.
I'm a big ole noob when it comes to local AI. What are you using for a harness? Or platform to interact with it?
I recommend trying pi.dev as your agent harness for local models. In my experience it has been the sweet spot of functionality (which you can and should extend with plugins) vs performance (OpenCode just swamps local models on my hardware).
you can try ollama, omlx or llama.cpp for instance to download a model and get an inference server running locally. They expose „open ai compatible“ endpoints, so you can configure almost any harness to use them.
On what hardware do you run the model locally, if so?
Not the GP, but I run this model as daily driver too. It runs great on a Macbook Pro 64GB (M3 Max). Token generation speed can be about 100 tokens/sec with multi-token prediction, although it depends on the context. Worst case speed is around 50 tokens/sec.
The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.
I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.
What are the specs of your laptop and what tokens per second do you get?
It's just a Macbook Air with the base M4 and 16gb ram, but I'm hosting the models on a Mac Studio with M1 Ultra and 64gb ram that I had purchased when it came out. I get about 45-55 tokens per second with this setup. I think I could get more if I spent some time fiddling with the parameters, but I don't really know what I'm doing there so I've just left most of it on oMLX's defaults.
I'm on the verge over here, the new Anthropic models have been a disappointment. I've tried the A3B variant, but had mixed results. What do you use as the coding agent, and have you heavily customized your workflows?
What laptop?
It's just a MacBook Air with an M4, cheap and nothing special. I host Qwen on my Mac Studio, an M1 with 64gb ram. The model uses around 20-25gb ram depending on what it's doing.
I wonder if I could get this running on my 48gb M4 Pro. Haven't been able to load anything beyond 27B
Qwen3.8-Max is the first in Qwen-Max series to be open-weight as well.
Kimi K3, GLM 5.2 and now Qwen3.8-Max - open weight models.
DeepSeek V4 Flash outperforming Gemini 3.1 pro, probably DeepSeek V4 Pro update is also coming soon
Chinese labs are cooking very hard. US closed weight labs are probably hard time to resist not calling Washington DC for more AI regulations
Kimi K3 is more like "weights available" in that you can download and use them but it is under a custom license that has a bunch of limitations where you have to pay Moonshot for doing some stuff. GLM 5.2 on the other hand is plain old MIT.
Not sure how Qwen3.8-Max is going to be licensed, hopefully it'll be Apache like the smaller ones.
You can do whatever you want with the model within your own organization. If you use it commercially—either as a model-as-a-service business or in a very large-scale product—you should check the additional license terms, which go beyond MIT. My interpretation is that Moonshot cares about the exact inference behavior and accurate representation of their model or derivatives, and perhaps also about capturing some additional value despite their own GPU limitations, so the extra license terms focus on those large-scale commercial deployments.
Considering that very few orgs are going to be able to host a 3T parameter model internally, chances are most deployments would be subject to these restrictions and require a separate license from Moonshot.
probably will be few clicks in dashboard of your org's cloud provider.
Any US organization can rent compute.
Kimi’s policy is extreme reasonable.
US AI companies are already sweating and 100% pressuring the Trump administration for more anti-Chinese regulation, since there have already been talk of Trump considering banning Chinese models. There's however another push back from the startup industry urging them not to ban it, since it will stifle the innovation. In other recent news OpenAI also greatly cut their model prices, 20% for 5.6 Terra and 80% for 5.6 Luna, to stay competitive.
Sure, it’s just the lowest end for now, and the enterprise money is at the top of the market. And there’s protectionism/enterprise lock-in/etc that complicate things somewhat.
But still, if the US AI labs ever tap the training brakes for a millisecond, the “inference is still a money maker” argument seems to evaporate when they’ll immediately have to fight a race to the bottom until margins are virtually nothing.
Or if the benchmaxing “line goes up” FOMO mindset starts to lose its luster and companies find their individual niches for productive use of AI and stop bothering with all the latest and greatest churn for top dollar.
Which might be even worse if it means the training arms race is still ongoing but neither Anthropic or OpenAI want to be the first to “lose”. While the marginal value of each new model training run keeps decreasing and enterprises signal they’re more concerned with cost reductions than solving ARC-AGI-7 puzzles.
Huh? V4Flash is still incredibly price competitive. The new version is right around GLM 5.2 and maybe slightly worse than opus 4.8 depending on which benchmark you use while being much cheaper(even factoring how most US zdr providers charge 10x Deepseek’s api caching price). K3 is also a tad behind fable/sol while being alot cheaper
Some advice I got from another HN Mac user was to run local models in energy saver mode. You'll get slightly reduced tokens, but the laptop won't overheat and the fans won't go wild.
And if you're running it on a dGPU, power limit it, because you lose very little in terms of token generation performance, since it's memory-bound.
Oh. I've been using an icepack under my laptop to keep mine cool. I'm watching it with llamatop to see if the GPU is actually active or not, aw activity monitor wasn't showing me what I wanted.
That's exactly how i fried a laptop. The condensation killed it.
This is what I've been waiting for. We are still using fine-tuned deployments of Qwen3.6-27B with a lot of success but could use a bump in intelligence. Here's hoping.
How and where do you finetune it?
I use Modal for fine tuning and unsloth mostly.
Easy to do with Unsloth Studio.
Really awesome. Though I wish they'd do a dense 48B, 60B or 72B.
There seems to be quite a gap between the small ones and the enormous ones these days.
Qwen 3.6 27b has been the sweet spot for me in terms of local models. I've had good luck using it with Pi harness. Looking forward to this.
I also "evolved" into 27b (q8 unsloth) and pi.dev (tried many combinations) feels for me the same as opus 4.5 that i use at work, faster even (using 2x 3080 20GB gives me 60-80tk/s). Though you do need to feed it more details up front (about what exactly you are planning to do and a good written skill.md) but I work that way anyways, im hyped for 3.8
Having invested in a machine with 128GB of RAM, I would love seeing something a bit larger than 27B / 35B, possibly a 54B dense model or 70B MoE would be much closer to the Qwen 3.8 Max experience.
All of us with a 96gb rtx 6000 would love to see a 70b moe. Maybe they are waiting for OpenAI and Anthropic to IPO so they can short their stock and release. Local LLM is going to get very interesting in the next 2 years.
The 27B have many more active parameters than much bigger models such as DS4Flash, MiniMax etc, which makes it punch above its tiny weight. A great fit for a 5090 in a closet for meat-and-potatoes, kind of work.
I've been running Qwen3.6-27B-IQ4 (4-bit quantized) locally and it's been great. I can't run the non-quantized version as I only have a 4090 w/24 GB of VRAM and it won't fit and leave any context room, but the quantized version only uses 18GB.
Yeah, Qwen3.8-Max is the new Flagship model for coding and harness system and many other benchmarks are reaching equal performance as Claude and other close models. That's gonna drop the price of LLM in agent landscape a lot.
the bonsai 27B 1bit quant version of Qwen3.6 27B is even more nuts, model fits in 4GB, and with 100k of content model+kv cache fits in 8GB. I’ve been running it locally on my mac mini 16GB. it gets around 4-6 tok/s, so not quite real-time ready, but good enough to let it run on task async for 20 min and come back. The 1bit model struggles a bit with multi-turn conversations though (e.g. when switching from plan to act mode it will still keep trying to make a plan) but that’s easy enough to reformat prompts into multiple one shot sessions of smaller work.
You consider that even Gemma 4 31B is not even competing with Qwen 3.6 27B?
For those of us who don't have the time to follow closely, Qwen3.6-27B being Open Source and Open Weight, what level is this compared to other Western paid version?
Just so that we know what 3.8 would be like.
I currently have about 150 Tabs of Antirez posting on AI and running local model I haven't had the time to read. And there are probably some prerequisite reading or other research in between as well. I just wish there are some very high level overview and news coverage on all these.
> what level is this compared to other Western paid version?
IMHO this is a difficult question to answer. Part of the power of paid models comes from the software supporting it. With local models, you have tons of workflows that can severely influence the quality of the result.
In my personal experience, the SOTA models are way more consistent and can handle more complex questions. Part of that is (probably) because I don't let my local model access the internet, while paid models do use the internet to look at docs etc.
You absolutely need to let models access the Internet if you want consistently good results. Pretty much any non-trivial task requires the model to do things like look up APIs, code examples, or existing discussions of a given topic.
You don't really need the Internet. Tons of documentation is available for download (either as a zip, or with the documentation site as its own git repo). Wikipedia is available for download. You can get reddit dumps, HN dumps, stack exchange dumps, etc. This can all easily fit on one hard drive.
Reading the actual code is also always a better source of truth than docs anyway (this is true for people and LLMs). Just clone whatever libraries you use.
If they trained it well, and can do computer use, it will be a new era. Companies can keep PCs, put Qwen 3.8 27b on it and get rid of the employees, lol...
Yes let’s get rid of employees so no one is employed but somehow they can afford to buy my stuff
It's the natural outcome of next-quarter short-termism. The board and C-suite will be fine (monetarily).
I am surprised that they keep going with it, seeing how fast it improves and basically soon running themselves too out of business.
What's even their end goal? Open source models make sense, if profit is not the target, but for OpenAI and the rest, once they achieve "AGI", don't they basically become useless?
There was an interesting interview by MLST with a team doing well on ARC AGI 3 who are using Qwen 3.6 27B, and said that it's actually better at coding than the larger 3.6 35B.
I guess which of the smaller 3.8 models is best for coding will depend on which one they put the training effort into.
The larger 3.6 35B is actually a mixture of experts (MOE). This means a small proportion of those B's are actually active. It's fast and suitable for agentic tasks but nowhere near good as the dense 27B model, which has all of its parameters loaded.
I have found 27 to just be so much more coherent than 35:
https://humanparadox.org/local-vs-frontier-benchmarks-for-my...
It can complete multi-step tasks much better, and has a bit more curiosity.
Do you have a news source for 3.8 27B pls?
https://xcancel.com/Alibaba_Qwen/status/2084100707423289643#...
as much as im excited for it, sadly it gonna be one of the reasons to push ram prices higher
This makes me wonder if AI companies even have a MOAT in the first place.
All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days.
And that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars. Or all of them are worth a lot less. One of those statements is true.
Also this makes me wonder if the next iteration of LLMs would be based on fine-tuning, where LLMs actually learn from your past behaviour so that it would grant some amount of stickiness to the product. OpenAI used to offer fine tuning runs for GPT-3.5, but they don't seem to do that anymore.
> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history
A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
Which is why big labs have been working hard on making their harness not be stateless any longer:
https://earendil.com/posts/session-portability/
“Just take the session thread to another provider” might not be feasible anymore soon-ish.
While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow.
The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow?
If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
Soooo… when can we expect an encrypted handoff.md to fully prevent session portability, then?
(Only half-/s)
And this is why you should not use the model provider's harness. It is to tight of a coupling of services that's only business purpose is lock-in.
Knower
Ding ding ding
Which is why folks should be jumping to pi / oh my pi as soon as possible
This is also why anthropic no finger lets you use the flat rate sub in non-claude-code harnesses
(They do but you are charged at api rates)
As a heavy pi user - this doesn’t help, for codex. The thinking traces are encrypted and you cannot move that session over to eg your local ds4. It will lack those precious thinking traces that you absolutely need to continue a session productively.
They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time.
This is harder to do on other architectures that themselves aren't fully deterministic though.
Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245...
My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences
an article explaining this in detail : https://news.ycombinator.com/item?id=45200925
Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction.
You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".
If you send the seed along, you can have a non-zero temperature: when it's time to select tokens, a PRNG will be seeded and used for the selection.
Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.
Perhaps they just used the exact same random seed everywhere?
I appreciate the replies on the determinism point and I’ve learned some new things here. In any case I probably should not have tagged that on, as my main point was to share that the sort of property that parent is talking about (whether true for all LLMs/providers/harnesses or not) is statelessness, not idempotency.
LLMs are, in theory, deterministic. Sampling is not intrinsic to LLMs.
Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.
But all of this is down to CUDA and/or kernel implementation issues.
It's a more or less solved problem (e.g. DeepSeek has batch-invariant kernels, vLLM also has reproducible outputs etc), with an obvious caveat that you need to lock your hardware setup, model snapshot, and inference stack if you want any useful reproducibility across longer time periods. It's up to specific providers to actually implement it to any extent, of course commercial providers are incentivized to do the opposite.
According to [1] there is - unfortunately - increasingly more state in LLM sessions.
1: https://earendil.com/posts/session-portability
>LLM responses are certainly not idempotent, as they are not even deterministic.
Isn't that more due to an optimization and not how the LLM itself runs?
Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.
For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.
Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.
It is much more subtle than your specific example, which is strictly speaking a bug, though ofc it has been used during pretraining for efficiency purposes. Sglang and miles have been working towards full determinism in open source codebases, so the LLMs can help explain the subtleties encountered in actual projects if you point them to these repos.
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.
Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks.
https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down.
Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.
I meant that companies like Anthropic are locking in users with proprietary formats in their harness where it's hard to leave.
Agree. Harness can not be a moat. There are many open harnesses and they are at least on par with the providers ones. It looks like Anthropic/OpenAI's approach to vendor lock-in is not so much the inference or the harness it is functional integration across the individuals and teams in a company. I don't think this will be a moat either, but I think it's all they have outside of compute.
Surely the moat is the training data... with the data you can explore new architectures much easier and get step changes in performance.
The training data, at least up to now, is very abundant and basically every lab has the same data from scraping the Internet. RLHF data is what's now valuable.
Arguably the majority of codebases are not available on the public internet
Is it RLHF if there is no human feedback?
And yet investment is continuing. What are they counting on?
Bubbles aren't always rational.
One possibility is that they're counting on another bailout package like the banks and others got in 2008.
The bets are absolutely huge and have been spread throughout the financial system via securitization. If the number does not continue to go up, a lot of people are going to find themselves backing their numbers with money that simply isn't there anymore. Therefore, the number will continue to go up and we (meaning American taxpayers) will back whatever it takes for that to happen.
Now, were the bets stupid to begin with? Of course. Who wants a machine that potentially unemploys major percentages of the population? No one. But the bets were made regardless, and then more bets were made on the hypothetical returns of the first round of bets, and so on and so on.
> that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified.
They aren't, not even if we forget about the capable Chinese models.
I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.
They have 2 moats.
The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are.
The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor.
If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.
compute is not a moat, it's a rapidly depreciating physical asset. buying up all the shovels in a gold rush does not give you a moat, it gives you a slight advantage for the time being. someone else will just start making shovels. and the data is clearly available, hence the number of open-weight models.
Isn't a literal "moat" about temporary deterrence? I can imagine makeshift bridges could permanently make the moat useless.
well the boards have been laid across the two shores, and the enemies are now climbing the walls, i guess is what i'm saying. if they had a moat, it's been breached
your post helped me realize a change Meta is pursuing on Instagram that is to give more weight to captions and long text posts so they can have more data to training that would usually go to websites/Google. Even AI slop is good for this.
China has the moat that they are cheap/free/open. The US corps have the moat that the other option is Chinese models. At least for some time.
> This makes me wonder if AI companies even have a MOAT in the first place.
They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.
Burdensome regulatory compliance is a moat.
These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through.
But all the money in the world cannot move government too quickly.
Other moats exist too. OS or browser can undermine performance and availability of alternatives.
> This makes me wonder if AI companies even have a MOAT in the first place.
Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style.
Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.
With their current API approach they're essentially a commodity. They need to start moving parts of the harness behind the API, otherwise they'll remain a commodity.
Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.
> They need to start moving parts of the harness behind the API
This isn't without it's challenges however.
1. This will increase costs drastically, since they would need to a run a sandbox per use to ensure data isolation.
2. Increased latency, and this directly limits how much of the harness can be moved to the cloud before the users notice sluggishness
Oh, I'm sure. If it wasn't a bit of a tricky needle to thread we'd have seen them start to do it already.
3. Users can still move to better open harnesses
> Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars
It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.
There's no mandate that says any of those bets are going to generate a return
This isn’t technically true. Most model providers don’t send the thinking tokens anymore, so if you switch from one provider to another, you will be missing large parts of the conversation.
fine tuning runs of models the size of gpt 5.6 are absurdly expensive. I'd guess at least $100k in cloud gpu time for a single run, and you have to do a few iterations to get things right
The major labs don’t allow assistant prefill, so you have to “summarize”
Mamba/SSMs could change this picture.
The moat is the US government.
I think you're right and I think it's why Google have taken their pedal off the metal for model releases to focus on integrations and tools. And why Microsoft have backed off from the OpenAI partnership to do the same. Anthropic and OpenAI are going to massively struggle to maintain their pace and reach profitability just selling commodity tokens.
Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.
They have a moat; they don't have $1 trillion valuations.
Which anyone who hasn't been sitting in the SV echo chamber could have told you years ago after applying even the smallest bit of thought.
From my experience these open source models are nowhere near the performance offered by Fable/Opus/GPT-5.6. Whenever I tried Qwen, Kimi, Deepseek, the results were much worse and it just took much more time to get something usable. When you consider that, the frontier offerings are still much cheaper.
That might be true right now, but how long until you have to move the goalposts? In my experience with DeepSeek and Kimi, they're as capable as the frontier was four months ago, which already solves a big chunk of the coding tasks that I'm interested in.
What provider did you use ? Synthetic's Kimi is a beast
The visual web development / perceptionbench scores are very promising for image->html flows. Here are some test results.
Original designs: https://image.non.io/257dc9cb-9e6b-4e00-8f12-23ea5e073649.we...
These are fairly rich, pattern-heavy, nuanced designs. I've asked each to create it as a SPA where the map flows behind it.
Opus 5 results: https://html.non.io/opusAcmeBooks
Qwen 3.8 max results: https://html.non.io/qwenAcmeBooks
Same prompt for both for the conversion. I used OpenCode for the qwen version, but I encountered a significant amount of errors / timeouts while it was running. Claude finished in around 16 min, but I spent close to 2 hours shepherding the Qwen build. For the implementation, there were signs it had good vision, but the timeouts make this very hard to use in a production setting.
Update: Tried using Qwen Desktop. It told me it succeeded, and linked me to the folder it created it in.
That folder didn't exist, I asked Qwen where the files were, and it attempted to create them again.
Stopped it, and asked what happened to the original files. Qwen Desktop apparently doesn't have access to the local file system, but continued merrily along without warning about that.
https://image.non.io/07a153cd-c31c-4f7b-a89d-363faec05a91.we...
Also tried Qwen Code. Tried it with a coding plan and it 401'd. Tried it with an api key I loaded up with pay-as-you-go credits and it 401'd.
It did a self update and it worked for a single request (me just saying hi). Pasted in the prompt to build the page and it 401'd.
AGI confirmed
Wow the Opus version is a lot more functional (try clicking some links).
I'm quite surprised at the difference.
So Opus introduced a container with overflow:scroll in the middle of the page, while Qwen didn't? That's interesting in terms of "smartness".
I've had nothing but trouble with Qwen on opencode. GLM too. I know a few people who use them on Ollama could without issue though, so I don't think it's a model issue.
Both horrific (from a user's perspective, perhaps not the underlying code).
I've noticed another type of AI slop that's prevalent in all the popular models; font sizes and variations like you wouldn't believe. It's very hard to instruct LLM's not to do this.
I thought it was pretty cool, its the kind of stuff that I wish there was more of on the internet. I guess some people's imagination runs a little more than others.
Whilst these coding models improve, they seem to cater for multiple languages, and for many, a trimmed-down LLM that supports just one language would be fantastic. This would be smaller, more able to run on the hardware people have at home, realistically (even on the CPU). Having one LLM that knows assembler, java, bain, C etc is neat, but when you only work and use one language at a time, it would be kinda neat to have those broken out into dedicated, smaller LLMs. After all Qwen3.8-Max handles over 90 programming languages - ask any programmer to name a list of computer languages and if they get over 20, they are doing well.
I believe the knowledge of multiple languages brings something onto the table that having narrow language knowledge does not. The LLM will be able to better generalize a problem, apply universal concepts and avoid mistakes that would otherwise be too "echo chambery". Even as a human, knowing multiple languages has made me better at my primary language.
That's not how LLMs work. If you're talking about number of parameters, you wouldn't be able to reduce the size much by "removing" support for other languages.
Yeah, I'd assume it's possible to extract all languages as steering vectors from a model and then substract the ones you don't need from its weights.
However, that would just change the weights values and not their dimensions.
Can you please explain why?
Only a tiny, tiny fraction of the parameters are encoding information that's specific to a particular programming language. Even if you could remove those without degrading performance, it would have a negligible effect on the model size.
Not who you responded to, but I’ll give it a try.
Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):
0.728201017
-1.9293839303
2.93020202929
5.82290101728
-0.2844920917
…
And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).
Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:
0.428201017
-1.6293839303
2.63020202929
5.52290101728
-0.5844920917
…
(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)
Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.
Please do correct me if this simplified too much or I had a mistake in my understanding.
This is oversimplified. The proposed question is whether a model with fewer parameters could achieve performance on one language similar to that of a larger model that’s been trained more broadly, which isn’t straightforward to do.
Oh, I didn’t interpret the above question as asking in that direction; but yeah, that’s of course something I didn’t attempt to answer with my comment.
Although I’d be intrigued in the answer to that small-narrow vs. large-broad model question, too!
The exception being the token embeddings and lm head (which scale with the number of tokens the model knows and presumably you need a smaller number in the tokenizer for only English and python). But those are a pretty small % of the total model weights on most LLM sizes
For a set parameter size, if you benchmark an LLM on a single language, you'd get better results by training it on multiple languages rather than just the one. Scaling quality training data almost always increases performance as the neural network picks up general rules and an understanding of code that are language agnostic.
I think you'd want to remove e.g. knowledge of harry potter universe and ancient egypt. Training on a bunch of high quality java code bases is still likely to improve your python model.
Actually, it's shown that even general knowledge helps coding models because their input is natural language itself so they need to understand it well enough to even turn into code.
Teaching model to code in say python does indeed increase its effectiveness in other seemingly unrelated areas. On the other hand I remember that early models of ChatGPT were great in chess notation and later models aren’t as good as OpenAI doesn’t prioritize this now
It was a matter of time for China to catch up with the US. In terms of infrastructure, manufacturing, and engineering workforce, China has the upperhand and I foresee them becoming the SOTA leaders. Maybe if the US wasn't so busy gatekeeping and keeping things proprietary, they would've had more trust from the open source community.
First and foremost, China isn't energy-limited like the US is. A modern grid, lost of cheap power coming online every day. The worlds most advance ultra-high voltage transport links are tying the deserts in the west to the cities in the south.
It probably sucks for you if the proposed powerline goes through your house, but the need of the many come first.
Having cheap and plentiful power is a huge market advantage. It was one of the factors driving Norway out of poverty 100 years ago.
I think this is really a rosy image of China’s electrify situation. They have huge needs in the east and green energy in the west, they can’t build their UHV links fast enough. They are doing ok at keeping up, but there isn’t a surplus of cheap electricity lying around in the same places where you have cheap water as well.
What is stopping them from just building their data centers in the west where they have energy surplus? Not like you need to locate a data center inside a city.
Water. The west is really dry. We could throw in data lines as well but it’s really bottlenecked by water. Actually the only place in China with water in abundance is southeast China.
Also they really don’t have the chips to supply those data centers, and the government isn’t so crazy to go that aggressive on AI, they have lots of other priorities as well.
Chinese data centers like to use air cooled chillers for cooling, not evaporative coolers. It's simple because power for the chillers is cheaper than water for the evaporative coolers. The opposite is true in the US
Evaporative cooling is a matter of convenience and cost, not a necessity. Other cooling technologies can be used. Intel has already proven a closed-loop system can work just as well.
One of China's big sources of power is hydroelectric such as the Three Gorges Dam, so if you want electricity and water in one place, then there it is.
I wonder if submerged system with coolant could mitigate this ?
There is economic benefits of the cost per a lumen continuously coming down.
The US also isn't energy-limited. They'll just price out all humans living there from buying any electricity.
EDIT: This was obviously meant to be facetious with the intent to highlight the negative effects on the general population that datacenter buildout has in the US.
USA vs. China energy production graph is wild https://stanfordreview.org/content/images/2025/10/us_china_p...
States have had trouble delivering expected EV charging capacity btw
Why aren’t we considered energy limited?
That graph features a completely-made-up superexponential AI growth curve coincidentally reaching 100% of US electricity generation in 2030, but even if you take the prediction up to 2026 at face value, it also agrees that AI is currently a small fraction of overall demand. Of course any individual data center site might struggle with insufficient local supply, but China would have the same problem if they wanted to build bigger data centers. (Production capacity is actually north of 25 000 TWh, but poorly utilized because of a mismatch in place and time between producers and consumers.)
> The US also isn't energy-limited.
I have not looked into it, but I have heard a lot of new data center have to run on site gas turbine on trailers to have enough power.
If the US is not energy limited, why they do that?
See your edit, got it :)
> but the need of the many come first.
The need of the rich, you mean. In these kinds of situations it's never about the many, if it was about them different choices would be made.
Also give it to Chinese labs for the vision of going open source and open models to compete. Apart from their great models, I quite like how they publish papers of their research too.
> trust from the open source community
volunteers in the open source community use whatever is easiest and cheap.
open weights is not open source. there is no "trust".
I think the window for a ban of open weight models is closing fast so let's hope US administration is going to miss it and we get Fable-level models (at least in some aspects) with open weights without infringing any newly introduced law as a long-term local baseline.
Even if the US does ban it, Europe due to its lack of European competitors, has no current similar protectionist incentives. Even if there is regulatory capture in the US, the rest of the world isn't going to follow suit until the current market leaders outright are replaced with other competitors that Europe would want to favor. The genie is out of the bottle in the West already.
> lack of European competitors
There's Mistral. Although they've been surprisingly quiet for the last 6 months.
They are profitable and active on the enterprise local model territory. You can RL a model with them for your own purposes and I heard good things about it.
Mistral's business model works even if they start using Chinese models instead of training their own.
If China keeps releasing LLMs with such permissive licenses it probably works better since the pretraining rnd is subsidized and de-risked - but that’s a big if.
> There's Mistral
https://trackingai.org
Mistral: eq.IQ ~70
Kimi, Claude, Grok, Gpt, Gemini: eq.IQ ~130+
France might do something mad and take that route, but EU wide ain't happening for Mistrals sake.
Would be hard for them to act alone due to free movement in EU. Would be hard for them to outlaw a German business from offering the models from a German datacenter to French customers.
Theres a pretty big chance imho that europe will follow suit , or at least the segmenets that are fully aligned with the US on all things china even to their own detriment. Remeber rip and replace Huawei 5G because of national security , leading to some parts of europe having worse coverage than some tiny african markets. Best bet would be for europe to look at it like what it is an open source commodity that is useful to build on rather than a moat to extract rents with.
Doubt it now when US declared that Europe is an enemy, Europe is banned from the best models and that US is considering an invasion of Europe (Greenland). The days of US influence in Europe is limited.
If it was 2021, probably. But Trump has soured relations in the last 18 months enough that the soft power about technology isn't there like it once was assumed to be. What's he going to do if Europe doesn't step in? Jack up tariffs? Is it that day of the week already?
Open weights is more beneficial to Europe spring-boarding a catch up than defending Claude/Anthropic.
Yeah… but last 5 years happened and UE is now in the overdrive to decouple from Usania :D
If CADA passes in its current state I doubt we'll be banning Chinese models any time soon. I think most people misunderstand the extent of the permanent damage caused by Trump
He threatened to go to war with Denmark over Greenland. In the US, that might just be a side story, a little joke from insane grandpa president. But it's not in Europe. And it's only one of the crazy adversarial things he did.
well, the EU just brought down the regulatory hammer which complicates things a bit. we will have to see if it operates as a proxy ban on open models depending on how they go about enforcing all the safety, bias, transparency and data provenance requirements.
I mean there is Mistral?
I mean it’s not competitive at all?
It’s more that they have a different business case than competing for the top spots on public benchmarks.
They seem to be oriented more toward customizing models for the concrete needs of a company, on-prem deployment, proprietary knowledge-bases, etc.
It has been more competitive than any other LLMs by non-US/CN companies. Including Korean and Japanese models in case you want to bring those up.
A distant third is not saying much.
It has produced some locally useful models.
>Europe due to its lack of European competitors, has no current similar protectionist incentives [...] The genie is out of the bottle in the West already.
I agree, but you are underestimating EU regulatory incompetence
Yeah, but the world is not limited to the US so who cares…
If it's fable level then I am sold. I didn't have so much success with Qwen compared to claude x supermax plus thinking, no matter what I did.
What can the US administration do about it?
Send you to jail.
Worked well with alcohol last time
It sure did. Even Al Capone went to prison.
How can the US administration send me, a citizen of Sweden, to jail?
Sweden is dependent on the American nuclear umbrella. Until that changes, the US will have a lot of leverage on Sweden.
I mean they did manage to make Swedish authorities go after Pirate Bay.
I mean, it's not like they have done it all over the place already...
No.
What they always do. Send in armed men with guns? Export Controls. Import Controls. National Security Laws.
Good luck doing that in China?
You can send armed men to your house, or restrict companies in America from using these models.
They can also restrict US providers from serving these models. Looking at OpenRouter, there is currently 11 providers for 'DeepSeek V4 Flash 0731' with the only non-US one being Deepseek themselves, which aren't even the cheapest.
> the only non-US one being Deepseek themselves, which aren't even the cheapest.
Interesting.
If I were a western lab, might I be tempted to subsidize at least one faster & cheaper provider for Chinese models to keep even more data from leaving the US?
(I know in this case there can just be more efficient providers, am asking more than speculating)
>Good luck doing that in China.
Crazy how all the 5 eyes nations banned Huawei within like 18 months of each other.
For certain specific uses. As a consumer I can still buy Huawei phones. As a business I can still buy Huawei routers, if I feel so inclined.
Europe still heavily uses Huawei though, so it’ll be the same with Chinese models
Import control would only affect US? Also good luck sending armed men to China.
Level 1: Anyone using unauthorized model weights in their business is prohibited from doing business with the government...
Level 2: ... or with anyone who does.
Level 3: Anyone caught with unauthorized model weights is subject to the same prosecution they could expect if they had been caught with CSAM.
Cool - how are they going to detect/police that?
The NSA will run Mythos against the entire US population and it will find the patterns. Powerful computer purchases. Energy consumption suddenly spiking. HN comments demonstrating intent and/or contempt for the glorious self-evident freedom preserving laws of the United States.
Then they call your local law enforcement with an anonymous tip about this suspicious dude who's long overdue for a totally random home inspection.
Yeah but I'm in Australia and just host AI models as a service - how are the NSA gonna convince the local law enforcement to do anything?
Ask Kim Schmitz. (Yes, I know, NZ is nowhere near Australia, the point still holds.)
When has enforceability ever been a concern of the government? This is the same government that had anti-sodomy laws on the books until the Supreme Court ruled it unconstitutional in 2003.
Making it illegal even without having a realistic means of enforcing it still chills behavior, and so just because it isn't practically enforceable in the general case doesn't mean it isn't without consequences.
Same way they detect if you do business with Russia, North Korea, Iran. They don’t detect 100% of the cases but the punishment are severe enough to deter most companies.
How exactly though? I have my 4 RTX 6000 pumping out Deepseek code for my website - how are they going to tell i've used Chinese models?
This post exactly ;-)
Whistleblowers. Personal use would still be possible of course.
It's not out of the realm of possibility to do AI audits. I'm not saying it's a good idea or likely, but certainly something they could do.
They can't enforce a ban. Companies can just download the models and run them on MacBooks or GPUs (or the cloud) and USG has no idea. It would also hurt the administration's funding from corporate tech (companies don't like to fund politicians who restrict them). I'm not saying the administration won't do it, it would just be very dumb.
> Today, we are officially releasing Qwen 3.8-Max, the most capable model in the Qwen family to date. This also marks the first time we will open-source the weights of a Qwen-Max-class model — the open weights will be released next week.
I don't understand. That's dated today, but:
https://twitter.com/alibaba_qwen/status/2078759124914098291
> Qwen3.8 is launching and going open-weight soon! [...] You don't have to wait to test it. Just now, the Qwen3.8-Max-Preview made its debut on Alibaba’s Token Plan, Qoder, and QoderWork.
That was on July 19th. I used it to draw this pelican: https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
So what are they releasing today?
July 19th post mentions “Max-Preview” vs. today’s post dropping the “Preview”. Unclear what changed if anything though.. Maybe broader availability or it’s a slightly improved checkpoint
My understanding is that these "preview" models are usually earlier RL checkpoints, and that "official release" happens when they're happy with the training run?
I believe they mentioned around the preview announcement that they'd be releasing improvements to capability, which I assume means continued training.
In Qwen's case, both 3.8 Max releases seem rushed may be in response to Kimi K3 getting all that attention.
3.8 Max Preview ran 90% discount on QwenCloud exactly as when Moonshot couldn't keep up with all the signups.
... other comments were right, this is the full qwen3.8-max model, two weeks ago was the qwen3.8-max-preview release.
Here's a pelican I just got out of the new model. It took 11 minutes and forgot the wheels! https://tools.simonwillison.net/markdown-svg-renderer#url=ht... (scroll to bottom)
The reasoning trace is pretty great:
> More additions: basket with fish in it? Cute detail — a fish poking out of a basket on the handlebars! This adds charm and pelican context.
If the price is $2/$6 that cost me 17 cents: https://www.llm-prices.com/#it=90&ot=29734&ic=2&oc=6
It gave the svg for the wheels in the reasoning trace then forgot to include them in its final answer. Lol.
It has a "definition" for wheel in SVG inside <defs>, but did not use it in the picture.
Wow, bike geometry is really good! Except for the missing wheels lol
Do pelican bikes need wheels? They've got wings, after all... I think Qwen is on to something here.
It's because of the basket with the fish. like ET, fishy fishy sail home.
And what if his name is Marty?
K3 also forgot the wheels when I tried it, but it turned out to be a bug in llama-server's built-in SVG viewer. I know you have your own rasterizer but just FYI, sometimes it isn't the model's fault.
they releasing Max.
Once OpenAI and Anthropic are public, every such announcement will become a reliable sell signal
Agree, I don't necessarily see a strong argument favoring OpenAI or Anthropic here. In the interest of perspective, can anyone (perhaps playing devil's advocate) give one?
The open models are now good enough for what I want to do with them, let alone any future improvements. And factoring in efficiency gains, a model in the ~70b range starting to satisfy my needs would completely obviate the need to pay others for inference. This does not seem far-fetched to me, comparing with where open models were at this time last year. What am I missing?
Seems to me OpenAI and Anthropic are kinda following the Apple business strategy. Those two offer a premium service that gets better results and works more seamlessly. I.e. the integration between Anthropic models and Claude Code is apparently nice and gets better results, and I've heard anecdotally that Codex is currently the best.
So just like in IPhone vs Android, you could end up with a situation where Chinese firms compete and get most of the revenue and usage with low profit margins but OpenAI and Anthropic capture a premium side of the market and still get a lot of profits.
> In the interest of perspective, can anyone (perhaps playing devil's advocate) give one?
I have numerous cases where Sol failed and only Fable could solve a problem. For example yesterday I was merging a Q2 curved with a Bezier curved face in 3D using OpenSCAD. I tried for over 2 hours with Sol 5.6 high and x-high.
Fable two-shotted it in about 30 minutes.
In my experience open models (or GLM, DS and Kimi) are radically worse than either of Claude or ChatGPT at these tasks.
I think there is a huge "long tail" of tasks like this where the frontier labs are ahead, and I think this long tail is quite important.
Hardware and electrical costs including power usage and electric wiring/outlet costs of such machine.
Unless you are spending more than a max subscription (200 a month+) its cheaper to use the cloud.
But things are priced cheaper in the cloud now to lock you in and restrictions around hosted models are getting worse.
If you only have a $300 dollar laptop its probably not worth the upgrade.
I'm personally excited by local AI but the experience for the average isn't the same. I'm willing to get .5/s running on 10-15 years old machines but what I can do with it is limited.
> What am I missing?
Their marketing department :-) . I'm only half-joking; those guys are hard at work finding the best product-market fit for ChatGPT/Claude. "Product market fit" means "strongest revenue", which is not necessarily going to bring the best tool for you or me, but the one that can either get more consumers to shell off money, or more enterprises to cough money for licenses, and in both cases those consumer basis will be narrowed down to what legal and geopolitical circumstances allow OpenAI and Anthropic (and this is why they want to ban the competition!). It also means dark patterns and enshitification, of which I'm already seeing some both in the Codex interface (it was just renamed "ChatGPT"!!!) and in Claude Code (which also is just "Claude" now and can't '@' properly any longer). So in the medium run most people will be better off running an open source harness that can use any model.
coding on a laptop is only one use case
you can't create a new drug by running a model on a laptop. You can't serve a customer support bot running on a laptop. You can't generate video in bulk for many users on a laptop. So there is still a case for paying others for inference.
Does it justify the valuations? No idea, but some major use cases are still there. That's why they are rushing to implement, OpenAI creating a "deployment company", Anthropic having some pharma rumors, etc.
Surely all those use cases can be done right now without a closed model?
I think GP was asking specifically about the use case for closed models.
yes, but at that point even open weight models are not free. Can someone set up the best open-weight model for free and charge pretty much just for GPU usage+operations+margin for inference? yes. But they would have to keep up with how good the private models are. The bet is private models would sell in a similar way at a premium and invest in staying ahead. Is this bet going to work? no idea, don't shoot the messenger.
> yes, but at that point even open weight models are not free. Can someone set up the best open-weight model for free and charge pretty much just for GPU usage+operations+margin for inference? yes.
Or, the company itself can rent the GPU and host the model of their choice on it.
> But they would have to keep up with how good the private models are.
Why? The whole point of the thread is that the marginal gains of using private models is a fraction of the additional cost of using those private models.
IOW, if you really need that improvement over the open models, you can go ahead and pay it. What we are seeing though is that the improvement in general is barely noticeable but the cost is 4x as much.
Can they still go public ? MiniMax M3 Pro is also coming, then DeepSeek-v4-Pro GA, then GLM5.5. There will only be bad news for them in the coming few weeks/months.
Fable 5.1 is coming, then GPT-6...
US AI labs really rub me the wrong way, especially with the doom and scare tactics they use. Both Altman and Dario keep talking about how AI will replace workers and how we should regulate LLMs for national security, Dario’s main point.
LLMs are useful. We can all see that in agentic coding. But replacing everyone’s job? Hardly. And what’s with the scare tactic of trying to get the US government to ban foreign models?
LLMs are useful, and dare I say they’re on par with the internet. Making them cheaper and affordable is good for everyone. The fear mongering from Anthropic and OpenAI looks like an attempt to corner the US market into using only US models so they can keep the profits, especially since China has proven that LLMs are a commodity. US AI labs should work on making LLMs cheaper or better harness. Altman and Dario are not trustworthy.
You are right to feel that way about the frontier labs, especially Anthropic. From https://stratechery.com/2026/anthropics-safety-superpower/
> "Anthropic believes that they are the ones who should have final say over how Anthropic is used; given that they think only they should be developing leading edge AI, they by extension think that only they should have final say over AI generally. When you further combine this realization with the company’s pronouncements about AI’s ability to conduct all economic activity, you realize that Anthropic’s leadership effectively wants to have power over everything and everyone."
To be fair, we're simultaneously mocking anthropic for believing in safety so much and also for them thinking they're the only ones that care enough about it. It's true that no one else seems to care as much. Judging by reactions from everyone, all their safety talk is very bad PR.
The criticism isn't that they believe in safety too much. It's that they patently using safety as a red herring with the actual goal of regulating away competition that they know the cannot beat.
The people involved have been talking about safety long before they had any users or a company.
"Concrete Problems in AI Safety" by Dario and another founder was published 2016, anthropic was founded 2021. Among a bunch of other examples, including other founders.
If safety is seen as a joke now, it was really seen as a joke back then. The reason they keep shooting themselves in the foot with bad PR is because they truly believe in risks.
I’m sorry, but this do-gooder concept doesn’t jibe with their other actions. For example, swindling their business partners (such as Figma). Filing an unnecessary trademark lawsuit against a customer whose brand predates Anthropic.
And for safety? Oh yeah, they literally sell the removal of guardrails in exchange for minimum spend commitments.
For example, swindling their business partners (such as Figma). Filing an unnecessary trademark lawsuit against a customer whose brand predates Anthropic.
Completely unrelated to AI safety.
they literally sell the removal of guardrails in exchange for minimum spend commitments
It seems blindingly obvious to me that you'd want to let trusted organizations use the models defensively, which means without guardrails, and that you'd probably gate that behind some kind of enterprise sales process, for multiple reasons.
Honestly, it just seems like you have an axe to grind and aren't really particularly knowledgeable (or curious) about AI safety.
What they believe now or care about is fundamentally not very relevant: allowing them a monopoly would be a big mistake. Google was supposed to do no evil and it probably started that way but nothing guarantees the future other than having alternatives.
If there are genuine society risks in a tech I don't want to discourage CEOs from talking about them. I feel like we've spent decades talking about how evil chemical companies (etc.) were about covering up issues in the 20th century. But yes, that's different to being a reason to ban external models.
Sam drank the "superintelligence" kool aid early on and said 30-40% of jobs could be impacted by AI, but recently admitted he was wrong
> “My scorecard, at the highest level, would be we’ve been roughly right on technological predictions and pretty wrong on the social and economic implications” https://www.cxtoday.com/ai-automation-in-cx/sam-altman-softe...
I agree re: Dario quietly pushing for government control. He also said LLMs would replace a lot of entry-level information jobs, doubling the unemployment rate from 4-5% to 10%.
Yale did a study recently showing little impact on employment in high-AI exposed jobs https://budgetlab.yale.edu/research/ai-probably-not-yet-reas...
I imagine it will be a long tail. Most companies won’t fire people for AI but probably won’t immediately replace a person that leaves, if at all.
Or won't hire that extra person when they need to increase output.
but this crap may take forever to play out even if the outcome is well-known. Self-driving is "here", it's obvious that once it's cheap enough having a human behind a car wheel or a freight truck wheel is an absurd waste of human life (kinda like digging canals with bare hands instead of an excavator), yet truckers and uber drivers are still employed. But everyone knows the writing is on the wall for them.
the writing has been on the wall for fifteen years, and yet the number of professional drivers has only increased since then...
Is it? I believe it is fair to hope so and even expect it, but this has been promised as being just around the corner for 10+ years.
Yet here we are, still driving our own cars, and laughing at silly behavior of most advanced self-driving systems.
I am certain we'll get to something of similar value, but I expect we'll have to improve infrastructure too (smart roads).
It's not so simple, if such a headline can get them closer to the regulatory capture they want to lock in American businesses and forbid them from using Chinese AI.
But the US are the country of freedom!
That's what Hollywood has been telling me my entire life!
Only the ones that beat expectations
It'll sell after 1-2 times, then the market will get jaded and ignore.
3.8 27b is the real news here
Can't wait, been running the 3.6 locally and it's been a boon.
real
AWS Bedrock still lacks the support for the latest open weights models like GLM 5.2, DSV4 Flash 0731, Kimi K3 etc. Even they only support Qwen 3 which is a very old model. Any specific reasons they are reluctant to support open weights models?
AWS Bedrock do support many, older, open weight models. Perhaps there is a reluctance to support anything more powerful that would compete with their partner Anthropic?
In practice I'm not sure how big a deal this is - Bedrock is not the cheapest or best provider (try Fireworks AI or DeepInfra who do support more recent models), and would seem to be of more interest to corporate users who already have an AWS account.
Perhaps also of relevance the US government is hassling US companies that are using Chinese models (currently DoorDash, previously AirBnB & Cursor), which may limit demand for corporate use.
I was wondering the same thing. Maybe it’s a licensing issue. Or the models are too big and there isn’t enough demand. But size is not a solid argument because they also don’t support Qwen3.6 27B and 35B-A3B, especially since they just added Gemma 31B and 26B-A4B.
If I recall most of the open weight models have provisions about large scale commercial hosting. So probably licensing issues.
> How Qwen Cloud handles your data during inference? > Qwen Cloud does not use your API inputs or outputs to train or improve models. > Learn more about how your data is handled during the inference process from Zero data retention.
If this is true, this is a big deal for me, but unfortunately I cannot find anything in their legal agreements, so this marketing sentence is worthless.
2.4 Trillion parameters with open weights releasing next week? The open-source community is going to need a collective GoFundMe just to buy enough VRAM to host this monster.
Kimi K3 is about the same
I like that Alibaba are emphasizing more challenging real-world "coding" tasks as well as just benchmarks.
The "Reproduce a research paper — then improve it" seems particularly impressive as well as actually useful - perhaps as close as we're going to get to "recursive self-improvement" given that these are data-driven not code-driven models.
I'm not sure how much real world use of AI is 100% hands off agentic coding, but I guess for objective evaluation purposes it needs to be something without a human in the loop.
It'd be interesting to see a comparison of each company's best models on a suite of real-world challenging tasks like this, but I guess difficult since each group of tasks would need to be one-time-use only to prevent subsequent benchmaxxing which makes comparisons useless.
Qwen3.6 35B-A3B Is my current backup model after Ornith—1.0, which has performed slightly better and faster at Python, db and some backend tasks. About 75 tokens/sec on a 5060ti 16gb depending on context size (usually 64k)
At $2/$6, it can be much more "approachable" than K3. Wondering what have they done differently to be able to afford this price (that K3 couldn't, apparently - most other providers offer similar prices to K3 itself).
Moonshot is printing money on k3. It likely costs the same to serve as qwen3.8. The license requires all major inference providers to sign an extra (secret) licensing agreement with moonshot that almost certainly requires them to agree to this price and pay royalties to moonshot. Watch as the k3 price plummets over the next 1-2 weeks.
If k3 is open-weight (and it's available on hugging face [1]), how could they force providers to sing an extra (secret) licensing agreement?
Edit to answer my own question:
License file [2] states:
> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
[1] https://huggingface.co/moonshotai/Kimi-K3
[2] https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
Can weights be copyrightable?
Licenses are not copyright law, they are contract law. Copyright is automatic and does not require any terms, as long as the work is covered by copyright. Contract law can then override the innate copyright rights (for example to permit people to use your copyrighted works).
If you create something that isn't copyright-able, you can still create a contract which people must agree to in order to use your work. The enforcement/validity of that contract then depends on making sure the user knew about the contract and agreed to it. The way they did that here (just publishing a LICENSE file) could be interpreted as a judge as no confirmation of agreement of contract - depending on if they find a "reasonable" user would have known to look for the file.
Copyright is an automatic protection (meaning you don’t need to register it, like registered trademarks), for certain types of creative works. Contracts are one way to dish out these rights (eg. for a fee).
If you have a work that is not copyrightable, and you give it to people lawfully, then you cannot protect it via copyright.
How would that ever be enforced, I wonder.
Qwen is Alibaba. Alibaba rents hardware to Moonshot. So Qwen always has the option of cheaper hardware. Qwen might also break even on inference as competitive advantage since Alibaba has wider pockets. Alibaba also owns a 36% stake in Moonshot, which must make pricing discussions interesting... but then again Google rents hardware to OpenAI and Anthropic.
I used the qwen 3.6 models and convinced myself to go ahead and get a gpu. I got the 7900 xt with 20 gb vram.
How’s that been going?
Bad move. Should have used the money to buy NVIDIA stock
I have a 5080 Super RTX but its still not enough to run these big models. I use Gemma 4 right now on a Debian Linux with no GUI that I access remotely. Its works pretty well, but still not as good as any of the big models like Opus 5... Can't wait to be able to buy a personal home server that can run much bigger models. I heard some companies have started building AI PCs only used to have a personal AI model at home, have any of you tried one yet?
Just arrived on open router. Was waiting for this.
https://openrouter.ai/qwen/qwen3.8-max
I noticed they are careful not show Apple products, what operating system are they using in the demo videos?
> This also marks the first time we will open-source the weights of a Qwen-Max-class model — the open weights will be released next week.
Nice!
Waiting for Qwen3.8-27B :)
Their base models and architecture has quickly become the go-to for local inference and fine-tuning, even when they introduced some tricky things like GDN, so many people use it, that it was matter of days/weeks until lots of OSS frameworks adopted it.
I used Qwen3.8 Max Preview for 2 weeks and can't tell if I truly miss Fable.
Qwen doesn't overuse em-dashes, weird sentence structures with bold text: normal text.
In my private GitHub Repo Qwen finished as much tasks as Fable did. Without hitting 5-hour session limits. Qwen's token-per-second performance fluctuated greatly. From 20 tps up to 80 tps.
Qwen failed on some "reading between the lines", but so did Fable.
The only noticeable difference is skill loading. Until "Must use" in a skills front matter, Qwen tends to be very lightly loading skills. Superpowers works, because of the same Must use wording. All my local skills use the same wording now. For installed Skills, it's more or less my responsibility to remind Qwen using a certain skill.
I used Qwen together with Qwen Code. I didn't compare Qwen+Claude Code.
With my Owl code scanner, Claude and Qwen produced similar results. Qwen repeatedly used the same output format, despite no format provided and disabling memory. Fable varied greatly. Both followed the file format output and produced valid files.
With Qwen 3.8 being now a visual understanding model, I don't really miss Fable/Opus/Sonnet for my private projects.
At work we are still exclusively using Anthropic models with Claude Code. No change incoming on that front.
Privately, I'm sold on Qwen and Qwen Token plan. No session limits, many open weights models available via a single API.
Since it's horrifying thinking about what US companies do with my data, it's no worse or better thinking about what Chinese companies do. It's a choice between the Plague and Cholera.
It seems this is the only mention of cost?
> Qwen3.8-Max comes with the official support for reasoning_effort, which can be used to adjust reasoning depth and control cost:
> xhigh (default): for complex tasks demanding thorough analysis
> medium: balancing accuracy and speed
> low: efficient reasoning optimizing for speed and cost
I hope this is significantly cheaper. I've been loving Deepseek for it's nearly free usage costs, hard to justify switching from cents per day.
They mention it explicitly in the Twitter post [0].
> Pricing: Input: $2.0 / M tokens Output: $6.0 / M tokens Implicit Caching: $0.25 / M tokens
[0]: https://x.com/Alibaba_Qwen/status/2084100707423289643
The linked qwencloud page has pricing; it's $2/6.
https://www.qwencloud.com/models/qwen3.8-max
Thanks, totally missed that!
I've been waiting for a tiny model release! Kimi, Deepseek, and GLM were all monsters, the 27B 3.8 is filling a gap thats been open since Qwen 3.5
Also I love how literal Qwen 3.5 was, hopefully 3.8 is still extremely literal. The token explosions were actually helpful in debugging prompts.
Lmao I love their video with the idea that people will be able to do their hobbies while ai does their job.
Surely Alibaba is leading by example here by reducing work hours per week while keeping pay the same right? Right?
That's the thing. Wny are companies like OpenAI/Anthropic/Alibaba/Kimi/Deepseek still hiring SWEs if their models have become so good?
https://en.wikipedia.org/wiki/Jevons_paradox
Otherwise known as the "no shit Sherlock" principle
OpenAI can't even build Codex app (not cli) for Linux... I thought code was solved?!
Ironically it's Electron BS, so actually, Sol could probably build itself...
The models are good even by skeptics standard, it's just that evangelists are overselling the capabilities. If you understand the limits of LLMs not using them as a business is shooting yourself in the foot.
However, they are not at the point where they can effectively train themselves, nor did they are capable of researching their own method of learning. SWEs in mid-corps on my country are right now relegated to reviews and sanity check, basically babysitting the LLMs and making sure they're not spouting nonsense. If you think about it, that's basically QA and can also be delegated to another AI. If Bun's rust rewrite that they tout as fully LLM-led can pass the test of time in a year or so I think that's it.
I believe all that is now constrained by compute and capital, not tech.
The world never runs out of problem. There is so much work to do.
I mean its pretty obvious right? This models are not flawless and sometimes reach stupid conclusions so there needs to be some one who watches it. Thought i must say u are right. Every one of them pretends that this new model is gonna finally take ur jobs lol
There's infinite work to be done, so higher productivity makes people worth more. (Obviously this doesn't apply if AI can do everything but we're not there yet.)
The percentage of the population that needs to work will continue to go down because of aging and automation although you might not see it as a reduction of hours for a given individual employed person.
Already 38% of Americans aged 16 or older do not work and are not looking for work and yet are not dying from hunger or exposure. This would have seemed like a utopian dream to someone from the 1800s.
Source?
Being 16, 17 and 18 and not working is expected, given they are still at school. A good percentage of people aged 18 to 24 are studying full time.
This stat seems a bit misleading and doesn't support the "you can live and not work argument". How many of those people are retired? How many of those are under 18? How many of those simply cannot work? How many of those live at home or are on food stamp?
>is expected
... these days. 100 years ago you'd have a job at 12 in many cases, 14-16 if you were more lucky.
I'm not even talking about peasants from previous centuries.
But how is that possible? How can one survive without income barring homelessness, begging, prostitution, and other not so utopian options?
In the 1800s most families only had a single provider, so this percentage may have been higher back then than it is now.
the single providers were the children in the factories?
Maybe you should ask AI to review your statistics and reasoning. A large amount of 16-22 year old kids are in school. And retirees 80+ are retired. I don't see how any of this is relevant to the idea of what would happen to the economy if there is an employment collapse. I don't know that this would also seem like a utopian dream to someone in the 1800s; I'm not sure what that means. I think someone from the 1800s would be surprised by a great many things and that statistic would probably be low on their list.
> I love their video with the idea that people will be able to do their hobbies while ai does their job...
Are you not already experiencing this? I think this is fairly common for people using AI now, though the time may not always go into hobbies or sports. It's common for me to setup Claude with an hour+ task while I catch up on housework, or while I'm getting ready in the morning.
In the last couple of weeks I've unfortunately had multiple family illnesses - it has been helpful to have Claude keep up with much of my product development programming work while I visit my mother in hospital and check on my father's recovery. I'm able to give more time to family without worrying that business progress isn't keeping up. The overnight Claude sessions while I'm asleep have been particularly helpful.
No I haven't had time to spend my afternoon rock climbing while ai generates documentation.
It's infinite work, I just did more work while codex was doing it's thing in the background.
We are in the golden period where this IS possible. Once it becomes the norm to "do something else while your agent works", we'll be asked to do more WORK while the agent works, rather than do hobbies/housework/nap/etc.
Before AI:
I have 10 tasks that take 1 hour each to do.
After AI:
I have 100 tasks that take 10 minutes each to do.
Same amount of time spent working, maybe even more stressful, just more productive.
So you were working 10 x 1 hour = 10 hours before.
And then you were doing 100 x 10 minutes = 100 / 6 hours = ~16 hours & 40 minutes.
My brain didn't work there.
In a 30M+ LOC repo we blew out Qwen3.6’s context thinking about a code review. This was via direct call to model. Do harnesses facilitate better context management, or is there something else to accommodate its smaller context window?
What was it reviewing? Was it just a "take a look at this pile of code over here"? Because I think asking any model to review 30M lines of code is a stretch...
in general with automated code reviews I've found it fruitful to tell the model what specifically to look for and where. I usually don't get over 10% of the claude opus context window for code reviews, but it of course depends on how your code is structured, how much the agent has to explore etc.
The real answer is to stop trying to have AI reviewing AI code because it's just the blind leading the blind.
Qwen3.8-35B-A3B would be really appreciated (-/\-)
The dream. Or a qwen-3.8-coder model.
Can a model be stripped off anything not relevant to coding and get a lot lighter? Or is that impossible?
Just like we have professors with specialisation wondering if AI models can also be so.
You can... but the trick is to do so without killing performance. Turns out a lot of random things help make coding performance good.
Does this desideratum make any sense? The whole point is that you write to it in English or Urdu, and it writes the specified code. If you cut off everything but the code writing, you cut off everything.
If you're literally using it as code autocomplete that's not functionally true. But it's still neither well specified nor a good idea.
Domain modelling as well: we have clear programming concepts, but the kind of autocomplete that can autocomplete a customer sales process needs to be able to refer/leverage ontological connections on words like ‘bass’, ‘season’, ‘rebate’, and ‘jamboree’.
Intuitively it feels like focused models should be better models, but human programmers are ‘better’ knowing programming alongside general stuff. We’re not reduced by reading economics or Tolkien, removing such knowledge would be premature optimization.
Thanks..non AI expert here just wanted to understand..cos one of the most talked about topics is running models locally on cheaper hardware. Helps many esp in poorer places.
Would be nice if a genuinely useful coding model can fit into an old laptop, give decent performance and know nothing but (say) Go or c language.
But I see yours + others points.
no, apparently, otherwise we'd already have specialized models. every bit of meaningful human-generated data appears to improve the overall capability of the model.
we do though and it can be stripped
Kind of yes, but this be get you are pretty stupid autocomplete yuo can probably avhieve without an LLM at all.
>we have professors with specialisation
Yes, but any professor with a specialisation is an erudite with vast knowledge across the board.
I'd argue we can have a meaningful BIG model with a specialisation but not vice versa.
PS: all of this is about a model that is expected to be able to execute development tasks on a human level. Obviously we can have small models that are very capable in things like TTS or STT.
I guess it can but it will be useless. After all the model superpower is awareness and ability to guess and infer some stuff. Right now a model saves you time not only by coding faster, but that it can figure out some stuff about the shape of the data and its purpose.
If you throw general purpose model at a codebase - it will look at the table and data logical connections beyond what is explicitly declared. It will figure out on its own that Salaries should be displayed on SalariesTable.php and it will "know" that your prices should include vat and so on.
A human knows that VAT and price go together and are related, full size LLM does too, stripped one - doesn't.
Yes, look up REAP.
Linking here since the pruning method you're talking about wasn't the top search result when I looked: https://github.com/CerebrasResearch/reap
unfortunately the alibaba cloud does not seem like it can keep up with the demand caused by this announcement, i'm just getting endless timeouts
So the news here is that this model left the preview stage, and they are also releasing an open-weight version of the Max series for the first time.
You can always try out this model for free on Qwen Chat. Alibaba Cloud has too much friction for me.
I was super excited to read this but lost the plot when I go to
> Qwen3.8-Max was asked to create the oh-my-cli project from scratch and, over a 10+ day long-horizon autonomous coding run
10+ days of building what exactly? Is that a shell prompt customization toolkit? Or a coding harness??
Neither - well, sort of the second. The poorly named thing is a self evolving coding harness, the self evolving part makes it a big deal.
I wish they had picked a different example.
> The open weights will be released next week.
Has anyone know whatever they will release any video / image generation capabilities? Video input obviously suppose to be present.
I'm so glad exciting releases like this are still going to have components with open weights like Qwen 3.8 27B dense
KimiK3-Thinking is not a matching model for comparsion?
Is "cowork" a general industry term now? Here I was just getting used to "coding" replacing "programming".
Cowork is to work as coding is to... ding?
Has anyone tried Qwen with the Fusion 360 MCP server? I feel like drawing with python is close enough but I'm curious
We will eventually need a self evolution benchmark to see where these large models can create recursive solutions that improve
the benchmark I trust most is whether the model can explain its own pricing page without getting confused
Not even humans can do that, you're literally asking for something beyond AGI
Bistromathics
https://www.hhgproject.org/entries/bistromathics.html
I'm trying and failing to find value running a potential Qwen 3.8 27b dense model on a 16 core, 128 GB of ram, 2080ti box. Yes, the GPU yells for help, but the problem is that no math works to upgrade this machine even when pouring $200 in rent every month into the large model providers...
How are you all justifying economical use of these local models right now? What's the cost efficient way to do this and do better (even with models evolving over time and losing now vs later) than the big labs?
You will simply not get more value out of running a local model vs paying for a subscription/API from the cloud in 2026. There is no math that will make local models come out ahead in $/intelligence/token.*
The point of local models is privacy, offline use, and maybe no guard rails.
* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.
Points are starting to be made in favor of value, to the contrary of what you are affirming. Specifically because the new open weights models lower the TCO of hardware in an environment where new open weights were previously thought to be a thing of the past.
> The point of local models is privacy, offline use, and maybe no guard rails.
But also, control and consistency. A local model cannot be changed out under your feet like an API model can be.
There is for $/creativity/token. LLM sampling settings are poorly supported even in open source serverless providers but are the single best lever you have for getting better outputs in regards to creativity (and quality for long context or highly quantized models).
I'm pretty sure you can adjust the creativity for many Chinese model inference providers.
Most of them don't expose more than top_p/top_k/temperature. Those are woefully inadequate compared to what open source inference engines support.
There's no point, except if you want privacy and independence. I am playing with a personal assistant that checks my emails, calendars, sends me an agenda and maintains my TODO list. I am not sending such data outside.
Running locally for me is mainly about learning, maintaining control+privacy, and helping shift my coding+design process to leverage LLMs. I guess if you made me boil it down to a single word to justify the cost I would just say: tuition.
Sounds like we have similar boxes - mine has a 10 core CPU, 64 GB of ram, and a 2070 Super. My motherboard had two unused PCIe3x8 slots and doesn't support Blackwell GPUs. I bought a couple of brand new Ada generation RTX 2000s with 16GB of memory for under $1400 to get to 40 GB of VRAM. That will easily run Qwen3.6-27b at a 6-bit quantization and 80,000 token context size. It isn't fast (19-21 t/s), but using pi-coding-agent is fine.
Now, my instinct is that I am giving up SOTA performance on agentic coding with this setup and LLM. But the gap between my setup and SOTA commercial models is small enough that it doesn't matter to me.
I use opencode go and pay 10$ a month. Works good for me.
The only scenario is if you have enough work to do batch inference. Using a tiny fraction of GPU capacity to decode a single request at a time just doesn't make sense, as you say.
If you can do inference on the CPU, drop the GPU : it should be faster.
> $200 in rent every month into the large model providers
We all know that is hugely subsidized, and I guarantee that OpenAI and Anthropic are looking to enshittify that ASAP.
The enterprise users, however, are not subsidized like that. They pay per token. And some developers in those companies are chewing down a lot of tokens. Self-hosting an open weight model could be a massive savings very quickly. It also gives them negotiation leverage when talking to OpenAI and Anthropic.
We all ASSUME that.
For all we know, inference might be dirt cheap, they might just be hiking the API prices so high for us to think subscriptions are subsidized.
Well, it is always possible that OpenAI and Anthropic have some secret sauce that they are hiding that nobody knows about. However, the pricing of tokens on places like HuggingFace and OpenRouter suggests that probably isn't true. In addition, the lack of reliability and general slowness of Opus and Fable at various times also suggests that.
Now, the one wildcard in all of this could be Google. They are on the eighth generation of their TPU and have been holding their cards extremely close to the vest. I don't think anybody has a good read on exactly how much capacity they have. Most things you can kind of figure out the overall business numbers and what's going on in Google--the TPU area is one of the exceptions. I know a couple of big customers and even they don't have any visibility on that front.
What is the best LLM that I can use right now that is optimized for everything other than coding?
The last thing I want my personal agent to do is to write up code and run arbitrary commands. That is practically a legalized RCE.
that is not an model issue but a harness one, if you don't want it to have io to files or run command then simple don't give it access to the tools.
Yes, I don't. I have my own harness where the agent is only able to do a set of predefined things and none of them involves the internet. However, I imagine the model could be a lot smaller if it does not have the baggage of coding and programming in general.
I mostly just need it to mostly be a very advanced NLP model that is able to figure the set of tools that it needs to call with what argument. Then it will just follow a predefined decision tree.
Check out cactus needle, might be what you're looking for.
is it the right time to perhaps switch to QwenCode ?
i might end up cancelling claude, anybody else thinking of the same ?
Already done it. But switched to Codex instead. See you on the other side buddy.
I'm oh so seriously tempted to switch at least for a month or two.
Have you retrained Qwen not to give Hoover Institution answers to geopolitical questions?
It is clearly distilled with an aggressive pro West bias to increase sales in the West.
Does the page actually load for anyone? I get stupid spa skeleton spinners.
they release new qwen model every two weeks?
It's been 2.5 months since Qwen 3.7-Max. Unless you mean since the preview of this 3.8-Max model.
AI is a commodity. This is proven now. And valuations will have to drop 90%.
There's a lot of AI models that each 'win' one week over the next. The pure definition of a commodity market and not a 'winner takes all market' as valuations would imply
Tokenpocalypse canceled
Now, if only we could afford a setup decent enough to run 2/3 instances at the same time...
Does anyone know how token- and reasoning efficient it is? The charts don't show how many tokens were used in any benchmark.
The imminent third-party benchmarks will cover that.
> In this case, Qwen3.8-Max was asked to create the oh-my-cli project from scratch and, over a 10+ day long-horizon autonomous coding run, build a self-evolving harness.
They don't explain how successful that went but it's a bit hilarious seen that an Anthropic dev explained that it's been 15 days Claude was hard at work --with nothing to show yet-- trying to rewrite itself in another language.
"You rewrite Claude Code, we rewrite oh-my-pi."
"You're nowhere after 15 days, we do it in 10."
Sure, it's apples to oranges and all that. But part of me thinks they know fully well what they did there.
Where did you read that Anthropic are experimenting with rewriting Claude Code?
ah so they distilled fable and sol, eh?
“self-evolves through feedback loops”
Does this mean they distilled Claude? Sounds like what Claude Code will often do.
Of course they did.
I mean sure but I’m curious to what extent and why it’s so obvious to some. I haven’t seen this sort of language in a marketing post before.
It's meaningless. Models have always been able to do this and this capability is strengthened during RL since being able to explore the solution space to figure something out will give it a reward.
What is important is how long it can go without requiring human intervention. Not just that it's possible to run on its own for a time.
Are these latest Qwen models still open weights or has Qwen moved away from that?
The second sentence of the page: "This also marks the first time we will open-source the weights of a Qwen-Max-class model — the open weights will be released next week."
Page won’t load for me it’s just grey bars fading back and forth forever.
You can always wait until the page loads before posting your thoughts on the Internet
Two days later page still doesn’t load.
Or let them just ask and if someone asking a question on the internet bothers you, just move along and ignore it.
Reddit has become a fairly worthless site because people just ask questions instead of taking anytime at all to figure out the answer. He just needed to wait for the page to load instead of adding noise to the discussion