Really appreciate the depth of this paper; it's a welcome change from the usual model announcement blog posts. The Zhipu/Tsinghua team laid out not just the 'what' but the 'how,' which is where the most interesting details are for anyone trying to build with or on top of these models.
The post-training methodology (Sec 3) is what really stands out to me. The idea of creating specialized 'expert models' for reasoning, agents, and chat, and then distilling their capabilities into a final unified model is a fascinating approach. It feels like a more structured way to solve the "jack of all trades, master of none" problem that can plague generalist models. Instead of just mixing all the data, they're essentially having a generalist learn from a committee of specialists.
A couple of the findings from their RL experiments are pure gold for anyone working in this space. The counter-intuitive result that a single-stage RL process at the full 64K context length outperforms a progressive, multi-stage approach (Fig 6) is a fantastic lesson. I've seen teams assume the opposite would be true. Also, the pragmatic choice to use an XML-like template for function calls to avoid JSON escaping hell (Fig 4) may be a small but brilliant engineering decision that makes a huge difference in practice. Wrangling escaped code inside JSON turns out to be a mess.
The performance on SWE-bench is impressive, putting it in the same league as much larger or proprietary models. What I’d love to see, and maybe others here have thoughts, is whether this hybrid training recipe holds up outside ARC-style evals. For example, do the agentic improvements transfer to messier, real-world workflows where APIs are undocumented, partial failures are common, and user input is full of ambiguity?
> Are all these post/mid-training tweaks important with abundant, verified, synthetic domain data?
No. Many are aimed at cleaning/aligning noisy, mixed-domain data. With abundant, high-quality domain data, you can skip most of the complexity and focus on direct SFT/RL on your corpus.
> Can a small team stick to scaling 2024-era best practices?
2024 was the year of SFT. I believe fitting reasoning traces to your final responses via RL is the technique-du-jour of 2025. Jumping from SFT to RL training might be biggest gain here if RL can be applied to your problem (e.g. math, coding etc).
We prefer the raw, flawed original version than the AI-polished version on HN. It makes HN feel more real and authentic. If other community members notice that your writing seems AI-enhanced, you've taken it too far.
I do admit your post had some apparent substance to it (at least from my naive perspective), but that substance is really diminished when it's passed through an LLM. I think something is lost when someone's unique writing is 'planed' by AI. I'd really encourage you to avoid this kind of editing and just use your own words.
The comment you're replying to is 100% AI-generated. How does obviously LLM-generated content continually make it to the front of HN, and why in God's name are you being downvoted for calling this out??
"...a fascinating approach..." (LLMs think everything is fascinating)
"...they're essentially having a generalist learn from a committee of specialists..." (analogies, analogies)
"...where APIs are undocumented, partial failures are common, and user input is full of ambiguity..." (typical AI rule of three template with semantically similar parameters that contribute nothing to the overall meaning)
It does worry me how defensive people can become over really obvious slop - I don't think I'm even particularly attuned to the style of LLM writing but it is incredibly obvious every time I see it. It's only going to get worse I think.
> and why in God's name are you being downvoted for calling this out??
Tinfoil hat time, but perhaps the bots don't like being called out? I don't actually take that statement seriously, but it seems an eventual avenue. They've long been seeding threads on Reddit to shape initial hive mind, i imagine that's going to get more advanced and widespread.
I've been playing around with GLM-4.5 as a coding model for a while now and it's really, really good. In the coding agent I've been working on, Octofriend [1], I've sometimes had it on and confused it for Claude 4. Subjectively, my experience has been:
1. Claude is somewhat better at whole-codebase tasks, where you need to reason over a bunch of context and consider system interactions.
2. GLM-4.5 is somewhat better at being "honest" — i.e. I rarely see it doing the things Claude does like making broken tests pass by changing the test instead of fixing the bug.
Both are quite good though, and GLM-4.5 has found bugs that both Claude 4 Sonnet and 4.1 Opus have failed to catch. In general I think Claude wins a little more frequently on debugging tasks than GLM-4.5, but it's close.
Compared to GPT-5, both Claude and GLM feel like they're more consistent, although GPT-5 sometimes has long brilliant runs where it nails everything with subjectively higher code quality than either of the latter. However, once GPT-5 goes off the rails, it's hard to get it back on track, so it can be a bit frustrating to work with in comparison.
I just read your comment and decided to give GLM-4.5 a try in Kilocode. I'd been using Gemini CLI all day to try to resolve a tricky bug in some compiler code (a compiler for a subset of C that generates microcode for... a weird architecture, I'll leave it at that). So GLM-4.5 zoomed in on the problem right away. A problem that's eluded Gemini CLI all day. Gemini was leading me on a wild goose chase implicating a function that turns out wasn't the problem (and trying to make all kinds of lame changes to the function saying that would fix the problem - and it never did because the problem wasn't that function).
Sometimes getting a second pair of eyes to look at the problem helps and is usually not a judgement of smartness of the first pair of eyes. Seems like it also applies to coding agents.
Indeed, I've also found that various models are good at various tasks, but I have yet been able to categorize "Model X is good at Y-class of bugs", so I end up using N models for a first pass "Find the root-cause of this issue", then once it's found, pass it along to same N models for them to attempt to solve it.
So far, which model can find/solve what is really scattered all over the place.
You are experiencing the jagged skills frontier. All models have these weird skill gaps and prompt phrasing sensitivity. This is the main problem solved by an llm-consortium. It's expensive running multiple models in parallel for the same prompt, but the time saved is worth it for gnarly problems. It fills in the gaps between models to tame the jagged frontier.
My very first use of the llm-consortium saw me feeding in it's own source code to look for bugs. It surfaced a serious bug which only one out of the three models had spotted. Lots of problems are NP-ish so parallel sampling works really well. Googles IMO gold and openais IOI gold both used parallel reasoning of some sort.
This is so true. Another thing, a model might be better at something in general, but worse if the context is too long. Looking at how GLM-4.5 is trained, on lots of short context, this may be the case for it.
GPT-5: Exceptional at abstract reasoning, planning and following the intention behind instructions. Concise and intentional. Not great at manipulating text or generating python code.
Gemini 2.5 Pro: Exceptional at manipulating text and python, not great at abstract reasoning. Verbose. Doesn't follow instructions well.
Another thing I've learned is that models work better when they work on code that they themselves generated. It's "in distribution" and more comprehensible to them.
The good old regression to the mean. Testing models as the second pair of eyes only when the first fails is going to give weird results... https://www.smbc-comics.com/comic/protocol
If you choose the worst performers, a part of their failure will be due to luck and part due to skill. Letting them retry will average the luck in that sample. (So likely increase) Of course you as the tester could be very unlucky and pull a sample of people with no skill who did get a bit lucky in the first place, but it's both unlikely and... it's a funny comic.
I don't seem to have to worry about burning tokens with Gemini CLI. It's all been free so far. That'll probably change sometime in the future, I'd guess.
I've had similarly good experiences with GLM-4.5 for smaller projects/requests. Unfortunately that did degrade with larger contexts, so I'm still treating it as a good fallback for Sonnet 4, rather than a full-blown replacement.
What’s your monthly bill (OpenRouter?) if I may ask? I have Claude Max and always on the lookout for alternatives, at least for the easier to solve problems.
I run a privacy-focused inference company, Synthetic [1], and I use our API of course :P I actually like GLM-4.5 enough that it's currently our default recommended model for new users. But yes, otherwise I'd use the official zai API most likely, or Fireworks. GLM-4.5-Air is quite good for a local model but GLM-4.5 is better; up to you if the tradeoff is worth it — there's definitely value in the data not ever leaving your machine, but it's not going to be as strong of a model.
What makes your service especially privacy friendly?
I think if you are striving for full privacy, you should implement the secure enclave idea presented by ollama, it makes the entire pipeline fully encrypted, I'm waiting for an actual provider to finally implement this.
We don't store prompts or completions for the API (our privacy policy says "for longer than 14 days," as mentioned elsewhere in this thread — we don't actually store them at all, but the 14 day legal guarantee is to make sure that if someone accidentally commits a log statement, we have a little bit of time to catch it and revert without being in breach of policy). And, we don't train on your data, even for messages in the UI: we only store UI messages in order to let you view your message history, not for training.
Compared to using — for example — DeepSeek from deepseek.com, I think we're much more private. Even compared to using OpenAI and opting-out of your data being used for training, we're still more private, since OpenAI makes no guarantees for individuals that they don't store the data — notably, any data ever sent to them is apparently now being shared with New York courts (and the New York Times!) due to their ongoing legal battle with the New York Times [1]. And compared to using OpenRouter with "data_collection: deny", we uh, actually work :P Surprisingly sad how many broken model implementations there are if you're just round-robin-ing between inference companies... Especially reasoning models, and especially with tool-calling.
(And if something's broken, you can email us and we'll generally fix it; OpenRouter doesn't actually host any models themselves, so there's not much they can do if one isn't working well other than just de-list.)
> our privacy policy says "for longer than 14 days," as mentioned elsewhere in this thread — we don't actually store them at all, but the 14 day legal guarantee is to make sure that if someone accidentally commits a log statement, we have a little bit of time to catch it and revert without being in breach of policy
I'd recommend to rephrase your marketing, because not storing prompts is a huge selling point compared to deleting after 14 days (a lot of things can happen in 14 days)
I don't think that's an accurate read of our privacy policy. What you've left out is:
These third-party service providers are prohibited from using personal information for any other purpose and are contractually required to comply with all applicable laws and requirements, which may include Payment Card Industry Data Security Standards if they are processing payments.
We use third parties like Stripe and Clerk, and by nature of using those services, your information is disclosed to them. This is an extremely common clause in privacy policies, and one we need to have unless we roll everything ourselves. We're much more private than using, say, OpenAI, DeepSeek, Anthropic, or most popular LLM services.
Oh sorry — our lawyers are American, I'll shoot them an email and see if we can get that fixed. Is there something you were looking for but couldn't find?
We have servers in the EU and US, but right now there's no way to route to only EU (or US) DCs.
- No nominated representative to act as local contact for data subjects and supervisory authorities.
- A few standard clauses relating to what and where data is transferred / stored etc.
Nothing too strenuous, but I thought since you've gone to the effort of having EU servers I guess you're expecting EU customers and so hence probably worth tweaking your privacy policy.
I look forward to experimenting with GLM-4.5 through your service !
I’m curious for your service, if it’s centered around privacy, why is the data stored for 14 days at all? My understanding with fireworks is that it’s 0 logging — nothing to store. To me that’s private.
Great question! We actually don't store prompts or completions for the API at all — but legally we say 14 days so that if someone accidentally commits a log statement, we're not in breach as long as we catch it quickly and revert.
We accept USDC and USDP crypto payments via Stripe. We don't currently support Monero or Zcash — right now all our payments are via Stripe since it simplifies security + compliance for us. It would be a pretty neat feature to build though.
Not OP. Chutes.ai charges $0.20 per 1M tokens. I don’t think it uses caching though because I ended up burning $30 in an hour or two. I had to move back to Claude Code.
So GLM-4.5 series omits the embedding layer and the output layer when counting both the total parameters and the active parameters:
> When counting parameters, for GLM-4.5 and GLM-4.5-Air, we include the parameters of MTP layers but not word embeddings and the output layer.
This matches with the calculation I did for GLM-4.5 (355B A32B):
In [14]: 356732107008 - (775946240 * 2) # token_embd / output are 775946240 each. assume omitted
Out[14]: 355180214528
In [15]: 356732107008 - 339738624000 - (775946240 * 2) # parameters that are always active
Out[15]: 15441590528
In [16]: 339738624000 * 8 / 160 # parameters from activated experts
Out[16]: 16986931200.0
Meanwhile, GPT OSS series includes both the embedding layer and the output layer when counting the total parameters, but only includes the output layer when counting the active parameters:
> We refer to the models as “120b” and “20b” for simplicity, though they technically have 116.8B and 20.9B parameters, respectively. Unembedding parameters are counted towards active, but not embeddings.
And Qwen3 series includes both the embedding layer and the output layer when counting both the total parameters and the active parameters.
Why there is no standard in counting? Which approach is more accurate?
I'd say it depends. For the total parameter count, you should just count all parameters, since that's what matters for memory requirements.
For activated parameters: All unembedding parameters are used in every inference step during token generation, but only one column of the embeddings is used (if done right). So count accordingly, since that's what matters for memory bandwidth and therefore latency.
Seems like we may get local, open, workstation-grade models that are useful for coding in a few years. By workstation-grade I mean a computer around 2000 USD, and by useful for coding I mean around Sonnet 4 level.
Current cloud based models are fun and useful, but a tool that is / will be so core to the developer experience, I want to be able to run locally.
This will be essential for the open source. Otherwise open source development will become unsustainable. I'm actually a little bit more optimistic. I think we will get something more than Sonnet 4 level in two years, that can be run on a $2,000 machine.
This feels like the first open model that doesn’t require significant caveats when comparing to frontier proprietary models. The parameter efficiency alone suggests some genuine innovations in training methodology. I am keen to see some independent verification of the results and to see how if does on Aider’s LLM Leaderboard.
The sheer number of things "they observed" in this paper that could be whole papers in themselves is astounding! Lots of great stuff in here around training processes and data collection+synthesis.
Does anyone have any background information on the authors? Have they published similarly impressive works in the past?
Yep I think it's the best, period. Qwen3-coder perhaps took the limelight but the GLM models perform and behave better in agentic loops. I cannot believe they had gone from a 32B frontend focused GLM-4 to these beasts that can challenge Claude, in a matter of months.
What’s up with the charts? In their first figure they show sonnet 4 as having a swebench score of 53; in the next figure it’s close to 70. In reality it’s closer to the latter. [0]
Close enough for now. I would love if these companies also published their datasets, but publishing datasets is just asking for legal trouble these days.
OpenAI, Anthropic and others are not even publishing their weights.
Ok so I asked "how many bs in bluberry" and it "thought" the following -
-----
First, the question is: "How many bs in bluberry?" That seems like a typo. I think it's meant to be "blueberry," but it's spelled "bluberry" with one 'e' missing. Also, "bs" probably refers to the letter 'b'.
So, rephrasing the question: How many times does the letter 'b' appear in the word "blueberry"?
But the word given is "bluberry," which might be a misspelling of "blueberry." Let me confirm the correct spelling. The standard spelling is "blueberry," which has two 'e's: b-l-u-e-b-e-r-r-y.
The question says "bluberry," which is missing the second 'e', so it's spelled b-l-u-b-e-r-r-y. That might be intentional or a mistake. I should answer based on the word as given.
The question is: "How many bs in bluberry?" So, I need to count the 'b's in "bluberry."
Let me write it out: B-L-U-B-E-R-R-Y.
Now, I'll go through each letter:
Position 1: B (that's one b)
Position 2: L (not b)
Position 3: U (not b)
Position 4: B (that's the second b)
Position 5: E (not b)
Position 6: R (not b)
Position 7: R (not b)
Position 8: Y (not b)
So, there are two 'b's in "bluberry."
But is "bluberry" the correct word? It's probably a misspelling of "blueberry." In "blueberry," it's B-L-U-E-B-E-R-R-Y.
Position 1: B (one)
Position 2: L
Position 3: U
Position 4: E
Position 5: B (two)
Position 6: E
Position 7: R
Position 8: R
Position 9: Y
"Blueberry" has nine letters, and there are two 'b's: one at the start and one after the 'e'.
In "bluberry," as given, it's eight letters: B-L-U-B-E-R-R-Y, with two 'b's.
Perhaps the question is tricking me with the misspelling. But I should answer based on what's written.
Another thought: "bs" might be plural for 'b', meaning how many 'b's, or it could be something else. But in context, it's likely about the letter.
Maybe it's a riddle. "Bluberry" sounds like "blueberry," but spelled differently.
Perhaps "bs" refers to something else, like the fruit or something, but that seems unlikely. The question says "bs," which is probably "b's" meaning the letter b.
Let me read the question carefully: "How many bs in bluberry?"
It might be a play on words. "Bluberry" could be intended as "blueberry," and "bs" might be misinterpreted.
Another idea: in some contexts, "b" could stand for something, but here it's probably straightforward.
Perhaps it's about the sound. But the question is written, so it's about spelling.
I think I should just count the 'b's in the given word.
So, in "bluberry": letters are B, L, U, B, E, R, R, Y. So, B appears twice.
Therefore, there are two 'b's.
But to be thorough, let's consider if "bluberry" is a word. It's not standard; it's a misspelling. However, for the purpose of this question, I'll use it as is.
Perhaps the question has a typo, and it's meant to be "blueberry." But since it's written as "bluberry," I'll go with that.
Final answer should be 2.
The question says "bs," which might be lowercase, but in counting, case usually doesn't matter, and here it's all lowercase or mixed? The word is "bluberry," which is lowercase, and "bs" is lowercase, so it's fine.
In English, when counting letters, we don't distinguish case unless specified.
So, I think two is correct.
But let me double-check the spelling. The user wrote "bluberry," which has one 'u' and no 'e' after 'u', but in "blueberry," it's "blue" with 'e'.
In "bluberry," it's "blu" without 'e', then "berry."
But in terms of letter count, it's clear.
Perhaps it's "blueberry" misspelled, and I should correct it, but I think for accuracy, I'll use the given spelling.
Another thought: the question might be "how many b's in blueberry?" but written with a typo.
But since it's "bluberry," I'll answer based on that.
So, response should be: There are two 'b's in "bluberry."
To make it clear, I can say: The word "bluberry" contains the letter 'b' twice.
But the question is "how many bs," so I can just say 2.
Perhaps it's a trick with the word "berry" having one 'b', but "blu" has one, so total two.
"Blu" has one 'b', "berry" has one 'b', but in "bluberry," it's combined, so still two 'b's.
Yes.
I think I'm overthinking it.
Final answer: 2
It still uses text tokenization, so it can’t even see the word. Not sure what tokenizer GLM uses, but OpenAI’s tokenizer renders “blueberry” as a single token (116500 or thereabouts IIRC).
It’s like asking us what the average wavelength is when looking at a blueberry: the information is actually there somewhere in our processing stack, but inaccessible to reasoning. It can be worked out logically from general knowledge, but probably inaccurately, and the gotcha of “you’re looking right at it and the photons are hitting your vision” is not much of a gotcha when you understand how it works.
Your spelling of "bluberry" almost made the model have an existential crisis. That gave me a good laugh. Out of curiosity, did you also run it using "blueberry" to see the differential?
Really appreciate the depth of this paper; it's a welcome change from the usual model announcement blog posts. The Zhipu/Tsinghua team laid out not just the 'what' but the 'how,' which is where the most interesting details are for anyone trying to build with or on top of these models.
The post-training methodology (Sec 3) is what really stands out to me. The idea of creating specialized 'expert models' for reasoning, agents, and chat, and then distilling their capabilities into a final unified model is a fascinating approach. It feels like a more structured way to solve the "jack of all trades, master of none" problem that can plague generalist models. Instead of just mixing all the data, they're essentially having a generalist learn from a committee of specialists.
A couple of the findings from their RL experiments are pure gold for anyone working in this space. The counter-intuitive result that a single-stage RL process at the full 64K context length outperforms a progressive, multi-stage approach (Fig 6) is a fantastic lesson. I've seen teams assume the opposite would be true. Also, the pragmatic choice to use an XML-like template for function calls to avoid JSON escaping hell (Fig 4) may be a small but brilliant engineering decision that makes a huge difference in practice. Wrangling escaped code inside JSON turns out to be a mess.
The performance on SWE-bench is impressive, putting it in the same league as much larger or proprietary models. What I’d love to see, and maybe others here have thoughts, is whether this hybrid training recipe holds up outside ARC-style evals. For example, do the agentic improvements transfer to messier, real-world workflows where APIs are undocumented, partial failures are common, and user input is full of ambiguity?
Are all these "post/mid-training tweaks" important if you have a specific domain with abundant/verified/synthesis data and labels?
Can a small team working on ASI/domain-specific stick to scaling 2024-era best practices training stack? Or will they miss massive improvements?
> Are all these post/mid-training tweaks important with abundant, verified, synthetic domain data?
No. Many are aimed at cleaning/aligning noisy, mixed-domain data. With abundant, high-quality domain data, you can skip most of the complexity and focus on direct SFT/RL on your corpus.
> Can a small team stick to scaling 2024-era best practices?
2024 was the year of SFT. I believe fitting reasoning traces to your final responses via RL is the technique-du-jour of 2025. Jumping from SFT to RL training might be biggest gain here if RL can be applied to your problem (e.g. math, coding etc).
I don't want to call you out unnecessarily, but your writing heavily smells of LLMs.
edit: looks like i'm not the first person to notice this either regarding this poster. https://news.ycombinator.com/item?id=44279662
I think we have a duty to call this out, before the web becomes ridden with slop.
Please don't do this here. If a comment seems unfit for HN, please flag it and email us at hn@ycombinator.com so we can have a look.
Yes, I occassionally use LLM for edits and re-writes. Opinions are mine. I thought most people do these days?
(Re: Other post you linked to. it is entirely my own thoughts.)
We prefer the raw, flawed original version than the AI-polished version on HN. It makes HN feel more real and authentic. If other community members notice that your writing seems AI-enhanced, you've taken it too far.
I do admit your post had some apparent substance to it (at least from my naive perspective), but that substance is really diminished when it's passed through an LLM. I think something is lost when someone's unique writing is 'planed' by AI. I'd really encourage you to avoid this kind of editing and just use your own words.
The comment you're replying to is 100% AI-generated. How does obviously LLM-generated content continually make it to the front of HN, and why in God's name are you being downvoted for calling this out??
"...a fascinating approach..." (LLMs think everything is fascinating)
"...they're essentially having a generalist learn from a committee of specialists..." (analogies, analogies)
"...where APIs are undocumented, partial failures are common, and user input is full of ambiguity..." (typical AI rule of three template with semantically similar parameters that contribute nothing to the overall meaning)
It does worry me how defensive people can become over really obvious slop - I don't think I'm even particularly attuned to the style of LLM writing but it is incredibly obvious every time I see it. It's only going to get worse I think.
> and why in God's name are you being downvoted for calling this out??
Tinfoil hat time, but perhaps the bots don't like being called out? I don't actually take that statement seriously, but it seems an eventual avenue. They've long been seeding threads on Reddit to shape initial hive mind, i imagine that's going to get more advanced and widespread.
> ...is what really stands out to me. The idea of...
> ...are pure gold for anyone working in this space...
Specifically OpenAI
不管黑猫白猫，能捉到老鼠就是好猫
I see your points, but is this actually slop in this case? Is the comment incorrect or misleading at all?
It felt interesting and informative to me, but I didn’t verify any of it.
Good eye btw.
You did call out.
If you read my comment closely, I didn't deny calling anyone out.
I've been playing around with GLM-4.5 as a coding model for a while now and it's really, really good. In the coding agent I've been working on, Octofriend [1], I've sometimes had it on and confused it for Claude 4. Subjectively, my experience has been:
1. Claude is somewhat better at whole-codebase tasks, where you need to reason over a bunch of context and consider system interactions.
2. GLM-4.5 is somewhat better at being "honest" — i.e. I rarely see it doing the things Claude does like making broken tests pass by changing the test instead of fixing the bug.
Both are quite good though, and GLM-4.5 has found bugs that both Claude 4 Sonnet and 4.1 Opus have failed to catch. In general I think Claude wins a little more frequently on debugging tasks than GLM-4.5, but it's close.
Compared to GPT-5, both Claude and GLM feel like they're more consistent, although GPT-5 sometimes has long brilliant runs where it nails everything with subjectively higher code quality than either of the latter. However, once GPT-5 goes off the rails, it's hard to get it back on track, so it can be a bit frustrating to work with in comparison.
1: https://github.com/synthetic-lab/octofriend
I just read your comment and decided to give GLM-4.5 a try in Kilocode. I'd been using Gemini CLI all day to try to resolve a tricky bug in some compiler code (a compiler for a subset of C that generates microcode for... a weird architecture, I'll leave it at that). So GLM-4.5 zoomed in on the problem right away. A problem that's eluded Gemini CLI all day. Gemini was leading me on a wild goose chase implicating a function that turns out wasn't the problem (and trying to make all kinds of lame changes to the function saying that would fix the problem - and it never did because the problem wasn't that function).
Sometimes getting a second pair of eyes to look at the problem helps and is usually not a judgement of smartness of the first pair of eyes. Seems like it also applies to coding agents.
Indeed, I've also found that various models are good at various tasks, but I have yet been able to categorize "Model X is good at Y-class of bugs", so I end up using N models for a first pass "Find the root-cause of this issue", then once it's found, pass it along to same N models for them to attempt to solve it.
So far, which model can find/solve what is really scattered all over the place.
You are experiencing the jagged skills frontier. All models have these weird skill gaps and prompt phrasing sensitivity. This is the main problem solved by an llm-consortium. It's expensive running multiple models in parallel for the same prompt, but the time saved is worth it for gnarly problems. It fills in the gaps between models to tame the jagged frontier.
My very first use of the llm-consortium saw me feeding in it's own source code to look for bugs. It surfaced a serious bug which only one out of the three models had spotted. Lots of problems are NP-ish so parallel sampling works really well. Googles IMO gold and openais IOI gold both used parallel reasoning of some sort.
This is so true. Another thing, a model might be better at something in general, but worse if the context is too long. Looking at how GLM-4.5 is trained, on lots of short context, this may be the case for it.
GPT-5: Exceptional at abstract reasoning, planning and following the intention behind instructions. Concise and intentional. Not great at manipulating text or generating python code.
Gemini 2.5 Pro: Exceptional at manipulating text and python, not great at abstract reasoning. Verbose. Doesn't follow instructions well.
Another thing I've learned is that models work better when they work on code that they themselves generated. It's "in distribution" and more comprehensible to them.
The good old regression to the mean. Testing models as the second pair of eyes only when the first fails is going to give weird results... https://www.smbc-comics.com/comic/protocol
How does the SMBC comic demonstrate regression to the mean? Seems like it's a better example of a post-hoc fallacy.
If you choose the worst performers, a part of their failure will be due to luck and part due to skill. Letting them retry will average the luck in that sample. (So likely increase) Of course you as the tester could be very unlucky and pull a sample of people with no skill who did get a bit lucky in the first place, but it's both unlikely and... it's a funny comic.
Gemini CLI uses whole file edit format and goes through the token very fast. I use aider for this reason with diff fenced, it burns very less tokens.
I don't seem to have to worry about burning tokens with Gemini CLI. It's all been free so far. That'll probably change sometime in the future, I'd guess.
I am curious about your setup? Is it just gemini cli? Or are you combining it with other frameworks?
I use gemini cli in kilocode. (same for GLM-4.5, using that in kilocode)
I've had similarly good experiences with GLM-4.5 for smaller projects/requests. Unfortunately that did degrade with larger contexts, so I'm still treating it as a good fallback for Sonnet 4, rather than a full-blown replacement.
I've been using architect mode in aider
Deepseek R1 (does high level planning) combined with Qwen3 480B (does low level coding) or whatever is available from qwen code apis.
It's working great.
It solves 99.99% problem on tis own.
The seperation isn't very good in aider so i later plan to make my own tool to achieve better workflow.
What’s your monthly bill (OpenRouter?) if I may ask? I have Claude Max and always on the lookout for alternatives, at least for the easier to solve problems.
About your first point, I also feel like Claude is better if there’s more in the context where 4.5 is getting "worse".
claude used to be better not anymore or atleast the difference is not that much
deepseek r1+ qwen3 is close enough along with gemini2.5 pro
so i don't see any point of claude anymore
How are you using glm-4.5? Are you consuming the api or running something like glm-4.5 air locally?
I run a privacy-focused inference company, Synthetic [1], and I use our API of course :P I actually like GLM-4.5 enough that it's currently our default recommended model for new users. But yes, otherwise I'd use the official zai API most likely, or Fireworks. GLM-4.5-Air is quite good for a local model but GLM-4.5 is better; up to you if the tradeoff is worth it — there's definitely value in the data not ever leaving your machine, but it's not going to be as strong of a model.
1: https://synthetic.new
What makes your service especially privacy friendly?
I think if you are striving for full privacy, you should implement the secure enclave idea presented by ollama, it makes the entire pipeline fully encrypted, I'm waiting for an actual provider to finally implement this.
https://ollama.com/blog/secureminions
We don't store prompts or completions for the API (our privacy policy says "for longer than 14 days," as mentioned elsewhere in this thread — we don't actually store them at all, but the 14 day legal guarantee is to make sure that if someone accidentally commits a log statement, we have a little bit of time to catch it and revert without being in breach of policy). And, we don't train on your data, even for messages in the UI: we only store UI messages in order to let you view your message history, not for training.
Compared to using — for example — DeepSeek from deepseek.com, I think we're much more private. Even compared to using OpenAI and opting-out of your data being used for training, we're still more private, since OpenAI makes no guarantees for individuals that they don't store the data — notably, any data ever sent to them is apparently now being shared with New York courts (and the New York Times!) due to their ongoing legal battle with the New York Times [1]. And compared to using OpenRouter with "data_collection: deny", we uh, actually work :P Surprisingly sad how many broken model implementations there are if you're just round-robin-ing between inference companies... Especially reasoning models, and especially with tool-calling.
(And if something's broken, you can email us and we'll generally fix it; OpenRouter doesn't actually host any models themselves, so there's not much they can do if one isn't working well other than just de-list.)
1: https://arstechnica.com/tech-policy/2025/07/nyt-to-start-sea...
> our privacy policy says "for longer than 14 days," as mentioned elsewhere in this thread — we don't actually store them at all, but the 14 day legal guarantee is to make sure that if someone accidentally commits a log statement, we have a little bit of time to catch it and revert without being in breach of policy
I'd recommend to rephrase your marketing, because not storing prompts is a huge selling point compared to deleting after 14 days (a lot of things can happen in 14 days)
From the privacy policy:
> we will not sell [personal information], except as follows: > - We work with business partners who support us.
Uhhm, that doesn't inspire a lot of confidence TBH!
I don't think that's an accurate read of our privacy policy. What you've left out is:
These third-party service providers are prohibited from using personal information for any other purpose and are contractually required to comply with all applicable laws and requirements, which may include Payment Card Industry Data Security Standards if they are processing payments.
We use third parties like Stripe and Clerk, and by nature of using those services, your information is disclosed to them. This is an extremely common clause in privacy policies, and one we need to have unless we roll everything ourselves. We're much more private than using, say, OpenAI, DeepSeek, Anthropic, or most popular LLM services.
Do you let people explicitly choose EU servers ?
Also, your Privacy Policy is not currently EU GDPR compliant. ;-)
Oh sorry — our lawyers are American, I'll shoot them an email and see if we can get that fixed. Is there something you were looking for but couldn't find?
We have servers in the EU and US, but right now there's no way to route to only EU (or US) DCs.
Thanks for the swift reply.
Couple of things stood out:
- No nominated representative to act as local contact for data subjects and supervisory authorities. - A few standard clauses relating to what and where data is transferred / stored etc.
Nothing too strenuous, but I thought since you've gone to the effort of having EU servers I guess you're expecting EU customers and so hence probably worth tweaking your privacy policy.
I look forward to experimenting with GLM-4.5 through your service !
I’m curious for your service, if it’s centered around privacy, why is the data stored for 14 days at all? My understanding with fireworks is that it’s 0 logging — nothing to store. To me that’s private.
Great question! We actually don't store prompts or completions for the API at all — but legally we say 14 days so that if someone accidentally commits a log statement, we're not in breach as long as we catch it quickly and revert.
Amazing! So I’m assuming that because it’s privacy focused, you accept payment in cryptocurrencies like Monero and Zcash?
We accept USDC and USDP crypto payments via Stripe. We don't currently support Monero or Zcash — right now all our payments are via Stripe since it simplifies security + compliance for us. It would be a pretty neat feature to build though.
You support logprobs, that's wonderful! Fireworks, Synthetic, (ik_)llama.cpp, now I have a quorum.
Not OP. Chutes.ai charges $0.20 per 1M tokens. I don’t think it uses caching though because I ended up burning $30 in an hour or two. I had to move back to Claude Code.
Caching makes price comparisons hard. Does anyone have tips?
So GLM-4.5 series omits the embedding layer and the output layer when counting both the total parameters and the active parameters:
> When counting parameters, for GLM-4.5 and GLM-4.5-Air, we include the parameters of MTP layers but not word embeddings and the output layer.
This matches with the calculation I did for GLM-4.5 (355B A32B):
Meanwhile, GPT OSS series includes both the embedding layer and the output layer when counting the total parameters, but only includes the output layer when counting the active parameters:> We refer to the models as “120b” and “20b” for simplicity, though they technically have 116.8B and 20.9B parameters, respectively. Unembedding parameters are counted towards active, but not embeddings.
And Qwen3 series includes both the embedding layer and the output layer when counting both the total parameters and the active parameters.
Why there is no standard in counting? Which approach is more accurate?
I'd say it depends. For the total parameter count, you should just count all parameters, since that's what matters for memory requirements.
For activated parameters: All unembedding parameters are used in every inference step during token generation, but only one column of the embeddings is used (if done right). So count accordingly, since that's what matters for memory bandwidth and therefore latency.
That makes sense, thanks for the info. Here's a quick recap of the recent MoE models based on the criteria..
correct activated params:
undercount activated params: overcount activated params:Seems like we may get local, open, workstation-grade models that are useful for coding in a few years. By workstation-grade I mean a computer around 2000 USD, and by useful for coding I mean around Sonnet 4 level. Current cloud based models are fun and useful, but a tool that is / will be so core to the developer experience, I want to be able to run locally.
This will be essential for the open source. Otherwise open source development will become unsustainable. I'm actually a little bit more optimistic. I think we will get something more than Sonnet 4 level in two years, that can be run on a $2,000 machine.
I've been waiting to do exactly this. Improved latency and no throttling is super appealing to me.
I would say by end of year, not 2 years.
Yeah, if we are lucky it might happen that fast!
This feels like the first open model that doesn’t require significant caveats when comparing to frontier proprietary models. The parameter efficiency alone suggests some genuine innovations in training methodology. I am keen to see some independent verification of the results and to see how if does on Aider’s LLM Leaderboard.
The sheer number of things "they observed" in this paper that could be whole papers in themselves is astounding! Lots of great stuff in here around training processes and data collection+synthesis.
Does anyone have any background information on the authors? Have they published similarly impressive works in the past?
Fantastic release, and it's under the Apache license too. I'm so happy that we've got open source models pushing the envelope.
For people who, like me, prefer to read the abstract first: https://www.arxiv.org/abs/2508.06471
I’ve been using it and I think it’s on par with sonnet.
This is a great model for software development - probably the best of the freely available ones.
Yep I think it's the best, period. Qwen3-coder perhaps took the limelight but the GLM models perform and behave better in agentic loops. I cannot believe they had gone from a 32B frontend focused GLM-4 to these beasts that can challenge Claude, in a matter of months.
Why was qwen3 omitted from the coding benchmark but not other benchmarks?
Section 4.3.2 includes Qwen3-Coder.
Qwen is still naive for large codebase understanding.
What’s up with the charts? In their first figure they show sonnet 4 as having a swebench score of 53; in the next figure it’s close to 70. In reality it’s closer to the latter. [0]
[0] https://www.swebench.com/
It's ok, somewhere between a qwen 2.5 VL and the frontier models (o3 / opus 4) on visual reasoning
Huge respect to the open-source culture in China. The Chinese are really leading the world in democratizing AI.
It's open-weight, not open source though
Close enough for now. I would love if these companies also published their datasets, but publishing datasets is just asking for legal trouble these days.
OpenAI, Anthropic and others are not even publishing their weights.
Impressive work! Can’t wait to spin it up on HF
Ok so I asked "how many bs in bluberry" and it "thought" the following -
-----
-----It still uses text tokenization, so it can’t even see the word. Not sure what tokenizer GLM uses, but OpenAI’s tokenizer renders “blueberry” as a single token (116500 or thereabouts IIRC).
It’s like asking us what the average wavelength is when looking at a blueberry: the information is actually there somewhere in our processing stack, but inaccessible to reasoning. It can be worked out logically from general knowledge, but probably inaccurately, and the gotcha of “you’re looking right at it and the photons are hitting your vision” is not much of a gotcha when you understand how it works.
Your spelling of "bluberry" almost made the model have an existential crisis. That gave me a good laugh. Out of curiosity, did you also run it using "blueberry" to see the differential?
"I think I'm overthinking it."
Kimi K2 has been the best so far in my experience
so sarcastic when it first came out, but now you are so polite.