> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
Why are resource limits considered at all aside from models accidentally fork bombing themselves?
I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?
Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness in a given situation.
A few examples from memory:
1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).
2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.
3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes
This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.
Thats what is wrong with close source models, we dont know exactly what we are paying for, a superior base model or a well thought harness for benchmaxing
I get your point but I'm not sure it matters all that much.
Did harbor / tb2.1 cap the swap available to docker runs?
There used to be a bug that would allow dockerized instance runs to use more memory than the specs allowed. Some of the original tasks weren't really possible to complete without exploiting swap. Even the oracle solutions didn't pass if you stopped docker from having access to swap.
I think crack-7z-hash and filter-js-from-html had that problem off the top of my head, but i haven't looked at this in months, so i'm not sure
What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it
Yes but my point is
- Resource limits are a "recommendation" and are not strictly enforced
- Significantly boosting resources up to 3 did not statistically shift performance results
Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results
I had a few days of preview access, which was long enough to put together a plugin for LLM. You can try the model out in the terminal like this:
uv tool install llm
llm install llm-meta-ai
llm keys set meta-ai
# paste API key here
llm -m meta-ai/muse-spark-1.1 "Generate an SVG of a pelican riding a bicycle"
It does feel like a chore in weeks like this one where there are new models landing every day.
I'm also increasingly worried that I'm not providing enough value in my model reviews. It's really hard to get a useful and credible idea for the strengths of the new models.
My blog genuinely is a day job now, I make enough from the (unobtrusive, cookie-free) sponsorship banner that I no longer regret not having a proper software engineering job.
> My blog genuinely is a day job now, I make enough from the (unobtrusive, cookie-free) sponsorship banner
Posting his links and engaging in the discussions is his main job. This is spam, even if you like it - because most people don't want to see one person posting their links on every AI discussion, ranked at the top just because among hundreds of thousands of people reading HN, some thousands are not annoyed by seeing his blog links in every post.
The corect term is being a tech influencer and even if you say it with a bad connotation it's completely different than spamming which is low/no effort posting and most of the times selling something.
If having a job makes you unable to use basic English, then it's time to take a break like he does maybe.
If you think all he does is post pelicans on bicycles then you aren't actually reading or seeing as many of his posts as you think you do. There's quite a bit more to it than that.
Also for others, the pelicans on a bicycle posts are comparing results between models using a benchmark. They're not just random pelicans for the art value.
No one said his post was high effort/quality but please reply again and with your amazing language skills twist my words to double down on petty points like that.
I see you're being blammed. "They hated him because he spoke the truth." Just because this online character is from your alma mater website doesn't mean he's not cynically and shamelessly self-promoting.
Maybe Zuck should double down on his "spoiler" role with models rather than compete head-to-head.
He doesn't have to match Anthropic or OpenAI model revenue if he can deflate theirs by 99%.
All he has to do is keep spending a few billion dollars developing frontier models, release them as open weights, and turn coding models into a commodity. He also needs a good OSS reference harness to match. Very few people are in a position to do this and for it to make business sense.
That's quite likely where things are headed regardless, and he could speed it up significantly.
We should all hope models move from proprietary products to commodities the way compilers did.
This may be one of the best things Zuck could do for the world.
The goal is not for meta to take their market, the goal would be for meta to damage their competitors.
If meta releases an open-weight LLM that is not Chinese made, cheaper to run than the SOTA premiums, etc, it would lower the number of people paying for frontier labs models. We saw with with early LLAMA models, but they didn’t keep up in the race with v4.
Meta would benefit from this, not from increased revenue at the hands of open LLMs, but from reduced competition. Meta competes with Google for ad spend, and lowering the Google revenue (or increasing costs) from AI reduces the competitive advantage. OpenAI wants to build an ad engine, so same thing will apply there too - make it less-revenue-generating to compete. Meanwhile G, OpenAI, and Anthropic are huge talent sinks that they have to compete with, especially for ML talent which is core to Metas business goals (ads). Finally, Meta needs lots of GPUs to train their ad engine models. By reducing the revenue-per-GPU of these labs, they’re reducing demand on a core revenue generating supply they have to compete for.
The problem is that Meta jumped headfirst into the circular financing of datacenter build out, so they're just as screwed if the infrastructure becomes less valuable.
all he has to do is that prove builing these inst hard anymore. because the whole moat these companies have is the perception that building models at frontier is really hard .
Software has several unique properties on both ends of its production process that make assertions of progress based on the software use case invalid.
Software is easy to define as “working”: just run it. But - useful software requires an absolute truck worth of code - 100k loc before you’re talking about a real product, or else dozens of iterations of a toy you make for yourself before it’s useful enough to quit toying with and just use for what you wanted it for.
Sure, the success of software is hard to anticipate and what “good” UX is is hard to pin down - that’s not what I’m talking about. I’m talking just making the code and having no lint errors. That shit is a slog but it’s a slog with a clear goal amenable to hill climbing.
Through that lens software is mostly pattern matching. It’s very rare that an activity in software construction is out of distribution because even if the core of the thing is novel it needs a massive blanket of UI and a tech stack and a production environment to run in and observability and and and and. You get it I hope.
Meanwhile most work out there is a mess of undocumented, un-codifiable detail with no objective criteria for success, only a very wide gradient of “job well done” to “what is this garbage go and fix it”.
We are solving the easy parts of software and soon all that’s left will be the parts that are just like other work. And then we engineers will also be doing mostly squishy subjective judgment stuff.
Yeah, this is most directly comparable to xAI Grok 4.5. In both cases, directionally "opus level intelligence for haiku prices" which is a really big deal for application developers who want to include models like this in their applications. I have been testing switching out haiku and sonnet for Grok 4.5, and may give this a try too (it is quite a bit cheaper, particularly for cached).
> Yeah, this is most directly comparable to xAI Grok 4.5.
Grok 4.5 has a relatively high $0.50 per 1M cached input token rate, compared to $0.15 on this model.
Grok 4.5 cached input costs the same as Opus 4.8 cached input, which is going to make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
> ... make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
There's a further sting in the tail, Grok 4.5 is only $2/$6 for the first 200k of context. Go above that, and the pricing is $6 / $12 - and you're still capped at only 500k context anyway.
This is still ridiculously expensive imagine having to pay $10 for 100 search results on Google, thats essentially what this is.
I really dont see how anyone's willing spend more than $1.50 per mm output. Let alone $15-50. Does anyone actually pay for usage based billing as a consumer?
Sometimes. It depends on the task. $15/Mtok is still a lot cheaper than human written code. It’s probably worth spending more for contract reviews. Tasks don’t have uniform value. If you’re using an AI for entertainment, then frontier premiums are hard to justify. For paid work, they’re a great deal if used reasonably.
Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead and now with xAI and Meta at least delivered something that's competitive with useful models and cheap too. Granted, the narrative that the two leading labs are ahead still holds with Fable (and perhaps an upcoming GPT6), but it's not as over as common knowledge by the opinion leaders would have us believe.
> Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead
Not the way you're implying?
The GLM 5.2 hype was blowing way before this. Neither xAI nor Meta have really made a difference in a different way - similar results / similar pricing (to GLM 5.2).
Think it’s less a shift in who’s ahead and instead more people acknowledging that making yet another dashboard app worked fine on last years SOTA and doesn’t really need today’s SOTA
People misinterpreted Google being behind as Anthropic and OpenAi being really ahead, when it was really just Google falling behind the same way it did with Tensorflow, Angular and GCP.
> when it was really just Google falling behind the same way it did with Tensorflow, Angular and GCP
Not sure I agree. Angular fell behind in popularity but was (is? unsure atm) still eminently usable. I gave gemini a test drive recently and it was horrendous, as in "picking dirt cheap Chinese model over gemini any day" bad, and with overzealous guardrails to boot. 3.1 pro feels a year behind and is extremely lazy. 3.5 flash feels like a model you’d run on your 128gb macbook, not something that was released a month ago and which costs a fair bit when used through api.
In any case: as of right now I think that we went from a three horse race to anthropic / openai as premium choices vs whatever is the Chinese fotm for a fraction of the cost. 3.5 pro better be a miracle if google wants to hang out with the big boys, otherwise their only strategy is hoping that both US labs go broke and they remain the last man standing.
Gemini is more than fine as a mobile application, and will be the “brains” of the currently braindead Siri, between than and Android it’s hard to come up with an argument they are behind.
Likewise, Gemma4 models are unbeatable at their size.
So I use Gemini for coding? Hell no, but that’s not the same as Google failing writ large.
Googles only real goal is to retain the ownership and dominance of the online world they have now, and Gemini is doing exactly what it needs to do
I dont think gemini needs to be opus level though... if its there to power google search then the scale for google to operate at is enormous compared to openai/anthropic.
Real Thing is going to come from a major breakthrough. I’m talking about competition amongst models. Difference between Fable and GPT Sol Ultra is so minor the only thing that really matters is that GPT is cheaper.
Hell, difference between Opus and Fable isn’t that big. I would still prefer Opus for most task because it’s cheaper. Break things down into smaller tasks and create deterministic parts of the process and you need the more expensive models less and less.
Anthropic needs to get their token efficiency under control or they will lose long term.
Yeah, I think it is definitely great. Having said that, I am still debating in my mind whether the volume of software engineers needed in the AI era is going to increase or decrease because of all of these advancements.
On the one hand, because it is easy to build products, more and more people will build. And more and more products and features will be built. However, a lot of people who are non-technical will also try to build, but they get stuck, and then they will need engineers. The sheer volume of product built by both experienced technical companies and non-technical novice startups and founders and wannabe founders is going to be massive. That is the bull case for having more software engineers needed in the near future.
On the other hand, in a year or so, people will build all these products, and most of them won't be able to market them, sell them and make money. Eventually, there won't really be a need for that many software engineers.
I think overall the bull case is probably going to win net net.
I see some similarities to 3D printing here. It’s great that everyone can make their own toothbrush holder (or whatever) but I’m probably not going to pay for someone’s weekend project.
I’m “seeing” more devs stepping into the SendCutSend stage where they’re cleaning up/fixing/productizing vibe coded projects so maybe there will be some new demand in that space?
3D printing is a good comparison - it allows almost anyone to make things, but in the end very few do.
Another example is when the WWW first became available, and suddenly everyone COULD be a publisher (browsers even included built-in HTML editors), and for a while MySpace pages proliferated until the excitement died down and people went back to being media consumers.
I expect we'll see the same thing with consumer use of generative AI. Suddendly everyone is generating 3-D worlds/games with Fable because they can, but I expect that just as with the web the novelty will wear off and they'll leave it up to the pros.
Professional use of GenAI, and coding in particular, is certainly here to stay, but it seems we're still in the early experimental/hype phase. At least tokenmaxxing has passed, and it seems most companies are now paying attention to, and limiting, how much they are spending, but it doesn't seem we've yet progressed to the stage where companies are paying attention to what they are actually getting out of it - is the money spent showing up on the bottom line in the form of increased revenues.
A comparison I find useful here is Excel (and spreadsheets in general). Those enabled huge numbers of non-programmers to build software-like things, while the demand for expert developers grew enormously at the same time.
It’s terrible and depressing work to take vibe coded garbage and make it a real product. There will be demand, but good engineers won’t want to touch it. And people paying will think they did the hard work so why pay a good rate?
I think LLMs/coding agents in particular are going to play out like automation always does.
1. You have a task and it requires an expert to perform it (software engineering, where we were)
2. You automate the task, it still requires the expert to babysit it (where we are now)
3. Management works out that the expert is just monitoring what the automation is doing and has actually lost expertise because they are just watching, not doing. Pay collapses. (where we are going)
> At least in China a lot of software developers are now struggling.
Do you think that Chinese software industry is that relevant to the kind of software market talked about on HN? I.e. lots of enterprise b2b and infra companies.
Chinese companies have always had a very low willingness to pay for software which kinda breaks the flywheel of B2B SaaS companies and companies to service those companies all the way down.
> Chinese companies have always had a very low willingness to pay for software
Are we still left with this mindset? Maybe once upon a time but it has definitely been changing.
There's plenty of B2B and enterprise SaaS companies in China serving the Chinese market. Maybe not as many, but no longer the very low of the past.
I also would not say enterprise were not willing to pay, even many years ago. It's the SME that refused to pay. Large CRM, ERPs etc have always existed.
I would say I'm still in this mindset. Numbers are hard to come by but analysts optimistically put Chinese SaaS market at ~10% of the size of the US. Also I see that the gross margin for public Chinese SaaS is around 50% vs 80% for the US reflecting that SaaS in China is much more services and implementation heavy. So it feels like the direct to business SaaS's like Salesforce aren't really there and then then the selling to SaaS titans like Datadog have a much smaller flywheel to work in.
Happy to be wrong though if I'm missing something.
> I would say I'm still in this mindset. Numbers are hard to come by but analysts optimistically put Chinese SaaS market at ~10% of the size of the US.
Exactly it's mindset. I used pulled up 1st result when I used Google [0][1] and it is ~20%. It also forecasts huge growth potential.
> reflecting that SaaS in China is much more services and implementation heavy
I wouldn't read into it too much. There's creative accounting and other things involved. Also what is services and not e.g. "premium support"?
> So it feels like the direct to business SaaS's like Salesforce aren't really there and then then the selling to SaaS titans like Datadog have a much smaller flywheel to work in.
US companies struggle for many other reasons that I rather not go into but definitely not a SaaS nor China issue. I mean yes it is a China issue as in businesses need to adapt to different cultures rather than 1 size fit all but that applies everywhere.
> On the one hand, because it is easy to build products, more and more people will build.
And those people won't need to be software engineers.
> but they get stuck, and then they will need engineers
You've implicitly assumed here that the AI systems will always be worse than the average engineer. That is IMO myopic. I'm not sure that it's even true now let alone in the nebulous future.
> And those people won't need to be software engineers....You've implicitly assumed here that the AI systems will always be worse than the average engineer.
Most of what we do as engineers is precisely describe or analyze the behavior we want or the behavior we don't want. All other engineering skills that are useful are ultimately downstream from understanding the behavior of software enough to know which parts to keep, improve, or jettison. Chatbots can take care, somewhat, of analysis or expansion of instructions.... but they can't read minds. I don't see that changing any time soon.
I don't know who needs to hear this, but neither can humans.
You've implicitly assumed here that AI systems will always be worse at contextualizing and framing questions than the average engineer. I'm not sure that it's even true now let alone in the nebulous future.
You haven't narrowed the fundamental myopia of the assumption here, just dressed it in slightly different clothing.
I don't understand what you mean. I can't build software I can't describe.
If you're implying chatbots can ask their "client" what to build, good luck with that—contractors are at least liable for what they produce and have extreme incentives to ensure that their clients are happy. To the extent of refusing to build anything if they don't know what they want....
By asking the user to explain what they want whenever there's ambiguity.
Plus all the other things that software engineers generally have not learned to a professional level even if they picked up the basics on the job by osmosis, because figuring out the customer's needs (and what they'll pay you for which may be different) is the job of a business analyst, a PM, or a UX researcher, and those are different skills and two of them may come with a Business Informatics degree rather than a CompSci one.
LLMs can be "eh, better than nothing" at many things, not just code.
And when an LLM runs up costs for a small company by getting them to lease a bunch of infrastructure they don't need, who can they sue? A contractor or advisor you can't hold liable is just a liability.
Same person they'd sue if they used any other power tool themselves and it didn't work out right.
Plus, this is software "Engineering" we're talking about, which famously gets scare quotes in comparison to all the other forms of engineering because unlike them we don't have as standard things like professional liability insurance to cover serious professional errors of judgment the way someone who signs off on a bridge that collapses would have.
> And when an LLM runs up costs for a small company by getting them to lease a bunch of infrastructure they don't need, who can they sue?
This question is completely disconnected from reality. If you try to sue a human for proposing something more complex than what you need you will waste a lot of money and then lose the lawsuit.
Also the annual cost of too much small company infrastructure is less than the cost of even a single good human engineer.
The big thing to me is why are we even running these models on top of an operating system?
What I really want is Claude as a deep part of the operating system.
If that happens then a whole lot of the abstraction of software vanishes along with what we think of today as software jobs. I think many new forms of knowledge work would emerge from this though.
I would think that needs massive local compute but I can't imagine that is not the future down the line.
It’s also not the future SV is incentivized to build. They want everything for rent, nothing can be owned.
Luckily, China is on the verge of a true breakout, I’m not sure what exactly it will be - but I’d make a very large wager the “next iPhone” is Chinese, and will constitute a full blown “Sputnik moment” for the US and SV.
If Americans weren’t forbidden to own Chinese EVs they’d know this. But tariffs mean the breakthrough will be even more unexpected.
Since Chinese actually “sell stuff” I’m guessing their unbeatable lead in AI efficiency, manufacturing, and distribution will produce a step change breakthrough within a decade.
> No wonder we still can’t get climate change under control
This is was historically a money issue, being green used to be wildly more expensive.
Now being green is cheaper, the limiting factor is how fast PV and batteries can be made or imported.
Recent reports of the sum of all US data centres currently in planning, has a power demand exceeding the (capacity-factor-adjusted!) global annual supply of new PV.
This would be less of a problem, but still a problem, if Trump wasn't trying to get in the way of anything green, or if the companies building data centres decided to also support factories to make more PV.
* Planned new demand: 300 GW; PV factory capacity ~ 600 GW nameplate, but the capacity factor is 14% so that's really 84 GW on average.
I'm looking ahead to the next wave of open-weight models that are as efficient as DSv4 (which is really efficient), and have been heavily distilled on GLM 5.2 (which is trivial, given it is open weight)
I use it all the time through Fireworks. The normal version when I pay it myself and the fast one when company pays. It's really fast and I never get rate limited with my daily use.
He came to X to post about this instead of his very own meta threads. This just shows how much interested he is to make this thing big, and of course, the cost can stay bearable for us considering all of these cash burn that these companies are doing
I would call the founders of DeepMind (Demis Hassabis, Mustafa Suleyman, Shane Legg) very smart people. Im pretty sure with the amount of funding everyone of these companies have, they have a long list of very smart researchers in their companies.
It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
This not being available on Openrouter really makes it hard to test. I was going to compare vs Grok 4.5 and GPT-5.6 Luna, but I don't want to deal with signing up for Meta for it unless it checks out. Please Meta make this available.
Where is the data retention policy information for paid API per-token uses? Every other provider has one and makes it clear how they handle your data. A quick look doesn’t show one for this new offering.
Paid prompts are not used for training. Data retention is not entirely clear, they will keep it if they think you're breaking the rules, but it's not clear how long they keep it if they don't think you're breaking the rules.
Just got it working with codex in a container! FYI I think there is a bug most others will run into at the Codex:Muse interface.
It's some kind of parsing or integration error due to what I think is codex not anticipating server-side tool calling and how meta treats those ids... first couple times running codex with muse, it would fail on its first non-web search call.
Got it fixed, not personally sold on the bespoke server-side tool calling and indefinite file storage yet, but also a very cool model that I'm enjoying using so far!
Debugging and diagnosis is very tool call heavy, whether that's grepping / transforming logs, calling out to profilers/tracers, or even just writing up incident reports.
Bug diagnostics is about being okay at coding but better at tooling.
Given a good diagnostic report, it can be handed to opus for the fix.
Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
I wonder if we'll start to see that pattern with every new release. Tool use likely changes rapidly, so the newest, rather than most intelligent, model may always have an edge.
What you mean.. The tools are all just invoking bash and terminal/cli cmds and http requests. Paradigms that have existed and stayed mostly unchanged for decades.
This sounds... kind of useless? Really good JSON or similar constrained decoder performance is interesting, but normal decoder > tool validator loop with good error message > tool retry is almost always able to get a tool to work second try, and input is cached so it's not expensive.
Yes, but each tool call has a different failure %. The tool calls that make up the majority of volume like grep are going to have nowhere near a 5% failure. A custom user-defined skill having a 5% failure rate is probably fine.
Exactly. I have zero trust in them, especially given their history of shady practices - like buying a VPN service (Onavo) and using it to track how people used competing apps.
Interesting that neither meta nor xai chose to do open source given that they are both clearly behind Google, OpenAI and anthropic - and a serious us open source offering would give them a clear foothold.
I suspect they have a brand problem from their social media ties and shady histories. I personally will never use their models, plenty of better alternatives. I'm now exclusively on open weight models
Nearly every model can be found on OpenRouter and used with a single key. Meta Spark is not among them, but Grok and almost every other model is. That's how I try models I don't already have an account for.
I missed the fact that Meta was developing and releasing closed-weights models... bummer. Would be great to see some more progress with American open-weights models.
Perhaps on brand for Meta, there appears to be no mention anywhere on this page or any of the docs pages, of any data retention policy. Maybe this is hidden in an agreement you need to sign when making an account? Has anyone actually done that?
Very strong pricing, cheaper than Grok 4.5, particularly the cached reads. We'll have to wait to see if it's actually worth using (it's not on OpenRouter yet).
Good to see Meta finally back to releasing something at least worth evaluating. And it sounds like they did at least a bit skate to where the puck is going by focusing on tool and computer use.
It's a high quality benchmark for sure, but it being public means it's at risk of leaking into the models (unintentionally or not), right? For that reason I prefer to look at the private ones, like: HLE, SimpleBench, Kagi, ARC-AGI.
Not open source, so not relevant. The only ones that can be relevant while being closed source are anthropic and openai. Not even Google except for search, not even spacexai. People use chatgpt more than Gemini for the average person who is not searching.
Thanks for the read. It seems to confirm that resource limits are an important factor for terminal benchmarks:
> The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.
> An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.
So changing the resource limits changes the benchmark. Yet their score table claims their score to be for Terminal-Bench 2.1, not Terminal-Bench 2.1 with raised limits.
Competition for cheaper and efficient models is a good thing, regardless of if you don't like SpaceX, Meta, etc. Especially from US based labs
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
Yes and Zuck effectively disbanded the entire team that did that. Not saying we shouldn't cast a critical eye on it, but it probably does warrant a second chance.
I don't think it even matters. Because noone will continue to use an LLM that doesn't work well for them, whether or not it has a good bench result. So for their own sake, the correct representation can actually win them some loyalty:
eg. Model X is weaker than Fable, but competes well with Opus/Sonnet and costs 1/5th as much etc - something similar playing out with Grok 4.5.
Everyone has been loving to shit on the Alexander Wang acquisition but this seems legitimately impressive to me?
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
As far as i remember, the entire AI org was essentially gutted and replaced with whoever Wang wanted to hire, and tbh that org completely failed to train llama 4 and I honestly doubt whatever techniques they used to ship llama 3 are at all relevant now. That was before reasoning models and the heavy emphasis on RL/post-training.
so yeah, this is essentially their first try with a completely new org.
I agree with parent, Meta has been at this a long time and its only because they have recently fallen off that they pushed this "oh give us credit its really a new org" thing. Basically, if you can't actually "win" then try to fake a restart and say we are the fastest.
Even given that, this is their second try (they had Spark 1.0). Spark 1.0 was uninteresting, this is potentially interesting, but we can't really try it yet it seems (at least not in Openrouter).
Ultimately, competition is now fierce in this broad level of intelligence/cost:
Spark 1.1, Grok 4.5, GPT 5.6 Luna, GLM 5.2
Sonnet not in the same ballpark of pricing (more expensive than Opus in many cases). Haiku has been basically abandoned.
It's great that we have yet another competing models. The more models we have, the less likely we are subject to the ideologies and the controls thereof by the cults like Anthropic. And of course, it drives down the cost of tokens.
Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio...
From Terminal-bench-2.1 details,
> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
Why are resource limits considered at all aside from models accidentally fork bombing themselves?
I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?
Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness in a given situation.
A few examples from memory:
1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).
2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.
3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes
[1] https://github.com/harbor-framework/terminal-bench-2-1/blob/...
[2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...
> Which test cases does this matter for?
The test cases of "don't melt my computer" and "be a good (computational) neighbor"
This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.
Out of curiosity, how often are the resource limits the bottlenecks? What do harnesses do to help here - limit parallelism? More efficient tools?
The task could be verifiable in the environment so limiting its CPU and RAM could be to discourage brute forcing the answer.
Thats what is wrong with close source models, we dont know exactly what we are paying for, a superior base model or a well thought harness for benchmaxing
I get your point but I'm not sure it matters all that much.
Did harbor / tb2.1 cap the swap available to docker runs?
There used to be a bug that would allow dockerized instance runs to use more memory than the specs allowed. Some of the original tasks weren't really possible to complete without exploiting swap. Even the oracle solutions didn't pass if you stopped docker from having access to swap.
I think crack-7z-hash and filter-js-from-html had that problem off the top of my head, but i haven't looked at this in months, so i'm not sure
Huh? What are you talking about?
https://www.anthropic.com/engineering/infrastructure-noise
Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either
What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it
Yes but my point is - Resource limits are a "recommendation" and are not strictly enforced - Significantly boosting resources up to 3 did not statistically shift performance results
Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results
I had a few days of preview access, which was long enough to put together a plugin for LLM. You can try the model out in the terminal like this:
Here's the result: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...For comparison, here's the pelican I got from Muse Spark 1: https://simonwillison.net/2026/Apr/8/muse-spark/
How do you find the time to “preview” so many models? It’s been a crazy time recently with the model releases. Does it ever feel like a chore?
It does feel like a chore in weeks like this one where there are new models landing every day.
I'm also increasingly worried that I'm not providing enough value in my model reviews. It's really hard to get a useful and credible idea for the strengths of the new models.
Please don't listen to those miserable people, I find your posts really interesting
His primary interest nowadays is LLM's. Unsure if he gets paid for it, but he seems to take it as a dayjob.
My blog genuinely is a day job now, I make enough from the (unobtrusive, cookie-free) sponsorship banner that I no longer regret not having a proper software engineering job.
Isn't time for you to give us a break and stop posting in all threads then? It's clearly spamming.
Some of us really like simonw's analysis, personality, and transparency. Just because you don't like it doesn't make it "spam"
He plainly stated that he's spamming HN. The fact that you like the content doesn't change that
link? I don't see it on this page but it's a big thread and may have missed it.
> My blog genuinely is a day job now, I make enough from the (unobtrusive, cookie-free) sponsorship banner
Posting his links and engaging in the discussions is his main job. This is spam, even if you like it - because most people don't want to see one person posting their links on every AI discussion, ranked at the top just because among hundreds of thousands of people reading HN, some thousands are not annoyed by seeing his blog links in every post.
Does he provide content of value? If so, it ain't spam.
That's not how it works (for most people).
Lots of brilliant HN users are moderated when they post 1/100 of the links that simonw is allowed to post.
The corect term is being a tech influencer and even if you say it with a bad connotation it's completely different than spamming which is low/no effort posting and most of the times selling something.
If having a job makes you unable to use basic English, then it's time to take a break like he does maybe.
Thinking pelicans on bicycles posts are high effort/quality tells more about you. They are spam.
Not sure if you were joking about the English part with so many typos.
If you think all he does is post pelicans on bicycles then you aren't actually reading or seeing as many of his posts as you think you do. There's quite a bit more to it than that.
Also for others, the pelicans on a bicycle posts are comparing results between models using a benchmark. They're not just random pelicans for the art value.
No one said his post was high effort/quality but please reply again and with your amazing language skills twist my words to double down on petty points like that.
Who are "us"? Maybe you can set up a simonw-free forum and bring your like minded over there?
I see you're being blammed. "They hated him because he spoke the truth." Just because this online character is from your alma mater website doesn't mean he's not cynically and shamelessly self-promoting.
It's his job, and this is advertising (he is very good at native advertising on HN).
Yes, this. It's annoying and the moderation should stop giving him a free pass
What do you think about Grok 4.5 in comparison to Muse Spark 1.1?
Maybe Zuck should double down on his "spoiler" role with models rather than compete head-to-head.
He doesn't have to match Anthropic or OpenAI model revenue if he can deflate theirs by 99%.
All he has to do is keep spending a few billion dollars developing frontier models, release them as open weights, and turn coding models into a commodity. He also needs a good OSS reference harness to match. Very few people are in a position to do this and for it to make business sense.
That's quite likely where things are headed regardless, and he could speed it up significantly.
We should all hope models move from proprietary products to commodities the way compilers did.
This may be one of the best things Zuck could do for the world.
> keep spending a few billion dollars developing frontier models, release them as open weights, and turn coding models into a commodity
the muse family is no longer open-source. it is still priced very "cheaply", but a significant shift in approach.
If he deflates their revenues, who is going to rent the compute from Meta?
The goal is not for meta to take their market, the goal would be for meta to damage their competitors.
If meta releases an open-weight LLM that is not Chinese made, cheaper to run than the SOTA premiums, etc, it would lower the number of people paying for frontier labs models. We saw with with early LLAMA models, but they didn’t keep up in the race with v4.
Meta would benefit from this, not from increased revenue at the hands of open LLMs, but from reduced competition. Meta competes with Google for ad spend, and lowering the Google revenue (or increasing costs) from AI reduces the competitive advantage. OpenAI wants to build an ad engine, so same thing will apply there too - make it less-revenue-generating to compete. Meanwhile G, OpenAI, and Anthropic are huge talent sinks that they have to compete with, especially for ML talent which is core to Metas business goals (ads). Finally, Meta needs lots of GPUs to train their ad engine models. By reducing the revenue-per-GPU of these labs, they’re reducing demand on a core revenue generating supply they have to compete for.
The problem is that Meta jumped headfirst into the circular financing of datacenter build out, so they're just as screwed if the infrastructure becomes less valuable.
If it's good enough, they don't need to sell to Anthropic/OpenAI.
all he has to do is that prove builing these inst hard anymore. because the whole moat these companies have is the perception that building models at frontier is really hard .
Hitherto he's pretty much proved the opposite.
I guess we'll see how Meta did this time.
they've failed at every hard thing they tried to do. so if meta did it then it must be easy.
Coding models are not the destination. Coding models are just part of the bootstrapping process towards general intelligence.
Software has several unique properties on both ends of its production process that make assertions of progress based on the software use case invalid.
Software is easy to define as “working”: just run it. But - useful software requires an absolute truck worth of code - 100k loc before you’re talking about a real product, or else dozens of iterations of a toy you make for yourself before it’s useful enough to quit toying with and just use for what you wanted it for.
Sure, the success of software is hard to anticipate and what “good” UX is is hard to pin down - that’s not what I’m talking about. I’m talking just making the code and having no lint errors. That shit is a slog but it’s a slog with a clear goal amenable to hill climbing.
Through that lens software is mostly pattern matching. It’s very rare that an activity in software construction is out of distribution because even if the core of the thing is novel it needs a massive blanket of UI and a tech stack and a production environment to run in and observability and and and and. You get it I hope.
Meanwhile most work out there is a mess of undocumented, un-codifiable detail with no objective criteria for success, only a very wide gradient of “job well done” to “what is this garbage go and fix it”.
We are solving the easy parts of software and soon all that’s left will be the parts that are just like other work. And then we engineers will also be doing mostly squishy subjective judgment stuff.
Commoditize your complements.
He tried that with llama?
the way he could really be the spoiler king is to release an their training dataset to open source… doubt he’d go that far.
if they release training dataset, they will be in trouble for copyright reasons
The pricing is insane: $1.25/$4.5 for 1M tokens, and $0.15 for cached input!
https://dev.meta.ai/docs/getting-started/pricing-rate-limits
Yeah, this is most directly comparable to xAI Grok 4.5. In both cases, directionally "opus level intelligence for haiku prices" which is a really big deal for application developers who want to include models like this in their applications. I have been testing switching out haiku and sonnet for Grok 4.5, and may give this a try too (it is quite a bit cheaper, particularly for cached).
> Yeah, this is most directly comparable to xAI Grok 4.5.
Grok 4.5 has a relatively high $0.50 per 1M cached input token rate, compared to $0.15 on this model.
Grok 4.5 cached input costs the same as Opus 4.8 cached input, which is going to make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
> ... make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
There's a further sting in the tail, Grok 4.5 is only $2/$6 for the first 200k of context. Go above that, and the pricing is $6 / $12 - and you're still capped at only 500k context anyway.
Here's the xAI pricing on OpenRouter:
https://openrouter.ai/x-ai/grok-4.5?endpoint=0e927811-b1a8-4...
Cheaper than Qwen 3.7 Max. Second indication, after Grok 4.5 ($2 in / $6 out), that the BigLabs are feeling the GLM 5.2 heat.
[flagged]
The cached input pricing is a good ratio.
Compare with Grok 4.5 which came out at $2/$6 but then quietly charges $0.50 per 1M cached input tokens. That's as high as Opus 4.8!
Meta isn’t right now on the radar for most folks picking models.
If they have a really good model, it makes sense to subsidise it, to gain users, before they align prices with competitors.
this is not subsidizing. this is way too expensive for a no-name model.
Depends on the quality
just played around, it is pretty low quality. lower than sonnet.
is that a lot or a little
This is still ridiculously expensive imagine having to pay $10 for 100 search results on Google, thats essentially what this is.
I really dont see how anyone's willing spend more than $1.50 per mm output. Let alone $15-50. Does anyone actually pay for usage based billing as a consumer?
This is pretty cheap compared to anthropic opus and fable.
https://platform.claude.com/docs/en/about-claude/pricing
Model Base Input Tokens 5m Cache Writes 1h Cache Writes Cache Hits & Refreshes Output Tokens
Claude Fable 5 $10 / MTok $12.50 / MTok $20 / MTok $1 / MTok $50 / MTok
Claude Opus 4.8 $5 / MTok $6.25 / MTok $10 / MTok $0.50 / MTok $25 / MTok
Note Fable costs $50 MTok and Opus 4.8 costs $25 / MTok.
Yeah, DeepSeek V4 (Flash and Pro) is below $0.004 for 1M cache hits.
Even with usage based billing I'm below $1 writing code all day.
> I'm below $1 writing code all day.
same, pi as a harness works best, better than claude code and open code.
What harness do you use to get that? Reasonix?
Pi
Yes, this is the only pricing that seems sustainable
Sometimes. It depends on the task. $15/Mtok is still a lot cheaper than human written code. It’s probably worth spending more for contract reviews. Tasks don’t have uniform value. If you’re using an AI for entertainment, then frontier premiums are hard to justify. For paid work, they’re a great deal if used reasonably.
Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead and now with xAI and Meta at least delivered something that's competitive with useful models and cheap too. Granted, the narrative that the two leading labs are ahead still holds with Fable (and perhaps an upcoming GPT6), but it's not as over as common knowledge by the opinion leaders would have us believe.
> Interesting how the prevalent opinion until yesterday seems to have been that OpenAI & Anthropic are irreversibly ahead
Not the way you're implying?
The GLM 5.2 hype was blowing way before this. Neither xAI nor Meta have really made a difference in a different way - similar results / similar pricing (to GLM 5.2).
Think it’s less a shift in who’s ahead and instead more people acknowledging that making yet another dashboard app worked fine on last years SOTA and doesn’t really need today’s SOTA
People misinterpreted Google being behind as Anthropic and OpenAi being really ahead, when it was really just Google falling behind the same way it did with Tensorflow, Angular and GCP.
> when it was really just Google falling behind the same way it did with Tensorflow, Angular and GCP
Not sure I agree. Angular fell behind in popularity but was (is? unsure atm) still eminently usable. I gave gemini a test drive recently and it was horrendous, as in "picking dirt cheap Chinese model over gemini any day" bad, and with overzealous guardrails to boot. 3.1 pro feels a year behind and is extremely lazy. 3.5 flash feels like a model you’d run on your 128gb macbook, not something that was released a month ago and which costs a fair bit when used through api.
In any case: as of right now I think that we went from a three horse race to anthropic / openai as premium choices vs whatever is the Chinese fotm for a fraction of the cost. 3.5 pro better be a miracle if google wants to hang out with the big boys, otherwise their only strategy is hoping that both US labs go broke and they remain the last man standing.
Gemini is more than fine as a mobile application, and will be the “brains” of the currently braindead Siri, between than and Android it’s hard to come up with an argument they are behind.
Likewise, Gemma4 models are unbeatable at their size.
So I use Gemini for coding? Hell no, but that’s not the same as Google failing writ large.
Googles only real goal is to retain the ownership and dominance of the online world they have now, and Gemini is doing exactly what it needs to do
I dont think gemini needs to be opus level though... if its there to power google search then the scale for google to operate at is enormous compared to openai/anthropic.
meh, models are mostly good enough now. Without a major breakthrough the only thing that matters now is cost.
For your applications. For navigating and managing a world that lazed on a "good enough" policy, we still need the Real Thing.
Real Thing is going to come from a major breakthrough. I’m talking about competition amongst models. Difference between Fable and GPT Sol Ultra is so minor the only thing that really matters is that GPT is cheaper.
Hell, difference between Opus and Fable isn’t that big. I would still prefer Opus for most task because it’s cheaper. Break things down into smaller tasks and create deterministic parts of the process and you need the more expensive models less and less.
Anthropic needs to get their token efficiency under control or they will lose long term.
I personally do not like Meta, but I'll say this. The more competition, the better for regular consumers. (Enterprise too)
- Chinese models
- Grok
- Meta
- Google
- OpenAI
- Anthropic
I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.
Meta's local llama models used to be the face of open source AI. The scene has really changed.
they likely got the Peter Theil newsletter proclaiming open source models are the antichrist
That person is Alexandr Wang. He made his money selling data annotation services to closed source companies: openai, anthropic, Google, even Meta
Only 666B models. Other sizes are fine.
Yeah, I think it is definitely great. Having said that, I am still debating in my mind whether the volume of software engineers needed in the AI era is going to increase or decrease because of all of these advancements.
On the one hand, because it is easy to build products, more and more people will build. And more and more products and features will be built. However, a lot of people who are non-technical will also try to build, but they get stuck, and then they will need engineers. The sheer volume of product built by both experienced technical companies and non-technical novice startups and founders and wannabe founders is going to be massive. That is the bull case for having more software engineers needed in the near future.
On the other hand, in a year or so, people will build all these products, and most of them won't be able to market them, sell them and make money. Eventually, there won't really be a need for that many software engineers.
I think overall the bull case is probably going to win net net.
I see some similarities to 3D printing here. It’s great that everyone can make their own toothbrush holder (or whatever) but I’m probably not going to pay for someone’s weekend project.
I’m “seeing” more devs stepping into the SendCutSend stage where they’re cleaning up/fixing/productizing vibe coded projects so maybe there will be some new demand in that space?
3D printing is a good comparison - it allows almost anyone to make things, but in the end very few do.
Another example is when the WWW first became available, and suddenly everyone COULD be a publisher (browsers even included built-in HTML editors), and for a while MySpace pages proliferated until the excitement died down and people went back to being media consumers.
I expect we'll see the same thing with consumer use of generative AI. Suddendly everyone is generating 3-D worlds/games with Fable because they can, but I expect that just as with the web the novelty will wear off and they'll leave it up to the pros.
Professional use of GenAI, and coding in particular, is certainly here to stay, but it seems we're still in the early experimental/hype phase. At least tokenmaxxing has passed, and it seems most companies are now paying attention to, and limiting, how much they are spending, but it doesn't seem we've yet progressed to the stage where companies are paying attention to what they are actually getting out of it - is the money spent showing up on the bottom line in the form of increased revenues.
A comparison I find useful here is Excel (and spreadsheets in general). Those enabled huge numbers of non-programmers to build software-like things, while the demand for expert developers grew enormously at the same time.
I'm hoping vibe-coding plays out the same way.
It’s terrible and depressing work to take vibe coded garbage and make it a real product. There will be demand, but good engineers won’t want to touch it. And people paying will think they did the hard work so why pay a good rate?
I think LLMs/coding agents in particular are going to play out like automation always does.
1. You have a task and it requires an expert to perform it (software engineering, where we were) 2. You automate the task, it still requires the expert to babysit it (where we are now) 3. Management works out that the expert is just monitoring what the automation is doing and has actually lost expertise because they are just watching, not doing. Pay collapses. (where we are going)
Source: https://en.wikipedia.org/wiki/Ironies_of_Automation
At least in China a lot of software developers are now struggling.
I think for a lot of type of software we have now reached peak employment.
Someone payed a few k just for a normal website.
> At least in China a lot of software developers are now struggling.
Do you think that Chinese software industry is that relevant to the kind of software market talked about on HN? I.e. lots of enterprise b2b and infra companies.
Chinese companies have always had a very low willingness to pay for software which kinda breaks the flywheel of B2B SaaS companies and companies to service those companies all the way down.
> Chinese companies have always had a very low willingness to pay for software
Are we still left with this mindset? Maybe once upon a time but it has definitely been changing.
There's plenty of B2B and enterprise SaaS companies in China serving the Chinese market. Maybe not as many, but no longer the very low of the past.
I also would not say enterprise were not willing to pay, even many years ago. It's the SME that refused to pay. Large CRM, ERPs etc have always existed.
I would say I'm still in this mindset. Numbers are hard to come by but analysts optimistically put Chinese SaaS market at ~10% of the size of the US. Also I see that the gross margin for public Chinese SaaS is around 50% vs 80% for the US reflecting that SaaS in China is much more services and implementation heavy. So it feels like the direct to business SaaS's like Salesforce aren't really there and then then the selling to SaaS titans like Datadog have a much smaller flywheel to work in.
Happy to be wrong though if I'm missing something.
> I would say I'm still in this mindset. Numbers are hard to come by but analysts optimistically put Chinese SaaS market at ~10% of the size of the US.
Exactly it's mindset. I used pulled up 1st result when I used Google [0][1] and it is ~20%. It also forecasts huge growth potential.
[0]: https://www.grandviewresearch.com/horizon/outlook/software-a...
[1]: https://www.grandviewresearch.com/horizon/outlook/software-a...
> reflecting that SaaS in China is much more services and implementation heavy
I wouldn't read into it too much. There's creative accounting and other things involved. Also what is services and not e.g. "premium support"?
> So it feels like the direct to business SaaS's like Salesforce aren't really there and then then the selling to SaaS titans like Datadog have a much smaller flywheel to work in.
US companies struggle for many other reasons that I rather not go into but definitely not a SaaS nor China issue. I mean yes it is a China issue as in businesses need to adapt to different cultures rather than 1 size fit all but that applies everywhere.
Its a signal. They were earning well and AI crashed the market in China.
China is in a very different economic state to the West (broadly construed).
They have had real issues with deflation rather than the inflation most Western countries have seen over the past five years.
> On the one hand, because it is easy to build products, more and more people will build.
And those people won't need to be software engineers.
> but they get stuck, and then they will need engineers
You've implicitly assumed here that the AI systems will always be worse than the average engineer. That is IMO myopic. I'm not sure that it's even true now let alone in the nebulous future.
> And those people won't need to be software engineers....You've implicitly assumed here that the AI systems will always be worse than the average engineer.
Most of what we do as engineers is precisely describe or analyze the behavior we want or the behavior we don't want. All other engineering skills that are useful are ultimately downstream from understanding the behavior of software enough to know which parts to keep, improve, or jettison. Chatbots can take care, somewhat, of analysis or expansion of instructions.... but they can't read minds. I don't see that changing any time soon.
> but they can't read minds
I don't know who needs to hear this, but neither can humans.
You've implicitly assumed here that AI systems will always be worse at contextualizing and framing questions than the average engineer. I'm not sure that it's even true now let alone in the nebulous future.
You haven't narrowed the fundamental myopia of the assumption here, just dressed it in slightly different clothing.
> You've implicitly assumed here that AI systems will always be worse at contextualizing and framing questions than the average engineer.
How would they know what to ask or contextualize if they don't know what the user wants?
Are you suggesting that psychic mindreading powers are real?
> How would they know
How would you? The answer is the same.
I don't understand what you mean. I can't build software I can't describe.
If you're implying chatbots can ask their "client" what to build, good luck with that—contractors are at least liable for what they produce and have extreme incentives to ensure that their clients are happy. To the extent of refusing to build anything if they don't know what they want....
> I don't understand what you mean.
If you had psychic mindreading powers you would understand what I mean.
By asking the user to explain what they want whenever there's ambiguity.
Plus all the other things that software engineers generally have not learned to a professional level even if they picked up the basics on the job by osmosis, because figuring out the customer's needs (and what they'll pay you for which may be different) is the job of a business analyst, a PM, or a UX researcher, and those are different skills and two of them may come with a Business Informatics degree rather than a CompSci one.
LLMs can be "eh, better than nothing" at many things, not just code.
And when an LLM runs up costs for a small company by getting them to lease a bunch of infrastructure they don't need, who can they sue? A contractor or advisor you can't hold liable is just a liability.
Same person they'd sue if they used any other power tool themselves and it didn't work out right.
Plus, this is software "Engineering" we're talking about, which famously gets scare quotes in comparison to all the other forms of engineering because unlike them we don't have as standard things like professional liability insurance to cover serious professional errors of judgment the way someone who signs off on a bridge that collapses would have.
> And when an LLM runs up costs for a small company by getting them to lease a bunch of infrastructure they don't need, who can they sue?
This question is completely disconnected from reality. If you try to sue a human for proposing something more complex than what you need you will waste a lot of money and then lose the lawsuit.
Also the annual cost of too much small company infrastructure is less than the cost of even a single good human engineer.
The big thing to me is why are we even running these models on top of an operating system?
What I really want is Claude as a deep part of the operating system.
If that happens then a whole lot of the abstraction of software vanishes along with what we think of today as software jobs. I think many new forms of knowledge work would emerge from this though.
I would think that needs massive local compute but I can't imagine that is not the future down the line.
It’s also not the future SV is incentivized to build. They want everything for rent, nothing can be owned.
Luckily, China is on the verge of a true breakout, I’m not sure what exactly it will be - but I’d make a very large wager the “next iPhone” is Chinese, and will constitute a full blown “Sputnik moment” for the US and SV.
If Americans weren’t forbidden to own Chinese EVs they’d know this. But tariffs mean the breakthrough will be even more unexpected.
Since Chinese actually “sell stuff” I’m guessing their unbeatable lead in AI efficiency, manufacturing, and distribution will produce a step change breakthrough within a decade.
While data centers are still using lots of energy created from fossil fuels and many still evaporate water for cooling?
No wonder we still can’t get climate change under control
> No wonder we still can’t get climate change under control
This is was historically a money issue, being green used to be wildly more expensive.
Now being green is cheaper, the limiting factor is how fast PV and batteries can be made or imported.
Recent reports of the sum of all US data centres currently in planning, has a power demand exceeding the (capacity-factor-adjusted!) global annual supply of new PV.
This would be less of a problem, but still a problem, if Trump wasn't trying to get in the way of anything green, or if the companies building data centres decided to also support factories to make more PV.
* Planned new demand: 300 GW; PV factory capacity ~ 600 GW nameplate, but the capacity factor is 14% so that's really 84 GW on average.
To expand on Chinese models:
- DeepSeek
- GLM (Z.ai)
- Minimax
- Kimi (Moonshot)
- Hy3 (Tencent)
- Qwen (Alibaba)
(Each one of these with weights available to download and run locally)
- MiMo (Xiaomi)
GLM 5.2 is great, but is so rate limited now I no longer recommend it
I'm looking ahead to the next wave of open-weight models that are as efficient as DSv4 (which is really efficient), and have been heavily distilled on GLM 5.2 (which is trivial, given it is open weight)
Aren't there multiple providers for it? is it rate limited in all providers?
I have no issues via openrouter.
I am using Z.AI coding plan, but will give OR another shot!
I use it all the time through Fireworks. The normal version when I pay it myself and the fast one when company pays. It's really fast and I never get rate limited with my daily use.
Rumors are Nvidia H200s got approved so infrastructure might be improving soon.
He came to X to post about this instead of his very own meta threads. This just shows how much interested he is to make this thing big, and of course, the cost can stay bearable for us considering all of these cash burn that these companies are doing
Its the biggest technology race we have ever seen. Richest companies, smartest people, richest countries.
I do not know if competition is good, we will see in a few years.
Looking forward having a physical job for a change :D
A bit much describing our tech leadership as smartest people we've ever seen.
I would call the founders of DeepMind (Demis Hassabis, Mustafa Suleyman, Shane Legg) very smart people. Im pretty sure with the amount of funding everyone of these companies have, they have a long list of very smart researchers in their companies.
I do not mean Suckerberg or Eric Schmidt.
Greediest, perhaps?
How is every company able to show itself at the top of every benchmark?
First look what models are worse in a set of self selected benchmarks.
Second, compare to older versions of competitor s models.
Still does not look good? Compare to own previous models.
Not much moat, incremental improvements, cherry picking models to compare.
To be fair, seems more correct to compare against similar strength models if your main edge is pricing.
At this point comparing to Gemini is a free Bingo space.
Wait to the exact moment your model is ahead on at least N benchmarks then publish.
Anyone deep in the AI realm know which is the gold standard benchmark for coding?
It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
[0] https://deepswe.datacurve.ai
[1] https://cognition.com/blog/frontier-code
[2] https://xcancel.com/i/article/2064210146558136827
They're being called "trust me bro benchmarks" for a reason ( ･ั ﹏ ･ั )
This not being available on Openrouter really makes it hard to test. I was going to compare vs Grok 4.5 and GPT-5.6 Luna, but I don't want to deal with signing up for Meta for it unless it checks out. Please Meta make this available.
Where is the data retention policy information for paid API per-token uses? Every other provider has one and makes it clear how they handle your data. A quick look doesn’t show one for this new offering.
Finally found it: https://dev.meta.ai/legal/
Paid prompts are not used for training. Data retention is not entirely clear, they will keep it if they think you're breaking the rules, but it's not clear how long they keep it if they don't think you're breaking the rules.
Just got it working with codex in a container! FYI I think there is a bug most others will run into at the Codex:Muse interface.
It's some kind of parsing or integration error due to what I think is codex not anticipating server-side tool calling and how meta treats those ids... first couple times running codex with muse, it would fail on its first non-web search call.
Got it fixed, not personally sold on the bespoke server-side tool calling and indefinite file storage yet, but also a very cool model that I'm enjoying using so far!
https://github.com/accretional/awesome-muse-spark/blob/main/...
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.
What kind of use case would be best for that shape?
Debugging and diagnosis is very tool call heavy, whether that's grepping / transforming logs, calling out to profilers/tracers, or even just writing up incident reports.
Bug diagnostics is about being okay at coding but better at tooling.
Given a good diagnostic report, it can be handed to opus for the fix.
Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
I wonder if we'll start to see that pattern with every new release. Tool use likely changes rapidly, so the newest, rather than most intelligent, model may always have an edge.
What you mean.. The tools are all just invoking bash and terminal/cli cmds and http requests. Paradigms that have existed and stayed mostly unchanged for decades.
These do make up a huge % of tool calls, but I don't think these make up a huge % of tool call failures.
I see models fail on tool calls that involve API requests to a specific API, internal or cloned Makefile calls, npm run commands, etc.
Gemini 3.5 flash is better than fable at tool calling. Tool calling is probably one of the easier things to do post training for.
I don't use Gemini cli because they're so bad at agentic work/tool calling. I use their chatbot all the time, though.
This sounds... kind of useless? Really good JSON or similar constrained decoder performance is interesting, but normal decoder > tool validator loop with good error message > tool retry is almost always able to get a tool to work second try, and input is cached so it's not expensive.
Things are not always that simple, eg https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
The avg coding session has hundreds or thousands of tool calls. Even a 5% failure rate noticeably notches up token use and cost. See Gemini.
Yes, but each tool call has a different failure %. The tool calls that make up the majority of volume like grep are going to have nowhere near a 5% failure. A custom user-defined skill having a 5% failure rate is probably fine.
Since it was making rounds yesterday here is what muse generated for the 3d rubik's cube prompt - https://ashish01.github.io/rubik-muse/
Hold my beer...
https://chat.z.ai/space/t19sx5kvw631-art
Yeah, no thanks. I cannot think of a worse company to trust with additional personal data.
Me neither, though LLMs also provide services that don’t involve personal or sensitive data
Exactly. I have zero trust in them, especially given their history of shady practices - like buying a VPN service (Onavo) and using it to track how people used competing apps.
Interesting that neither meta nor xai chose to do open source given that they are both clearly behind Google, OpenAI and anthropic - and a serious us open source offering would give them a clear foothold.
I suspect they have a brand problem from their social media ties and shady histories. I personally will never use their models, plenty of better alternatives. I'm now exclusively on open weight models
Open source would make them an instant credible leader, major fumble (still can be fixed)
Why are the plans and pricing for all these products so complicated.
I don't know where I need to sign up to try it out. What is pricing? Is it API or subscription, what?
I had the exact same experience with Grok 4.5 as well.
Nearly every model can be found on OpenRouter and used with a single key. Meta Spark is not among them, but Grok and almost every other model is. That's how I try models I don't already have an account for.
It seems to trade blows with GPT 5.5 and Opus 4.8 in performance while being cheaper than GLM 5.2.
I missed the fact that Meta was developing and releasing closed-weights models... bummer. Would be great to see some more progress with American open-weights models.
> Model API is not available in your region.
:(
Well, Vietnam is not in the list of restricted territories.
Anyway, what is "your region" ?
Is this where I am now, or is it where I activated my Oculus 2 five years ago ?
Same in Argentina. It's almost surely a region whitelist for now (it's the only reason Argentina ever gets blocked).
Can’t you just use VPN?
Your Meta account profile region has to be in the U.S., and they do Geo IP and SMS verification.
why?
Same in Canada. lol
Perhaps on brand for Meta, there appears to be no mention anywhere on this page or any of the docs pages, of any data retention policy. Maybe this is hidden in an agreement you need to sign when making an account? Has anyone actually done that?
Glad to see Meta back on track! Users will benefit greatly from this competition.
Very strong pricing, cheaper than Grok 4.5, particularly the cached reads. We'll have to wait to see if it's actually worth using (it's not on OpenRouter yet).
That's what one does when its product and public perception is way behind competitors.
How are people trying this? I don't see it on openrouter. Any ways of testing this without subscribing to meta stuff?
Probably need to wait some hours/1-2 days and openrouter will add it.
Thanks. I was asking because I couldn't find even their previous 1.0 model there.
Good to see Meta finally back to releasing something at least worth evaluating. And it sounds like they did at least a bit skate to where the puck is going by focusing on tool and computer use.
Despite Muse being relatively average, I've actually used the Meta AI webchat LLM since it released.
The reason: Its writing style feels "unique", and I find it pleasant to read for science-based topics.
I never ask _ONLY_ Meta AI, but the answer it gives is almost always in a distinctly different style than other frontier LLM's.
I think this is because of the unique JEPA architecture they have, but that's a layman's hunch.
I think it does not use JEPA.
Considering the DeepSWE result (imho if you're gonna give value to benchmarks this is one of the best) it's not good enough.
It's a high quality benchmark for sure, but it being public means it's at risk of leaking into the models (unintentionally or not), right? For that reason I prefer to look at the private ones, like: HLE, SimpleBench, Kagi, ARC-AGI.
A lot of these benchmarks are unfamiliar. Are labs just choosing the ones that make them look best?
Haha their demo is AI spamming restaurants on Instagram. This is going to go really well.
Not open source, so not relevant. The only ones that can be relevant while being closed source are anthropic and openai. Not even Google except for search, not even spacexai. People use chatgpt more than Gemini for the average person who is not searching.
I'm still confused is it available to public via some sort of subscription?
Is building models on the frontier really easy now or something?
My trust factor is gone with Meta right now. Has there been any independent analysis to confirm they didn't cheat on benchmarks again?
Can't we use it even if we don't trust it?
I don't see a reason to use something that is, at best, about as good as models that are produced by labs who don't have a reputation of cheating.
They cheated again: https://news.ycombinator.com/item?id=48847019
No they didn't. Please read: https://www.anthropic.com/engineering/infrastructure-noise
Thanks for the read. It seems to confirm that resource limits are an important factor for terminal benchmarks:
> The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.
> An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.
So changing the resource limits changes the benchmark. Yet their score table claims their score to be for Terminal-Bench 2.1, not Terminal-Bench 2.1 with raised limits.
This is not open-weights, right?
Correct
Competition for cheaper and efficient models is a good thing, regardless of if you don't like SpaceX, Meta, etc. Especially from US based labs
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
There are US companies hosting open weight models for enterprise, we just enabled Fireworks.ai for the devs
Is this the model trained on Meta "draftees"? Are we seeing this in the jump on JobBench?
Meta is back in the game, albeit not at the top. Impressive stuff, nonetheless.
Weren't they caught multiple times gaming the benchmark even more so then the rest?
Yes and Zuck effectively disbanded the entire team that did that. Not saying we shouldn't cast a critical eye on it, but it probably does warrant a second chance.
Zuck was part of that team.
Let me assure you, literally everybody does this
I don't think it even matters. Because noone will continue to use an LLM that doesn't work well for them, whether or not it has a good bench result. So for their own sake, the correct representation can actually win them some loyalty:
eg. Model X is weaker than Fable, but competes well with Opus/Sonnet and costs 1/5th as much etc - something similar playing out with Grok 4.5.
They are not open source anymore, right?
Everyone has been loving to shit on the Alexander Wang acquisition but this seems legitimately impressive to me?
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
Not their first try. There’s been reporting about how they’ve kept pushing their model releases back because of underwhelming performance.
... i dont think internal iteration counts dude. thats just called in-development.
How is it their first try? They were leading the race with Llama 3.x a few years ago.
As far as i remember, the entire AI org was essentially gutted and replaced with whoever Wang wanted to hire, and tbh that org completely failed to train llama 4 and I honestly doubt whatever techniques they used to ship llama 3 are at all relevant now. That was before reasoning models and the heavy emphasis on RL/post-training.
so yeah, this is essentially their first try with a completely new org.
I agree with parent, Meta has been at this a long time and its only because they have recently fallen off that they pushed this "oh give us credit its really a new org" thing. Basically, if you can't actually "win" then try to fake a restart and say we are the fastest.
Even given that, this is their second try (they had Spark 1.0). Spark 1.0 was uninteresting, this is potentially interesting, but we can't really try it yet it seems (at least not in Openrouter).
Ultimately, competition is now fierce in this broad level of intelligence/cost: Spark 1.1, Grok 4.5, GPT 5.6 Luna, GLM 5.2
Sonnet not in the same ballpark of pricing (more expensive than Opus in many cases). Haiku has been basically abandoned.
They were leading the race in a niche category a few years ago. Now they are, according to some benchmarks, even on the right playing field.
Not opensource.
Tried to get access to the API, apparently the model API is not available in my region...
I have questions regarding if I should even care but I don't so Meta please keep enjoying the irrelevance. lmao
Right, amazing because for me also... "My region" being Canada.
I'm going to assume the only "region" that's permitted is the USA.
haha no thx
It's great that we have yet another competing models. The more models we have, the less likely we are subject to the ideologies and the controls thereof by the cults like Anthropic. And of course, it drives down the cost of tokens.
[dead]
Still no weights? Useless