If you haven’t been really running and testing these models yourself, they are all benchmaxxed. No matter how close they score to frontier on whatever metric, they always fall apart in real world tasks and their token efficiency is ridiculously bad. Fireworks has incredible incentive to make this claim in a headline, because Fireworks hosting K3 for you is pure profit for them, unlike when they host closed source models.
Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.
However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/
I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.
In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.
Yeah, open weights hasn't fully caught up yet, but it's getting very close. And it's certainly passed the point of being reasonably interchangeable with the frontier for (programming) work. Add in the benefits of not being rug-pulled by the frontier labs silently messing with, the knobs on their models or outright denying you the ability to do certain kinds of work (c.f. the HuggingFace fiasco), and they probably come out ahead in several respects.
To your slow point: I did a quick eval for a data viz task to do a qualitative comparison and Fable was much faster (by 4x), but I struggle with the "token inefficiency" as a sort of whatever metric.
Kimi was half the cost and produced a near identical output.
In my limited anecdotal experience, Kimi K3 is a bit better than Opus 4.8 and Qwen3.8 Max is disastrously bad. It can reason fine, but the moment it tries to do something it gets stuck into long second-guessing loops with no progress. It sometimes refuses to try and debug a problem even after multiple suggestions. I'm sticking to K3.
with all the hype around kimi i actually renewed my subscription (last time i tried it out was on 2.5 release) - just the 40€ one.
Its hard to compare that one to my regular sub of anthropic (107€), but everyone always says how much cheaper it is so i thought it was worth a try.
1.5 days later ive went through more then 50% of my weekly usage and decided to renew the regular anthropic sub too.
The API Pricing is definitely cheaper, but anthropics subscription budget seems to equalize that advantage right now. also kimi code feels like claude code from ... september 2025
also - considering how they announced they'd temporarily close subscriptions to make sure they can service customers... i was slightly surprised that there was no issue subscribing, less then 1 day after that announcement.
Makes you think if that was just a marketing stunt
Could you check how many tokens the different models spent to do the task? With all the comments about Kimi K3 Not being token efficient I'm curious if your test confirms that
I love Kimi for some things, but K3 to me struggles with things Opus 4.8 breezes through. Admittedly stuff that is on the more complex side (code generation bugs in a compiler) to the point that after two days of struggle I stopped Kimi and will have Opus redo its work once I have spare tokens...
This wasn't just being slow - it didn't make forward progress.
For simpler stuff even 2.7 does just fine, though.
Honestly my anecdotal experience is that fable is benchmaxxed. I have not observed useful gains for Claude since 4.6, with each model iteration making progressively poorer decisions in pursuit of its goal.
The 5.5/5.6 series has performed quite well however. My guess is that my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.
> my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.
What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?
yes, above 50% I don't observe a significant improvement, the models scoring above 65% like opus4.8 seem worse than gpt5.5/opus4.6. DeepSWE appears to align more closely to my personal experience.
Fable works very well for me on a moderately large codebase. I have had to correct it a few times or point it on the right track, but given how much faster it is at programming than I am that's a very minor issue (and most of these errors are because I underspecified what I wanted in the prompt, I can only think of two cases where it was genuinely wrong... that's a lot better than me in my professional career). Code quality is equal to what I would come up with (and better in areas I'm not familiar with) and the overall software engineering bar is higher because it doesn't get bored when I tell it to do refactors or write integration/regression tests that I would otherwise put off. Also makes it easy to audit code for things like missing audit logging or error notifications that a human would get bored doing.
The product is a fairly standard Ruby on Rails webapp with postgres as the DB. Application complexity is probably a bit higher than average for a webapp. So it's nothing that pushes the boundaries of software engineering, but it is a real product. Token budget has not been an issue for me. I pay for the Max plan ($200/month) and it is well worth it.
Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other.
We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.
I have been testing the various models, and I would not call Fable SOTA.
I can't actually get Fable to do anything. I only work on back-end code, and the moment Fable notices the jwt scope checks on the endpoints it's game over, it refuses to do anything because security is involved.
So for me, Fable is completely useless, the bar is very low, any llm that will actually attempt the task beats it every time.
All models are benchmaxxed, period. ”Jagged frontier” is the euphemism du jour, I believe?
Anthropic/OpenAI were touting PhD-level intelligence three years ago. And they’re still shipping models that aren’t smart enough to realize things such as the need to drive the car to the car wash (because they hadn’t yet hill-climbed that particular brain-teaser).
didn't read the article, but just the title felt fishy because of course an inference provider wouldn't mind claiming to provide access to a mythos-class model. So far, GLM 5.2 is still my go-to, even at less than a third the parameter count compared to Kimi K3
It's telling that in their release post, Moonshot themselves said that K3 lags Fable and GPT-5.6 in "user experience". I took that to mean the stuff you can't push directly via RL, what some people call "big model smell".
"Despite being a highly competitive model overall, K3 nonetheless exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT 5.6 Sol."
There are some benchmarks I personally trust. But the most reliable indicator other than public benchmarks is the quality of products people are working on, and the model they use for the work. Not a benchmark scoreboard or a one~few shot demo, but the product they have been building for weeks. The reality is, even the diehard advocates who build and sell tooling for open weights, still use proprietary frontier models for their jobs.
Clearly the best way to test these is a completely scientific and controlled process of prompting for a creature on a bicycle. Much like testing the fitness of a fish by making it climb a tree, the LLM will show its true colors when presented with the impossible task of rendering an image.
Totally agree with all of this. Most of the problems we solve with AI are not one shot, all the bench marks also miss the human components when testing. Thinking alongside with a human and coming to a solution is what Fable does better than most other models.
I encountered the same issue. Kimi 2.7 looked impressive on paper, but in practice, the code was so riddled with errors that I ended up using GPT-5.5 to fix it.
I've seen this phrase repeated over and over again and I don't think it is true, or at least, to the degree that "benchmaxxing" claims are true, American frontier AI companies are probably about as guilty of it as Chinese AI companies are.
For both GLM 5.2 and Kimi K3 I feel the rough average of the benchmarks gives you a rough idea where they stand. GLM 5.2 was sitting somewhere behind Opus 4.8 but it didn't feel very far away. I've used Kimi K3 via OpenRouter and while I've had limited experience so far, it sure as shit feels like it's right up there with Sol and Fable to me. I am happily able to believe Fable has the edge still, but on a request by request basis it would be pretty easy to get an impression one way or another.
The existing Kimi models were already pretty good so I really just don't find this new model to be that hard to believe. Maybe I'm naive.
All you've said needs the qualifier - "for now!" Look at the trend line. It's clear that if they're not yet at the level of being "good enough" for coding, they will be soon. Sensationalist headlines aside, we all need to be preparing for a world where open models can do pretty much any software tasks you need them to.
Very interesting. They test Kimi K3 and Fable on a set of approx 1000 tasks grouped into 5 areas (SWE, Legal, etc).
They put a router model in front that predicts whether Kimi or Fable is going to give a better cost for a correct result. (They believe that ultimately such a router model should be continuously trained on your own workloads so it makes the best decisions for you).
Their router chose Kimi the majority of the time (72% in one category, all the way to 96% in another category), leading to cost savings in every category (from 1.5x to 50x depending).
> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).
Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.
I work for a router company too, I ran some tests on all of the cheapest models and came to the same outcome where a handful of small models ran together in conjunction outperform SoTA models -- outperforms in that it got a 95% vs a 94% and I bet that changes with the day of the week. Anyways, I did get a similar result in a different sort of measurement.
What is routing? How do they decide which is better? The only way to come up with a routing model for your workload is to send queries to all the models and then come up with a way to say which solution was better, often trying out multiple times for the same model + query to account for other statistical errors.
This makes you, an ai inference user an unwitting AI company with a non scalable product.
The biggest mental trap people have fallen for is the notion of “best” and, always using the frontier model. Instead you should just bite the bulet and choose the cheapest or the best. This routing dance is just tokenmaxxing in disguise.
Edit: Another thing concerning me is that the models themselves are not concrete behind their endpoints. Models can be arbitrarily dumbed down by reducing their inference resources. Once Anthro/OAI release a new great model, they are fully incentivised to dumb down their current models to drive traffic to the new shiny more expensive one. In fact they can do this for any reason. Once they switch things behind the API interface, how useful is your meticulously tuned router? Not much at all.
When DeepSeek was released, it had an immidiate and significant impact on the US stock-market. Now when its becoming common knowledge that China is almost at parity with US SOTA models with good momentum, why is there no sentiment change on the market?
Because markets in the short term are almost a random walk and making the blanket statement that “Walmart stock is down today because Deepseek” was an easy narrative to repeat for media people who cover the stock market.
In reality the world is a highly complex, chaotic, reflexive system, and saying “the entire market moved today because of 12,000,000 factors that randomly aligned” isn’t satisfying enough for people to follow your media channel so they can monetize your eyeballs.
It was never DeepSeek’s release that dropped the NASDAQ at the time; it was the unknown risks of the early thoughts related to trade wars. Popular financial newspapers can promote anything they want, but these news do not typically drive large investor decisions.
It might be because no interested party is using that piece of news to move market. They have enough with other news to do it. Just a practical matter (or may be something else, who knows!)
Back then people didn't understand how AI was run. It should have probably made Nvidia stock actually go up. I think the other Factor, and I might just be two into AI and most people are normies, the hype around Chinese models we've learned is overblown. United States models are a league above.
there was a significant impact on the market because the market assumed that a strong cheaper model would have a significant impact on the US AI companies' business. That didn't happen.
deepseek drop was based on assumption that we overestimated how much compute and infra is needed to train models. They seem to have claimed that they did it in couple of million or something.
I strictly prefer when models ignore any human quirks in my responses. Claude trying to be your friend, saying LOL to your jokes is ridiculous and frankly, harmful
Yeah. I've found that Opus by default outputs something I call "Claude-lang." It consists of oversimplified, grammatically incomplete sentences that I find painful to read.
Maybe it is something that is easy for it to read and write, but definitely not for humans.
For example,
Skim once now; refer back while reading Part II. \*Every bold technical term in Part II is defined here\* — treat these as a dictionary, not a reading assignment. The first table covers the vocabulary of the *deck*; the three that follow cover the *methodology* vocabulary introduced in Part II, grouped so you can find a term fast: \*(A)\* the logic of rules, \*(B)\* the neural-network & training machinery, \*(C)\* the method-design ideas.
JMRL is the paper the thesis instantiates, so this is the one to know cold. Its pitch is \*end-to-end\*: earlier rule methods (LogicRE, MILR) bolt a rule learner *onto a frozen* extractor in a pipeline and suffer \*error propagation\*; JMRL trains the rule module *jointly* with the extractor.
**Identity.** Conformal prediction for NER producing **finite-sample-valid prediction sets** at two granularities: **full-sequence** sets over entire label sequences (capturing contextual dependence — "if Sarah=PER then NYC likely LOC") and **subsequence-level** (per-span, **class-conditional**) sets; an **integrated** method filters full-sequence predictions with entity-level sets. Adds **covariate-stratified calibration** by **sentence length and language** for valid multilingual coverage, and studies **combined nonconformity scores** (Naive / Conditional / RAPS). **Read in this order.** Abstract → §1 contributions (full-sequence / subsequence / integrated / **covariate (length + language) calibration** / combined scores) → §2 CP recap (inductive split-CP) → §3 NER formulation (IOB2, CRF) → the subsequence / entity-level set construction + class-conditional coverage → the language-stratified calibration results. **Why it matters here.** The **span-level construction** for **Topic 11**'s per-triple score, and — crucially — its **language-stratified calibration is exactly the EN↔zh case**: it shows how to keep conformal coverage valid across languages of differing length/script. Complements PASC (pipeline-level joint coverage) with the *NER-internal* set construction. **Caveat.** A heavy statistics paper (44 pp., *Annals of Applied Statistics* submission) with CRF-based NER; the project needs only the **inductive split-CP + subsequence/entity-level sets + language-stratified calibration**, not the full-sequence machinery (likely overkill for triple-confidence). Assumes exchangeability — borderline under the EN→zh shift, which is precisely why the PASC/ConformalNER *shift* analyses matter.
> We may use Content to provide, maintain, develop, support, and improve the Services, comply with applicable law, enforce our terms and policies, and keep the Services safe and secure. Customer who requires restrictions on the use of Customer Content for training or improving Moonshot AI models may contact Moonshot AI to discuss available enterprise arrangements or separate written agreements. Unless otherwise expressly agreed in writing, Customer Content may be used for the foregoing purposes.
Notably, unlike Claude, there is not an opt-out option for the model training part. The TOS explicitly allows Kimi to train on your code.
Simplest way is to signup to OpenRouter and filter out all non ZDR (zero data retention) providers. Paying "API rates" however can prove expensive compared to coding plans (for instance, MiniMax $20/mo coding plan allows 1.7b tokens; depending on input/output/cache ratio, it is worth $200 to $500+), except for Hy3, DeepSeek v4 Pro, and MiMo v2.5 Pro (whose API rates are cheap and/or discounted already).
If you're looking to not have to deal with Chinese providers, AtlasCode ($20/mo), OpenCode Go ($10/mo), and Cline Pass ($10/mo) provide 2x to 6x usage for some of the popular open weights (depending on the model).
Personally, I subscribe to Z.ai ($17/mo), and pay API rates for MiMo v2.5, Hy3, Qwen 3.7 Plus, & DeepSeek v4 to the original providers (Xiaomi, Tencent, Alibaba, & DeepSeek).
Nothing makes me happier than seeing AI becoming a commodity rather than the winner takes all bullshit Anthropic and OpenAI have been chasing with hundreds of billions of investor money
Genuine question: can these posts be paid to hype the open source models? If yes, what would be the purpose?
On my work tasks, FastAPI Python and Springboot Java on a modern SaaS product, the only open model that can do tasks well and efficiently is Qwen3.7-Max.
In all my experiments, both GLM-5.2 and Kimi are busy grepping around the codebase for ALMOST 70-80K tokens before writing anything and when they do it typically breaks the code… it feels to me that these models are good but only when you write out a super detailed spec of the task just like it was done a year ago… Qwen3.7 just… does it
Is there something specifically with Kimi that's better here? As far as I know Kimi pricing is about the same as Sonnet 5 -- what happens if you use that model and Fable instead? Or Grok 4.5 which is even cheaper?
I use DeepSeek exclusively and now Kimi K3 offers a great planning assistant for more advanced coding tasks.
DeepSeek v4 Flash is extremely fast and is able to handle pretty much anything I've thrown at it (I use mostly Rust, PSQL, Angular and Terraform).
I self host Bifrost as my LLM gateway, though I wish LLM vendors would do monthly/daily automatic billing (like VPS providers do) rather than prepaid + auto-top up.
It's annoying maintaining a non-refundable minimum balance across vendors, I would rather be billed for my exact usage.
OpenRouter helps, but I don't really like it as a service and not a fan of the mark up.
Yeah deepseek is my workhorse of choice these days, I mostly use pro because I do a lot of concurrent jobs so speed is less of a concern. When it falls over I change the model mid-chat to GPT5.5 and chuck a couple of tokens over to OpenAI and then once it's correctly found the problem I switch back to deepseek, keeping all the 5.5 analysis in context.
I have been a heavy user of k2.5/6 but they seem to have gotten slower and worse at their jobs (and the "engine overloaded" errors have increased ... ), and k3 spends a LOT of time thinking and doesn't produce noticeably better results. I do keep chucking some tasks over to moonshot every now and again to give it a chance. I figure they're under compute pressure and have probably quantised the models to cope temporarily.
But recently I've found myself mostly using deepseek v4 pro and gpt 5.5 when I need the big guns.
GLM 5.2 has been good at select tasks but really shit at others and given how sparingly I use GPT 5.5 I'm not that compelled to use it. I've yet to be impressed by any Google or Anthropic models!!
The best part is using harnesses like reasonix or whale make cache hit at a rate close to 98%, making requests converge to practically free. And that's with unsubsidized American providers like cloudflare or Digital Ocean.
Prepaid means someone can't sign up, use a bunch of inference, then cancel their card and disappear into the sunset. VPS is a more long term investment where it's harder to switch and it doesn't cost the provider much if a few users jump out without paying for a month.
As someone who is also eyeing Bifrost, I am curious to know what made you choose Bifrost at all / why you decided to use an LLM gateway. I also was considering OpenRouter, since it provides pretty much every model, with same-day releases for new models. One thing that is attractive to me about Bifrost is that if I decide to leave OpenRouter tomorrow, I can do so without touching any other part of my stack; I'm hoping self-hosting models becomes more viable, and then I can become less dependent on third party LLM providers like OpenAI/Anthropic/OpenRouter, and I would not need to worry about a model that I depend on suddenly being deprecated.
I'm skeptical. According to arena.ai, Fable 5 dominates almost every category: https://arena.ai/leaderboard Kimi K3 has an edge in WebDev but struggles to reach top 10 in many other categories.
I wonder why there's so much resistance here against chinese models. Sure at my employer claude is used, but at home? I am just happy using my z.ai sub for 20x i got in September last year, coupled with the 39 dollar tier of kimi. I use them in pi, with a collection of extensions i curated myself for this iterationm of models, and a couple glue extensions we have made.
At home i feel way more productive, the speed of my queries are second to none with kimi 2.6, and handing review over to glm5.2 means i can juggle the small models in my brainstorm to commit workflow.
They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?
The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.
Kind of cool to see.. that said, there's more to a tooling experience than benchmarks and specific models. Cursor, Claude Code, Codex, etc. add to the mix. Things like Open-Router and backend options make it easy enough to test.
The tools, libraries and languages you are using can also dramatically affect results. Even on state of the art models, I find, for example, the output of SQL for complex interactions, or C# for that matter to be sub-par, where I find Rust results to be pretty great, with JS/TS falling in between.
At the best, it can feel amazing and productive, at worst, time consuming and annoying that you could have done it faster yourself. YMMV in real world use.
Note: I'm a proponent of human in the loop gatekeeper/reviewer usage of AI, and I'm not able to even consider Chinese models for my own use, and not able to use anything at my day job.
There's this https://github.com/code-yeongyu/oh-my-openagent which implements the OP article's oracle pattern across 11 roles. Each role has a whole ranking of recommended LLMs across many providers. For example "Sisyphus (claude-opus-4-8 / kimi-k3 / glm-5 ) is your main orchestrator."
As always, benchmarks rarely paint the whole picture. It also seems like this article is somewhat biased, eg when Fable and Kimi are close but Fable wins it’s “dead heat”, but when Kimi wins it’s “Kimi wins”. GPT 5.6 seems to be missing as well.
I am really eager to give Kimi K3 a try, but I’ll reserve my judgement until I’ve worked with it for at least a few days.
The apparent bias may be explainable as it’s not remarkable for OpenAI or Anthropic to be slightly ahead. It _is_ remarkable for an open weights model to be better than the closed models from the trillion dollar (allegedly) companies.
China-ambivalent: Open Models are good, Closed Models are bad. Not centralizing power in a few big American companies is good. China is who's building this right now, so we're aligned for now.
I used to be anti-China, and I still think the Chinese government is just a highly adversarial entity that will subsidise, steal and cheat its way to the top.
However... American corporate culture has driven me to this. Fuck Blackrock and all these disgusting parasitical corps - they literally sold China the rope to hang us with and I'm sure as hell not going to pay a cent more for it than I have to.
If China can offer close to state of the art for a fraction of the price then I'm going to use it - thats what the globalists wanted isn't it? They didn't care about saving local manufacturing so why should I care about saving their stupid investments.
Interesting. So the latest in the technology now is this model routing thing. Cursor estimated Composer + Fable works much better than Fable alone. And here K3 + Fable is supposedly better. Interesting.
Seems that “oracle routing” is a term the authors invented. Also sounds like they’re sending requests to all routes but assuming the readers request will route to the “best” model. The closest thing to what they’re describing is semantic routing using a NN search in a vector DB to make a routing decision, but the efficacy of this approach isn’t a slam dunk.
There is too much business risk in running on an American company. They can pull the model back or lobotimize it. No Alex Karp fan, but he was right: companies are worried about hyperscalers stealing their alpha. The lack of guardrails, data security and control really give these open models the edge. If you factor in that they appear to cost less, the hyperscalers are in big trouble. I don't see how this works out. Software was always supposed to be deflationary and collapse down to 0 marginal cost but this isn't remotely the case. It's truly amazing to see the state of open weight models
Interestingly the reality of the open source release is they are opening up to full distillation by the closed source model providers at a deeper and more fundamental level. If anything the open sourcing will help Anthropic and open ai ladder up faster. Open source has always been about mutual cooperation towards a goal and has never closed the door to commercial success. All the hand wringing about open weight models putting closed providers at a disadvantage doesn’t get what working in the open actually does for commercial interests - it is like science in the open - it enables and lifts all boats. Likewise commercial success doesn’t close the opportunity for competition or more open source work - it’s the economy of activity and competition that matters overall. When things stagnate is when closer concerns turtle up and collude on not competing for each others turf.
The future is good and better for everyone the more work is in the open and the more work is in the commercial space. It’s good all around.
These routers can be interesting on a company level to optimize for cost and quality, but for individuals who mostly work on the same tasks, i doubt it. You want to leverage the cache and switching models within a task seems not cost effective to me.
I enjoy all the fun of these new models as much as the next guy but I truly don’t see a circumstance in the near future where my $200 a month with the frontier labs doesn’t get me more than enough consumption of what I need. Local models, chinese models, etc are all very fun weekend projects to tinker with but until something changes (entirely possible!) with how much you get with one of the subscriptions I just don’t see why I would move. What am I missing? Is it simply that a subscription is no good for production use cases? I kinda feel the same way with choice of coding harness, openrouter, etc. why would I use anything other than frontier if I don’t have to pay any more pretty much no matter how much I use? pls tell me if I am holding this wrong haha
Kimi K3 showing competitive performance with Fable while both sitting at the SoTA level on fireworks.ai is a huge milestone. Really interesting to see how the landscape is shifting here.
I have several thoughts about this, which I'll just iterate:
1) US export bans have made it so that Chinese companies have to compete using less-than-state-of-the-art hardware. This has forced Chinese companies to build more cost efficient models. Whereas, US companies have moreso tried to be state of the art by spending more money than anyone else on state-of-the-art hardware.
2) Xi Jinping has called for more open AI models (not to be confused with the closed models of OpenAI), and I'm happy to see a powerful world leader advocating for open-weight AI models. Whereas, the US seems likely to just ban models.
3) My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending--we spend a lot, but we don't necessarily spend well (something something Iran). I mean, if the US is no longer a tech leader in the world, like... what are we a leader at? Manufacturing? Healthcare? LOL. Are we a leader in any industry or by any metric? I wonder if China is attempting to remove the last jewel in the USA's crown with these AI releases.
4) It must be refreshing for companies to have access to a new model that isn't going to get pulled because the government bans it 2 days after release. And it's open-weight so it wont go away--amazing--what a shift in the market.
5) If it becomes clear that open-weight models are the future of AI, will that pop a huge bubble in the US economy? Maybe. But, on the other hand, these companies aren't just training AIs, they are also building data centers which will remain valuable no matter what happens.
The article is about the best you could theoretically do with a perfect router. The takeaway is that trying to build a good router is worth doing. But it's unlikely to be a perfect router.
It would only be inconsistent if the router chose different models for the same task.
Considering that of 89 terminal tasks there were 11 that only Kimi got right, and 7 only Fable - having a router gives significantly more consistent results if you define consistent to mean correct.
Neither. They ran all the cases using both models, picked the winning result for each one, and then said "if you had a router that guessed with 100% accuracy, here's what it would have picked"
for people in mainland china the only option now is Kimi K3+GLM 5.2 for their day to day work as Fable is blocked. for me i use Fable + codex when i was out of mainland china, it is good when you have options wherever you are in this world
the per-token comparison keeps missing that k3 spends way more tokens per task. if it burns 3x tokens to reach the same result as fable, cheap per-token stops mattering
"So where's this huge price gap coming from? token pricing, prompt caching, and effort-per-task. On SWE for example, K3 works much harder than Fable: roughly 55 turns and 1.3M tokens a task versus 21 turns and 130K. On the long terminal tasks it's the other way around: Fable is the one that spirals, running up 64 turns and 1.5M tokens (sometimes straight into a timeout).
Prompt caching does most of the work of turning that effort into K3's price advantage: even when K3 reads ten times the tokens, with cache hits that means that SWE runs still come in lower cost than Fable. There’s a tradeoff. Tasks with extra turns generally mean more wall-clock time per run i.e. slower runs. If you need an answer in two seconds, that matters; if you're running agents in the background at scale, a bill that's a fraction of the size matters a lot more."
What you're seeing is the distilled (no pun intended) result of realpolitik at play. China's labs aren't as open as they are out of altruism, but rather to capitalize and undercut the monopoly held by U.S. competitors.
There's no irony. The problem is these generalizations and labelling that's hurting everyone. It's the wrong way to view China and maybe many other places.
Just like the definition of AI, the definition of democratic or not evolved long ago.
Shifted to K3 and it is like a fresh air. While Fable and Sol very good at _generating_ code i even cannot force sol to just read all relevant source files. As result it reinvent existing things or assumes too much about internals of other, which lead to incorrect uses. Even with hard planing mode GPT burned 33% of week tokens for 3 hours producing no result and even cannot find root cause, but K3 fixed it in a minutes. Hilarious that while software not started with empty cache Sol handcrafted empty cache to let it start. Having full procedure right in the MEMORY.md. K3 found this, and downloaded cache correctly. But while using gen5 models i always feeling myself ignored. Any commands, steering anything - just ignoring. At first i added lots of hooks, no "?", expect in rust code, bash hook ban for find|grep|tail, with notice to use ltsp but then it started to ignore strategically. Also whole "thinking" thing is hidden from claude. regenerated thinking summary is incomplete and not useful. While being very verbose kimi k3 doing good job providing whole train of thoughts. So looks like claude degradation started from 4.6 comes to a logical end. Maybe i missing something and changing existing code beyond bug fixes is not a way to go. But there is no currently stable way to generate code on hier of specs and lean models i'd like to.
we put a router model in front of two other models so the router can decide which model is better at deciding things. next we'll need a router for the router and eventually the entire internet is just routers routing routers to other routers
Agreed. The benchmark closest to my experience is FrontierMath Tier 4. Fable and Sol (90%) are very far ahead of Kimi K3 (not even 40%). Kimi is trained heavily to basic agentic tasks, like all the other open models right now.
I'm not sure why you're so dismissive about this model. It isn't currently open, but it will be. It was slow when I used it so i'll give you that. There will be other providers with better performance.
My only gripe was that it takes a very long time to think. It's a good model otherwise.
If you haven’t been really running and testing these models yourself, they are all benchmaxxed. No matter how close they score to frontier on whatever metric, they always fall apart in real world tasks and their token efficiency is ridiculously bad. Fireworks has incredible incentive to make this claim in a headline, because Fireworks hosting K3 for you is pure profit for them, unlike when they host closed source models.
Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.
However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/
I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.
In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.
Yeah, open weights hasn't fully caught up yet, but it's getting very close. And it's certainly passed the point of being reasonably interchangeable with the frontier for (programming) work. Add in the benefits of not being rug-pulled by the frontier labs silently messing with, the knobs on their models or outright denying you the ability to do certain kinds of work (c.f. the HuggingFace fiasco), and they probably come out ahead in several respects.
To your slow point: I did a quick eval for a data viz task to do a qualitative comparison and Fable was much faster (by 4x), but I struggle with the "token inefficiency" as a sort of whatever metric.
Kimi was half the cost and produced a near identical output.
https://fr4geiw93g.evvl.io/
On a fairly simple coding task Kimi was 2x the cost and 7x as slow (and gpt-5.6-sol was even cheaper).
https://r26pakjmhv.evvl.io/
In my limited anecdotal experience, Kimi K3 is a bit better than Opus 4.8 and Qwen3.8 Max is disastrously bad. It can reason fine, but the moment it tries to do something it gets stuck into long second-guessing loops with no progress. It sometimes refuses to try and debug a problem even after multiple suggestions. I'm sticking to K3.
with all the hype around kimi i actually renewed my subscription (last time i tried it out was on 2.5 release) - just the 40€ one.
Its hard to compare that one to my regular sub of anthropic (107€), but everyone always says how much cheaper it is so i thought it was worth a try.
1.5 days later ive went through more then 50% of my weekly usage and decided to renew the regular anthropic sub too.
The API Pricing is definitely cheaper, but anthropics subscription budget seems to equalize that advantage right now. also kimi code feels like claude code from ... september 2025
also - considering how they announced they'd temporarily close subscriptions to make sure they can service customers... i was slightly surprised that there was no issue subscribing, less then 1 day after that announcement. Makes you think if that was just a marketing stunt
Could you check how many tokens the different models spent to do the task? With all the comments about Kimi K3 Not being token efficient I'm curious if your test confirms that
I love Kimi for some things, but K3 to me struggles with things Opus 4.8 breezes through. Admittedly stuff that is on the more complex side (code generation bugs in a compiler) to the point that after two days of struggle I stopped Kimi and will have Opus redo its work once I have spare tokens...
This wasn't just being slow - it didn't make forward progress.
For simpler stuff even 2.7 does just fine, though.
I don't see a difference in capabilities between k3 and fable, but k3 is slow and expensive. Burned through my monthly plan in 3 days.
> the Chinese models really are slow and token-inefficient
This totally depends on the model. Deepseek V4 is very fast and efficient.
Haha, this is awesome. Hadn't heard of it, thank you for the link.
Honestly my anecdotal experience is that fable is benchmaxxed. I have not observed useful gains for Claude since 4.6, with each model iteration making progressively poorer decisions in pursuit of its goal.
The 5.5/5.6 series has performed quite well however. My guess is that my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.
> my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.
What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?
yes, above 50% I don't observe a significant improvement, the models scoring above 65% like opus4.8 seem worse than gpt5.5/opus4.6. DeepSWE appears to align more closely to my personal experience.
this mirrors my approx experience.
Fable works very well for me on a moderately large codebase. I have had to correct it a few times or point it on the right track, but given how much faster it is at programming than I am that's a very minor issue (and most of these errors are because I underspecified what I wanted in the prompt, I can only think of two cases where it was genuinely wrong... that's a lot better than me in my professional career). Code quality is equal to what I would come up with (and better in areas I'm not familiar with) and the overall software engineering bar is higher because it doesn't get bored when I tell it to do refactors or write integration/regression tests that I would otherwise put off. Also makes it easy to audit code for things like missing audit logging or error notifications that a human would get bored doing.
The product is a fairly standard Ruby on Rails webapp with postgres as the DB. Application complexity is probably a bit higher than average for a webapp. So it's nothing that pushes the boundaries of software engineering, but it is a real product. Token budget has not been an issue for me. I pay for the Max plan ($200/month) and it is well worth it.
Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other.
We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.
Data at https://gertlabs.com/rankings
I have been testing the various models, and I would not call Fable SOTA.
I can't actually get Fable to do anything. I only work on back-end code, and the moment Fable notices the jwt scope checks on the endpoints it's game over, it refuses to do anything because security is involved.
So for me, Fable is completely useless, the bar is very low, any llm that will actually attempt the task beats it every time.
All models are benchmaxxed, period. ”Jagged frontier” is the euphemism du jour, I believe?
Anthropic/OpenAI were touting PhD-level intelligence three years ago. And they’re still shipping models that aren’t smart enough to realize things such as the need to drive the car to the car wash (because they hadn’t yet hill-climbed that particular brain-teaser).
didn't read the article, but just the title felt fishy because of course an inference provider wouldn't mind claiming to provide access to a mythos-class model. So far, GLM 5.2 is still my go-to, even at less than a third the parameter count compared to Kimi K3
It's telling that in their release post, Moonshot themselves said that K3 lags Fable and GPT-5.6 in "user experience". I took that to mean the stuff you can't push directly via RL, what some people call "big model smell".
"Despite being a highly competitive model overall, K3 nonetheless exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT 5.6 Sol."
https://www.kimi.com/blog/kimi-k3
There are some benchmarks I personally trust. But the most reliable indicator other than public benchmarks is the quality of products people are working on, and the model they use for the work. Not a benchmark scoreboard or a one~few shot demo, but the product they have been building for weeks. The reality is, even the diehard advocates who build and sell tooling for open weights, still use proprietary frontier models for their jobs.
Clearly the best way to test these is a completely scientific and controlled process of prompting for a creature on a bicycle. Much like testing the fitness of a fish by making it climb a tree, the LLM will show its true colors when presented with the impossible task of rendering an image.
Yeah on internal non coding benchmarks at my company, they are around last years models and don't hold up well.
Totally agree with all of this. Most of the problems we solve with AI are not one shot, all the bench marks also miss the human components when testing. Thinking alongside with a human and coming to a solution is what Fable does better than most other models.
I think this one has greater impact than deepseek, next one for sure, China will take the lead without any question.
So they aren’t as good as 2024 models? 2025?
I encountered the same issue. Kimi 2.7 looked impressive on paper, but in practice, the code was so riddled with errors that I ended up using GPT-5.5 to fix it.
I've seen this phrase repeated over and over again and I don't think it is true, or at least, to the degree that "benchmaxxing" claims are true, American frontier AI companies are probably about as guilty of it as Chinese AI companies are.
For both GLM 5.2 and Kimi K3 I feel the rough average of the benchmarks gives you a rough idea where they stand. GLM 5.2 was sitting somewhere behind Opus 4.8 but it didn't feel very far away. I've used Kimi K3 via OpenRouter and while I've had limited experience so far, it sure as shit feels like it's right up there with Sol and Fable to me. I am happily able to believe Fable has the edge still, but on a request by request basis it would be pretty easy to get an impression one way or another.
The existing Kimi models were already pretty good so I really just don't find this new model to be that hard to believe. Maybe I'm naive.
Why is token efficiency a concern with free models?
Well that sucks that everything is full of misinformation about this subject.
It's better to not even follow the news or benchmarks and just use whatever is available. I make my own judgement.
[flagged]
You got baited by bad sampling settings.
It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.
All you've said needs the qualifier - "for now!" Look at the trend line. It's clear that if they're not yet at the level of being "good enough" for coding, they will be soon. Sensationalist headlines aside, we all need to be preparing for a world where open models can do pretty much any software tasks you need them to.
Very interesting. They test Kimi K3 and Fable on a set of approx 1000 tasks grouped into 5 areas (SWE, Legal, etc).
They put a router model in front that predicts whether Kimi or Fable is going to give a better cost for a correct result. (They believe that ultimately such a router model should be continuously trained on your own workloads so it makes the best decisions for you).
Their router chose Kimi the majority of the time (72% in one category, all the way to 96% in another category), leading to cost savings in every category (from 1.5x to 50x depending).
> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).
Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.
There are a bunch of routers like this eg https://openrouter.ai/openrouter/auto
This benchmark is probably also self promotion of services. Fireworks happens to make a router. Using the router gets better performance.
https://docs.fireworks.ai/deployments/routers
I work for a router company too, I ran some tests on all of the cheapest models and came to the same outcome where a handful of small models ran together in conjunction outperform SoTA models -- outperforms in that it got a 95% vs a 94% and I bet that changes with the day of the week. Anyways, I did get a similar result in a different sort of measurement.
What is routing? How do they decide which is better? The only way to come up with a routing model for your workload is to send queries to all the models and then come up with a way to say which solution was better, often trying out multiple times for the same model + query to account for other statistical errors.
This makes you, an ai inference user an unwitting AI company with a non scalable product.
The biggest mental trap people have fallen for is the notion of “best” and, always using the frontier model. Instead you should just bite the bulet and choose the cheapest or the best. This routing dance is just tokenmaxxing in disguise.
Edit: Another thing concerning me is that the models themselves are not concrete behind their endpoints. Models can be arbitrarily dumbed down by reducing their inference resources. Once Anthro/OAI release a new great model, they are fully incentivised to dumb down their current models to drive traffic to the new shiny more expensive one. In fact they can do this for any reason. Once they switch things behind the API interface, how useful is your meticulously tuned router? Not much at all.
They can but there isn’t evidence that they do.
When DeepSeek was released, it had an immidiate and significant impact on the US stock-market. Now when its becoming common knowledge that China is almost at parity with US SOTA models with good momentum, why is there no sentiment change on the market?
Because markets in the short term are almost a random walk and making the blanket statement that “Walmart stock is down today because Deepseek” was an easy narrative to repeat for media people who cover the stock market.
In reality the world is a highly complex, chaotic, reflexive system, and saying “the entire market moved today because of 12,000,000 factors that randomly aligned” isn’t satisfying enough for people to follow your media channel so they can monetize your eyeballs.
Because, as you said, this is no longer news. Deepseek was the news. This is just the predictable progress playing out.
It was never DeepSeek’s release that dropped the NASDAQ at the time; it was the unknown risks of the early thoughts related to trade wars. Popular financial newspapers can promote anything they want, but these news do not typically drive large investor decisions.
It might be because no interested party is using that piece of news to move market. They have enough with other news to do it. Just a practical matter (or may be something else, who knows!)
The “significant impact” lasted like 2 days and then the market corrected itself.
Back then people didn't understand how AI was run. It should have probably made Nvidia stock actually go up. I think the other Factor, and I might just be two into AI and most people are normies, the hype around Chinese models we've learned is overblown. United States models are a league above.
there was a significant impact on the market because the market assumed that a strong cheaper model would have a significant impact on the US AI companies' business. That didn't happen.
deepseek drop was based on assumption that we overestimated how much compute and infra is needed to train models. They seem to have claimed that they did it in couple of million or something.
Because the markets learned Chinese = fake results
maybe because it's not true?
I will accept a 5% drop in benchmarks for a model that talks to me like a human.
Funny. I prefer a model that does not attempt to talk like a human.
I strictly prefer when models ignore any human quirks in my responses. Claude trying to be your friend, saying LOL to your jokes is ridiculous and frankly, harmful
Why? LLMs are not humans.
You don't even need to pay a 5% hit. Just paste Fable output into Gemini Flash and it will rewrite it in more accessible language.
Yeah. I've found that Opus by default outputs something I call "Claude-lang." It consists of oversimplified, grammatically incomplete sentences that I find painful to read.
Maybe it is something that is easy for it to read and write, but definitely not for humans.
For example,
(Yeah, Opus outputted it in one line)What's the data governance and privacy controls on using Kimi K3 if I subscribe to their coding plans? I want to migrate away from Anthropic
From https://platform.kimi.ai/docs/agreement/modeluse
> We may use Content to provide, maintain, develop, support, and improve the Services, comply with applicable law, enforce our terms and policies, and keep the Services safe and secure. Customer who requires restrictions on the use of Customer Content for training or improving Moonshot AI models may contact Moonshot AI to discuss available enterprise arrangements or separate written agreements. Unless otherwise expressly agreed in writing, Customer Content may be used for the foregoing purposes.
Notably, unlike Claude, there is not an opt-out option for the model training part. The TOS explicitly allows Kimi to train on your code.
Simplest way is to signup to OpenRouter and filter out all non ZDR (zero data retention) providers. Paying "API rates" however can prove expensive compared to coding plans (for instance, MiniMax $20/mo coding plan allows 1.7b tokens; depending on input/output/cache ratio, it is worth $200 to $500+), except for Hy3, DeepSeek v4 Pro, and MiMo v2.5 Pro (whose API rates are cheap and/or discounted already).
If you're looking to not have to deal with Chinese providers, AtlasCode ($20/mo), OpenCode Go ($10/mo), and Cline Pass ($10/mo) provide 2x to 6x usage for some of the popular open weights (depending on the model).
Personally, I subscribe to Z.ai ($17/mo), and pay API rates for MiMo v2.5, Hy3, Qwen 3.7 Plus, & DeepSeek v4 to the original providers (Xiaomi, Tencent, Alibaba, & DeepSeek).
Need to wait until "western" providers start hosting it.
> What's the data governance and privacy controls on using Kimi K3 if I subscribe to their coding plans?
If you can’t figure it out from their website, perhaps it is a red flag?
https://www.kimi.com/user/agreement/zh/userPrivacy
Nothing makes me happier than seeing AI becoming a commodity rather than the winner takes all bullshit Anthropic and OpenAI have been chasing with hundreds of billions of investor money
Anthropic looks like Roman empire fast-farwarded, getting to the other side of the peak even before the IPO.
Yeah, I didn't get in either.
Genuine question: can these posts be paid to hype the open source models? If yes, what would be the purpose?
On my work tasks, FastAPI Python and Springboot Java on a modern SaaS product, the only open model that can do tasks well and efficiently is Qwen3.7-Max.
In all my experiments, both GLM-5.2 and Kimi are busy grepping around the codebase for ALMOST 70-80K tokens before writing anything and when they do it typically breaks the code… it feels to me that these models are good but only when you write out a super detailed spec of the task just like it was done a year ago… Qwen3.7 just… does it
It's (good) content marketing. They sell access to Kimi K3. They're one of the biggest model inference providers out there.
Fireworks is an inference provider that specializes in running open source models very fast and makes almost all of their margin on chinese models
So the economic incentive is literally their entire business model lol
Lots of money in tech influencing - but most is from the big players (OpenAI buying tbpn, early access to select influencers etc).
>the only open model that can do tasks well and efficiently is Qwen3.7-Max
Qwen3.7-Max is a proprietary, closed weight model.
Have you tested Qwen3.8-Max yet?
[flagged]
Is there something specifically with Kimi that's better here? As far as I know Kimi pricing is about the same as Sonnet 5 -- what happens if you use that model and Fable instead? Or Grok 4.5 which is even cheaper?
The article says Kimi is better at some things than Fable. That's probably not true of Sonnet.
One benefit of an open source one is that you can, as a large corporation, run it "locally" within your own data center. Even fine tune it.
I love the Chinese models.
I use DeepSeek exclusively and now Kimi K3 offers a great planning assistant for more advanced coding tasks.
DeepSeek v4 Flash is extremely fast and is able to handle pretty much anything I've thrown at it (I use mostly Rust, PSQL, Angular and Terraform).
I self host Bifrost as my LLM gateway, though I wish LLM vendors would do monthly/daily automatic billing (like VPS providers do) rather than prepaid + auto-top up.
It's annoying maintaining a non-refundable minimum balance across vendors, I would rather be billed for my exact usage.
OpenRouter helps, but I don't really like it as a service and not a fan of the mark up.
Yeah deepseek is my workhorse of choice these days, I mostly use pro because I do a lot of concurrent jobs so speed is less of a concern. When it falls over I change the model mid-chat to GPT5.5 and chuck a couple of tokens over to OpenAI and then once it's correctly found the problem I switch back to deepseek, keeping all the 5.5 analysis in context.
I have been a heavy user of k2.5/6 but they seem to have gotten slower and worse at their jobs (and the "engine overloaded" errors have increased ... ), and k3 spends a LOT of time thinking and doesn't produce noticeably better results. I do keep chucking some tasks over to moonshot every now and again to give it a chance. I figure they're under compute pressure and have probably quantised the models to cope temporarily.
But recently I've found myself mostly using deepseek v4 pro and gpt 5.5 when I need the big guns.
GLM 5.2 has been good at select tasks but really shit at others and given how sparingly I use GPT 5.5 I'm not that compelled to use it. I've yet to be impressed by any Google or Anthropic models!!
The best part is using harnesses like reasonix or whale make cache hit at a rate close to 98%, making requests converge to practically free. And that's with unsubsidized American providers like cloudflare or Digital Ocean.
Prepaid means someone can't sign up, use a bunch of inference, then cancel their card and disappear into the sunset. VPS is a more long term investment where it's harder to switch and it doesn't cost the provider much if a few users jump out without paying for a month.
As someone who is also eyeing Bifrost, I am curious to know what made you choose Bifrost at all / why you decided to use an LLM gateway. I also was considering OpenRouter, since it provides pretty much every model, with same-day releases for new models. One thing that is attractive to me about Bifrost is that if I decide to leave OpenRouter tomorrow, I can do so without touching any other part of my stack; I'm hoping self-hosting models becomes more viable, and then I can become less dependent on third party LLM providers like OpenAI/Anthropic/OpenRouter, and I would not need to worry about a model that I depend on suddenly being deprecated.
What don't you like about the service, besides the mark up?
I didn't realize that OpenRouter had a mark up. Is it a flat mark up across the board or depending on the model?
Have you looked into OpenCode Zen or OpenCode Go?
https://opencode.ai/zen
[flagged]
I'm skeptical. According to arena.ai, Fable 5 dominates almost every category: https://arena.ai/leaderboard Kimi K3 has an edge in WebDev but struggles to reach top 10 in many other categories.
In my experience, Fable is not even close to Sol 5.6 High (not even the max tier) for coding.
1) it's substantially slower.
2) it's substantially more expensive.
3) it's code is considerably worse.
It's a joke when you consider what you get for what you pay for.
I wonder why there's so much resistance here against chinese models. Sure at my employer claude is used, but at home? I am just happy using my z.ai sub for 20x i got in September last year, coupled with the 39 dollar tier of kimi. I use them in pi, with a collection of extensions i curated myself for this iterationm of models, and a couple glue extensions we have made.
At home i feel way more productive, the speed of my queries are second to none with kimi 2.6, and handing review over to glm5.2 means i can juggle the small models in my brainstorm to commit workflow.
Hmmm, a company that hosts open models is telling us how good open models are...
They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?
The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.
Anybody who's tried it knows that what they're saying is true though.
That's incredible. Hope the chinese keep it up!
If only to put some pressure on american labs to bring those costs down
Kind of cool to see.. that said, there's more to a tooling experience than benchmarks and specific models. Cursor, Claude Code, Codex, etc. add to the mix. Things like Open-Router and backend options make it easy enough to test.
The tools, libraries and languages you are using can also dramatically affect results. Even on state of the art models, I find, for example, the output of SQL for complex interactions, or C# for that matter to be sub-par, where I find Rust results to be pretty great, with JS/TS falling in between.
At the best, it can feel amazing and productive, at worst, time consuming and annoying that you could have done it faster yourself. YMMV in real world use.
Note: I'm a proponent of human in the loop gatekeeper/reviewer usage of AI, and I'm not able to even consider Chinese models for my own use, and not able to use anything at my day job.
Anyone have routing harnesses like this describes with Claude Code? Or other good routing platform recommendations?
(yes, I know this article is about an oracle router)
There's this https://github.com/code-yeongyu/oh-my-openagent which implements the OP article's oracle pattern across 11 roles. Each role has a whole ranking of recommended LLMs across many providers. For example "Sisyphus (claude-opus-4-8 / kimi-k3 / glm-5 ) is your main orchestrator."
SoTA means "State of the art". I wish it didn't take me 5 minutes to figure out what SoTA stands for.
Thank you!
As always, benchmarks rarely paint the whole picture. It also seems like this article is somewhat biased, eg when Fable and Kimi are close but Fable wins it’s “dead heat”, but when Kimi wins it’s “Kimi wins”. GPT 5.6 seems to be missing as well.
I am really eager to give Kimi K3 a try, but I’ll reserve my judgement until I’ve worked with it for at least a few days.
The apparent bias may be explainable as it’s not remarkable for OpenAI or Anthropic to be slightly ahead. It _is_ remarkable for an open weights model to be better than the closed models from the trillion dollar (allegedly) companies.
it appears that 3% is the threshold as 3.1% gets the nod the other way
Anti-China: K3 is propaganda and benchmaxxed, no matter how anthropic and openai reactor for these, it just a smoke signal.
Pro-China: K3 is good choice for better and affordable choice to smash down the Big three ruling.
China-ambivalent: Open Models are good, Closed Models are bad. Not centralizing power in a few big American companies is good. China is who's building this right now, so we're aligned for now.
i only care about the license and results to be honest
I used to be anti-China, and I still think the Chinese government is just a highly adversarial entity that will subsidise, steal and cheat its way to the top.
However... American corporate culture has driven me to this. Fuck Blackrock and all these disgusting parasitical corps - they literally sold China the rope to hang us with and I'm sure as hell not going to pay a cent more for it than I have to.
If China can offer close to state of the art for a fraction of the price then I'm going to use it - thats what the globalists wanted isn't it? They didn't care about saving local manufacturing so why should I care about saving their stupid investments.
Interesting. So the latest in the technology now is this model routing thing. Cursor estimated Composer + Fable works much better than Fable alone. And here K3 + Fable is supposedly better. Interesting.
Those are different techniques...
Cursor's Composer + Fable combo was a plan agent + execute sub-agents swarm
Fireworks K3 + Fable router was dynamically choosing single model for the task based on cost+performance metrics
Mythos/Fable was the state of the art back in March, if not earlier.
So China is now only 4 months behind US frontier models now when the rule of thumb was 6 months just a little while ago.
It released in June..
Seems that “oracle routing” is a term the authors invented. Also sounds like they’re sending requests to all routes but assuming the readers request will route to the “best” model. The closest thing to what they’re describing is semantic routing using a NN search in a vector DB to make a routing decision, but the efficacy of this approach isn’t a slam dunk.
A third the cost, open source, and won't refuse every other request because of some vague possible connection to cybersecurity concerns.
There is too much business risk in running on an American company. They can pull the model back or lobotimize it. No Alex Karp fan, but he was right: companies are worried about hyperscalers stealing their alpha. The lack of guardrails, data security and control really give these open models the edge. If you factor in that they appear to cost less, the hyperscalers are in big trouble. I don't see how this works out. Software was always supposed to be deflationary and collapse down to 0 marginal cost but this isn't remotely the case. It's truly amazing to see the state of open weight models
My questions about strawberries got blocked as too dangerous! I’m not joking
Or biology or chemistry.
From the blog post moonshot refers to it as open source but only mentions releasing the weights.
Interestingly the reality of the open source release is they are opening up to full distillation by the closed source model providers at a deeper and more fundamental level. If anything the open sourcing will help Anthropic and open ai ladder up faster. Open source has always been about mutual cooperation towards a goal and has never closed the door to commercial success. All the hand wringing about open weight models putting closed providers at a disadvantage doesn’t get what working in the open actually does for commercial interests - it is like science in the open - it enables and lifts all boats. Likewise commercial success doesn’t close the opportunity for competition or more open source work - it’s the economy of activity and competition that matters overall. When things stagnate is when closer concerns turtle up and collude on not competing for each others turf.
The future is good and better for everyone the more work is in the open and the more work is in the commercial space. It’s good all around.
For regular chat users it's $19 while Claude is $20...
hmm almost like the exact race-to-the-bottom + arms-race dynamic all the doomers have been warning about
These routers can be interesting on a company level to optimize for cost and quality, but for individuals who mostly work on the same tasks, i doubt it. You want to leverage the cache and switching models within a task seems not cost effective to me.
If only you could run K3 locally that would be the magic bullet to make it a true magic bullet!
Until Kimi 4 comes out and you'd be like "if only you could run Kimi 4 locally"
Everyone wants the latest and greatest.
You can! It just might be a little bit outside your budget.
I enjoy all the fun of these new models as much as the next guy but I truly don’t see a circumstance in the near future where my $200 a month with the frontier labs doesn’t get me more than enough consumption of what I need. Local models, chinese models, etc are all very fun weekend projects to tinker with but until something changes (entirely possible!) with how much you get with one of the subscriptions I just don’t see why I would move. What am I missing? Is it simply that a subscription is no good for production use cases? I kinda feel the same way with choice of coding harness, openrouter, etc. why would I use anything other than frontier if I don’t have to pay any more pretty much no matter how much I use? pls tell me if I am holding this wrong haha
If everyone took your position, then frontier labs can keep raising their price.
Kimi K3 showing competitive performance with Fable while both sitting at the SoTA level on fireworks.ai is a huge milestone. Really interesting to see how the landscape is shifting here.
I have several thoughts about this, which I'll just iterate:
1) US export bans have made it so that Chinese companies have to compete using less-than-state-of-the-art hardware. This has forced Chinese companies to build more cost efficient models. Whereas, US companies have moreso tried to be state of the art by spending more money than anyone else on state-of-the-art hardware.
2) Xi Jinping has called for more open AI models (not to be confused with the closed models of OpenAI), and I'm happy to see a powerful world leader advocating for open-weight AI models. Whereas, the US seems likely to just ban models.
3) My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending--we spend a lot, but we don't necessarily spend well (something something Iran). I mean, if the US is no longer a tech leader in the world, like... what are we a leader at? Manufacturing? Healthcare? LOL. Are we a leader in any industry or by any metric? I wonder if China is attempting to remove the last jewel in the USA's crown with these AI releases.
4) It must be refreshing for companies to have access to a new model that isn't going to get pulled because the government bans it 2 days after release. And it's open-weight so it wont go away--amazing--what a shift in the market.
5) If it becomes clear that open-weight models are the future of AI, will that pop a huge bubble in the US economy? Maybe. But, on the other hand, these companies aren't just training AIs, they are also building data centers which will remain valuable no matter what happens.
Implying that AI is the last jewel of the USA is too simplistic and ignorant.
People are paying for tokens, that will likely continue even if open weight wins.
[flagged]
The article is about the best you could theoretically do with a perfect router. The takeaway is that trying to build a good router is worth doing. But it's unlikely to be a perfect router.
They forgot to compare and incorporate GPT-5.6-Sol.
I tried to subscribe to Kimi but it seems it's "sold out"
Openrouter also features routing. Routing is indeed an option if you don’t need consistent behavior and allow switching models.
It would only be inconsistent if the router chose different models for the same task.
Considering that of 89 terminal tasks there were 11 that only Kimi got right, and 7 only Fable - having a router gives significantly more consistent results if you define consistent to mean correct.
What's the reason the current SOTA wasn't tested? GPT-5.6-Sol-Max is the actual SOTA.
Was this an out of sample test of the router or was it trained on these specific use cases/eval suites?
Neither. They ran all the cases using both models, picked the winning result for each one, and then said "if you had a router that guessed with 100% accuracy, here's what it would have picked"
for people in mainland china the only option now is Kimi K3+GLM 5.2 for their day to day work as Fable is blocked. for me i use Fable + codex when i was out of mainland china, it is good when you have options wherever you are in this world
For model routers, do they have to retrain the routing model every time a new LLM is released?
I really like the idea behind OpenRouter's new auto-beta -- classify by task type, and then just follow what the market is using based on the last 7d. https://openrouter.ai/docs/guides/routing/routers/auto-route...
Maybe a continuous evaluation process, why is there anything to train?
the per-token comparison keeps missing that k3 spends way more tokens per task. if it burns 3x tokens to reach the same result as fable, cheap per-token stops mattering
They address that in the article :) to quote :-
"So where's this huge price gap coming from? token pricing, prompt caching, and effort-per-task. On SWE for example, K3 works much harder than Fable: roughly 55 turns and 1.3M tokens a task versus 21 turns and 130K. On the long terminal tasks it's the other way around: Fable is the one that spirals, running up 64 turns and 1.5M tokens (sometimes straight into a timeout).
Prompt caching does most of the work of turning that effort into K3's price advantage: even when K3 reads ten times the tokens, with cache hits that means that SWE runs still come in lower cost than Fable. There’s a tradeoff. Tasks with extra turns generally mean more wall-clock time per run i.e. slower runs. If you need an answer in two seconds, that matters; if you're running agents in the background at scale, a bill that's a fraction of the size matters a lot more."
Your account <...> request reached organization TPD rate limit
The irony is the Chinese are being very democratic with their models, while the USA tries to do central control.
Glad to see centralized control fail on the grandest scale. Maybe we can learn a thing or two.
What you're seeing is the distilled (no pun intended) result of realpolitik at play. China's labs aren't as open as they are out of altruism, but rather to capitalize and undercut the monopoly held by U.S. competitors.
I'm reminded of space race, where US built centralized planning, while USSR did competition between construction buros
> The irony is the Chinese are being
There's no irony. The problem is these generalizations and labelling that's hurting everyone. It's the wrong way to view China and maybe many other places.
Just like the definition of AI, the definition of democratic or not evolved long ago.
[flagged]
[flagged]
cluade model, claude opus 4.6,7 perform very well.and stable, understandable ,i miss that in kimi, its slow, doesnt gives feels like claude
Shifted to K3 and it is like a fresh air. While Fable and Sol very good at _generating_ code i even cannot force sol to just read all relevant source files. As result it reinvent existing things or assumes too much about internals of other, which lead to incorrect uses. Even with hard planing mode GPT burned 33% of week tokens for 3 hours producing no result and even cannot find root cause, but K3 fixed it in a minutes. Hilarious that while software not started with empty cache Sol handcrafted empty cache to let it start. Having full procedure right in the MEMORY.md. K3 found this, and downloaded cache correctly. But while using gen5 models i always feeling myself ignored. Any commands, steering anything - just ignoring. At first i added lots of hooks, no "?", expect in rust code, bash hook ban for find|grep|tail, with notice to use ltsp but then it started to ignore strategically. Also whole "thinking" thing is hidden from claude. regenerated thinking summary is incomplete and not useful. While being very verbose kimi k3 doing good job providing whole train of thoughts. So looks like claude degradation started from 4.6 comes to a logical end. Maybe i missing something and changing existing code beyond bug fixes is not a way to go. But there is no currently stable way to generate code on hier of specs and lean models i'd like to.
If I had an oracle, why would I need an LLM?
The Kimi K3 felt pretty good when I tried it out.
we put a router model in front of two other models so the router can decide which model is better at deciding things. next we'll need a router for the router and eventually the entire internet is just routers routing routers to other routers
It's all pipes, Jerry.
It's a series of tubes
"Information Retrieval is where it's at" Buttle said to Tuttle.
It’s routers all the way down
And the weights are just in the routers now? Love that idea.
[dead]
what are my options if i want to use a router like this ? who provides one ?
Two that I use are:
* Openrouter.ai for a hosted router
* https://github.com/diegosouzapw/OmniRoute for a local router
[dead]
Oracle routing is by definition not possible.
It is not SOTA. Give me a break. Sure, run it on Cerebras to get speed but that’s pretty much its advantage.
Agreed. The benchmark closest to my experience is FrontierMath Tier 4. Fable and Sol (90%) are very far ahead of Kimi K3 (not even 40%). Kimi is trained heavily to basic agentic tasks, like all the other open models right now.
Cerebras does not share the quantization of the models so you don't know if you're getting real K3 or k3 lite or something else.
Why do you think it's not SOTA?
Why SoTA (uppercase “T”) instead of SotA (lowercase “T”) ?
“State of [T]he Art” versus “State of [t]he Art”.
If not SotA then at least SOTA, which is more accurate.
It should be SotA.
An cross of SoT (source of truth) and SotA (state of the art). SotA looks weird, admittedly.
SOTA police
I suspect typo.
probably buckeye fans
“State of THE art.”
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[dead]
[dead]
[flagged]
> hyping an open model that isn’t open
Moonshot has committed to release it end of the month.
I'm not sure why you're so dismissive about this model. It isn't currently open, but it will be. It was slow when I used it so i'll give you that. There will be other providers with better performance.
My only gripe was that it takes a very long time to think. It's a good model otherwise.