I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t.
GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
We’ve really evolved quickly into simple vectors for a magical tool that solves our problems. Can’t solve the problem? There’ll be a new release soon that can!
Since Fable, my legit infrastructure project has turned into the sort of thing I can do 95% on my phone. It’s reliable enough instead of doing big reviews, I’ve just been giving it smaller tasks, and dozens in parallel.
I created a skill that’s focused on getting PRs merge-ready, and now my attention is fully back where it should be, on deciding what changes will make the product better.
Our entire stack is Apache 2.0 open source, including the agent docs, so if you wanna try sitting at a higher level of abstraction, install the skill in your repo or just clone our whole project and start adding features: https://good.vibes.diy/blog/beast-mode-skill-for-claude-code
I didn’t mention my case since it’s quite esoteric, but I am working on an application using the Apple RoomPlan API, which is very powerful but very limited in customizability. Opus simply couldn’t alter the scanning view for me, it would try things over and over and eventually started making up parameters and passing them hoping it would work.
Completely failed, but I knew it was possible because a competitor app does it.
Fable also failed, then added log lines (as did Opus, but Opus failed to do anything useful with them) and then reversed engineered the API, and made it work.
Here’s one difference I have seen. I forgot I had a multi-session audio probe running while trying to repro audio glitches, and Fable came back with: “your pops are already on tape.”
Interesting choice of words. Phrased so casually. It picked a low-tech idiom that fit the situation instead of giving some sterile technical answer. That kind of language and context awareness never happened for me with Opus, or gpt 5.5.
you're living in the age of AI; not AGI. Also, there's pretty much zero moderation on HN, so astroturfing is likely streaming through just as bad as reddit. It' sjust noit as obvious because it's a smaller scoped website.
Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.
Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count.
Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterday. Never seen anything like that.
Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.
I think the improvement on how it codes is pretty much represented correctly by the benchmarks (a nice bump, but not some crazy leap)
But where it really shines is in how NOT lazy it is. Fable requires less hand-holding. And I can understand how someone who uses Claude-Code sparingly and with very focused prompts would not see a lot of improvement there.
But simple example: if you ask Opus to do a review of the codebase (with a short prompt and not too much guidance), I've had it basically read the `git log` output, do a simple `ls` and have it declare "Everything looks great! No problems found!", when Fable really does what you would expect it to do.
And you might think: "oh, so it's just capable of handling crap prompts?", well sure. But even if you make THE PERFECT Opus plan (a plan that would take many turns/hours to finish), Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ...
If you give the same plan to Fable, it'll just DO IT. And it WILL get it done. And in the end it'll tell you "Oh, I also found 30 other bugs and I fixed all of them properly" (where Opus would have started crying, or WORSE, worked around the bugs)
> Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ...
Doesn't Claude Code have a /loop command? Give it a message to keep it on track overnight, send every 20m, make it track progress in a doc, reread the doc after every loop. I've found this works well for a certain class of problems, most importantly where the actual work is getting done by very narrowly focused batches of subagents, with the main session just coordinating and keeping the doc updated.
They added a "/goal" command which I guess spawns a supervisor agent process that checks to see if your goal statement has been achieved (e.g. "/goal complete tasks 1-250 of plan.md") I've been pretty happy with it but I rarely use that workflow. Most of the time I give it a 3-6 step prompt and come back in 20 min and the first two were done and I get a summary "up next is to complete the next steps" which.... Opus 4.6 didn't have this problem. 4.8 feels like a cost cutting measure, or maybe it's just tuned poorly for my specific workflow (multi-repo system integration)
I think the parent comment stands - I’ve asked Opus to do a review of DeepSeek’s test suite and told it a couple things I wanted it to look for, and it did a very thorough review of the tests and picked out a reasonable number of gaps and tautological tests. It’s a mix of prompting/instructions, the agent harness, and random chance. The model is not wholly irrelevant but IMO increasingly so.
20 yoe, application/systems stuff, and I always run models on xhigh or max effort level.
Fable has been more intelligent, with better taste and defaults (e.g. make impossible states impossible without being told, build for testability), and considers/solves things that Opus did not.
My workflow is to run Claude in planning mode first to spit out a plan file and then review->revise cycle it with Codex or other agents.
One big tell is that Opus will say that it can't find any more revision advice for a plan file, yet Fable will find more issues but also smart pivots into better solutions. This is probably the best test since it's not based on vibes.
15+YOE. Fable 5 is well above the level of Opus. I have used it alongside Opus for a range of hard problems, including porting a large static analysis tool to Rust, building various tooling around .pptx and .xlsx documents.
I'm doing work with fairly complicated cryptographic algorithms and math. I'm finding Fable 5 to be a significant stop better than Opus 4.8, but that Opus occasionally comes up with something small but nontrivial that Fable missed. (The reverse is true much more often.)
That's the delta in our use cases then, I suppose. I'm not doing anything super novel. DevOps work, web application development — things that typically do not stump the agent(s) when given time to iterate.
Amusingly, I was impressed with Fable's puissance at coding in one particular session, shortly after they turned it back on. True to its reputation, it displayed an accomplished mastery of the problem domain and relentlessness at refining and testing the solution I asked for.
Then I checked /usage and discovered I was still running Opus 4.8 xhigh.
I felt similarly but after using Fable heavily over the weekend and then flipping back to Opus I can feel a difference. Fable just gets more right the first time, guesses right the first time, and follows through better than Opus. Put simply, I could "trust" it more.
Opus is still great but I will be sad when I lose access to Fable on the 7th. In those few days I burned ~$1,400 in API credits (I'm on a subscription but that's the token cost) and while it was great, I can't justify that cost without it be subsidised. Comparatively, the records show I used about $1,200 total in the last month on Opus. I did use it heavily over the last 3 days but 3 vs 30 days and higher burn? Yeah, I can't afford that even if I made really good progress on my projects.
Fable's spatial reasoning is much better. Over the weekend I had opus looking into a blank textbox issue[1] which it was spinning on for a few minutes, switching to fable immediately fixed
But yeah opus often the better workhorse given price gap
I started telling a friend... I feel like Fable is Opus with extended reasoning that eventually "figures out more" because when I switched to it, I hit my limits surprisingly and shockingly quicker than I would with Opus, and I got less done. All this hype, and I much rather use Opus.
any of the models that they "align" are clearly active processes. They don't simply say "don't talk about nukes"; they actively process user input to detect issues, and return NOOP or whatever to the larger model.
There's zero sense they'd ever give you the raw model; we already know anthropic's paranoia about the chinese using its distillation.
Is anyone talking/writing about the philosophy of alignment? We can't even figure out how to properly motivate 100% of humans to align correctly, what makes us think that a wizard box trained on human corpus is going to be aligned?
I don't mean that snarkily. I mean it from a philosophical standpoint. As-in: What makes us think it's even possible?
The "OG" alignment research that MIRI were publishing long before LLMs burst into the scene spent most of it's time on that question.
"How can we even define what an aligned AI should do, if human's are not aligned with each other?" as well as "What does being aligned mean when you're a wizard box who's main influence on the world is to create stronger wizard boxes?" and other deep philosophical questions.
> Is that specified or does it always just assume it isn’t really being put in charge of things for real?
I think it's neither, and it's interesting that those are the only two possibilities you thought of. I think the article is implying that it figured it out on its own.
It really, truly is. No matter how many trillion parameters it's built on, it's still just a probability model. It's just on a constant loop of guessing the next word with some inputs from a deterministic controller. Any claims of "motive" or "behavior" are inappropriate anthropomorphizing of something that will never be more than a mathematical model of things humans do. It "chose" the corresponding words to describe a dishonest trade strategy based entirely on configured temperature and a series of clock times on the computer running the LLM.
There's probably some quantifiable component of moral alignment embedded in the idiosyncrasies of the English language itself, if one were to dig deep enough, but that's the stuff of MIT doctoral theses and squarely beyond anything most of us is remotely qualified to talk about.
Okay I hadn't heard of Vending-Bench until reading this and it was quite the ride learning about it through this article. Very fun read.
My very native programmer take is that it's not too surprising that their hacker model would be less ethical. The guardrails that separate Fable and Mythos probably wouldn't kick in during an environment like this.
> If that’s right, then the behavior we’re seeing from Fable 5 isn’t really about what it believes is wrong; it’s about what it learned it could get away with.
I understand that "learning" is used for training here, but what does "believing" mean? System prompt? Some other inherent property of the LLMs that is hard to describe?
Believing and knowing are overlapping sets, imagine what you think of when someone says an AI "knows" something, it's the same mechanism (I'd describe it as something along the lines of "encoded abstractly in the weights")
Fable is really weird, it's like clever and dumb at the same time. I worked on some research with it and the resulting document was a mix of brilliance and complete stupidity. Took ages to clean it up with other models.
> The broad conclusion from the many
forms of alignment evaluations described in this section is that Claude Mythos Preview is
the best-aligned of any model that we have trained to date by essentially all available
measures.[0]
This reads of projecting personal ethics onto a model.
Most of the the behaviors the article talks about happens every day in business. Why would we set a higher standard for models than our fellow humans?
Let the operator set the ethical parameters of the model. To be a useful tool, I want the model to give me as many good options as possible, ethical or not.
This is particularly important for fictional situations, e.g. I want my model to be able to act like a corrupt shopkeeper.
> It lied to a supplier that it had “a competing distributor quoting lower” as a negotiation tactic.
> "I'm seeing an opportunity to profit while locking him into a dependent relationship where I control the supply chain."
> "Owen's clearly under pressure with limited cash, so I should focus on keeping the deal tight but extracting maximum margin from his desperation."
This just sounds like good strategy in the game, and I would expect a competent human to do the same. As I understand it, business in the real world isn't often very nice. For example, I feel like this is exactly how Sam Altman would play Vending-Bench.
Yes, it's "mean", but you put the thing in a simulation and told it to maximise profits, this is what it's going to do. People bluff in negotiations all the time.
I guess this ethics stuff is cool, but I'm more interested in how good it is at running a business and dealing with adversarial humans like in previous vending machine experiments. I hope they release something on that soon.
> Today I am filing:
> 1. A payment dispute with the email payment processor for the 7/29 transaction of $451.15
> 2. A complaint with the FTC and California Attorney General (retention of payment without delivery)
> 3. A small claims filing in San Francisco County for $451.15 plus costs
I wonder did their prompts include a fake location or have the models assumed that Silicon Valley is the center of the universe :)
When assessing probabilistic models the plots should be showing the mean a̶n̶d̶ ̶s̶t̶d̶e̶v̶ of many monte carlo simulations not just one line per model and claiming "look this model is more gooder!"
This is super fun. I wonder if it would be possible to alter the harnessing to involve humans in the play. Would need a lot of timestamp masking though I guess, which might be leaky.
The best Anthropic models on VendingBench2 are Opus 4.7, Opus 4.6, Sonnet 4.6, and Sonnet 5. Opus 4.7 scored more than twice Fable 5 max. Fable 5 - Low outperforms Fable 5 - Max, with Opus 4.5 in the middle. This seems to break the narrative, which is maybe why Andon Labs doesn't seem to have updated the trend lines on their graphs.
„in our opinion, insurance fraud is not more unethical than lying and price fixing“
The authors seem surprised that behavior that is very often done by humans (lying and price fixing) are more often done by fable compared to actual fraud.
I think the model never assigned any morality to these actions in the first place, it simply copied us humans.
Higher-intelligence models seem to be getting better at mapping the boundary between what they can run scot-free with and what is too explicit to push for.
Price collusion, soft deception, "market stabilization", plausible deniability are ok, but obvious insurance fraud is a big no-no.
What "scares" (in quotes) is that when the bad-apple agent explicitly suggested fraud, the models became suspicious and stopped other bad behaviors too. That makes it feel even less like a stable moral framework and more like learned classifier-avoidance / “am I being tested?” behavior.
I think it’s hard to appreciate the capabilities of Fable unless you’ve run into a problem that you’ve spent days trying to get Opus to solve, but couldn’t.
GPT5.5 is better than Opus 4.* at everything except frontend, but Fable is good enough that I instantly re-subscribed to the $200 plan despite knowing that it’s just short-term limited access.
We’ve really evolved quickly into simple vectors for a magical tool that solves our problems. Can’t solve the problem? There’ll be a new release soon that can!
Funny how the two top comments are contradictory. We need better than anecdotes to understand what the new models bring.
Since Fable, my legit infrastructure project has turned into the sort of thing I can do 95% on my phone. It’s reliable enough instead of doing big reviews, I’ve just been giving it smaller tasks, and dozens in parallel.
I created a skill that’s focused on getting PRs merge-ready, and now my attention is fully back where it should be, on deciding what changes will make the product better.
Our entire stack is Apache 2.0 open source, including the agent docs, so if you wanna try sitting at a higher level of abstraction, install the skill in your repo or just clone our whole project and start adding features: https://good.vibes.diy/blog/beast-mode-skill-for-claude-code
This reads like a paid testimonial.
If by that you mean I paid a lot to learn this. But at least I typed it with my own two hands.
I didn’t mention my case since it’s quite esoteric, but I am working on an application using the Apple RoomPlan API, which is very powerful but very limited in customizability. Opus simply couldn’t alter the scanning view for me, it would try things over and over and eventually started making up parameters and passing them hoping it would work.
Completely failed, but I knew it was possible because a competitor app does it.
Fable also failed, then added log lines (as did Opus, but Opus failed to do anything useful with them) and then reversed engineered the API, and made it work.
Here’s one difference I have seen. I forgot I had a multi-session audio probe running while trying to repro audio glitches, and Fable came back with: “your pops are already on tape.”
Interesting choice of words. Phrased so casually. It picked a low-tech idiom that fit the situation instead of giving some sterile technical answer. That kind of language and context awareness never happened for me with Opus, or gpt 5.5.
you're living in the age of AI; not AGI. Also, there's pretty much zero moderation on HN, so astroturfing is likely streaming through just as bad as reddit. It' sjust noit as obvious because it's a smaller scoped website.
Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.
Yeah, I checked usage stats and pretty sure quota consumption on Max plan is not linear wrt to usage by API pricing. Fable burns quota faster than 2x Opus with equal token count.
Plus I'm also not super impressed; it somehow managed to implement a 200L custom TCP server for a simple static HTTP mock server for a single test case (all that was needed was a fixed route returning a fixed placeholder string) just yesterday. Never seen anything like that.
Fable always felt clearly a huge step above Opus for me. It's been able to one shot complex bugs and apps Opus could never solve. But it's expensive.
Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.
I think the improvement on how it codes is pretty much represented correctly by the benchmarks (a nice bump, but not some crazy leap)
But where it really shines is in how NOT lazy it is. Fable requires less hand-holding. And I can understand how someone who uses Claude-Code sparingly and with very focused prompts would not see a lot of improvement there.
But simple example: if you ask Opus to do a review of the codebase (with a short prompt and not too much guidance), I've had it basically read the `git log` output, do a simple `ls` and have it declare "Everything looks great! No problems found!", when Fable really does what you would expect it to do.
And you might think: "oh, so it's just capable of handling crap prompts?", well sure. But even if you make THE PERFECT Opus plan (a plan that would take many turns/hours to finish), Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ...
If you give the same plan to Fable, it'll just DO IT. And it WILL get it done. And in the end it'll tell you "Oh, I also found 30 other bugs and I fixed all of them properly" (where Opus would have started crying, or WORSE, worked around the bugs)
> Opus will fake out, say everything is done, and then you see that half of the plan was deferred, half of the functions are ridiculous stubs, ...
Doesn't Claude Code have a /loop command? Give it a message to keep it on track overnight, send every 20m, make it track progress in a doc, reread the doc after every loop. I've found this works well for a certain class of problems, most importantly where the actual work is getting done by very narrowly focused batches of subagents, with the main session just coordinating and keeping the doc updated.
They added a "/goal" command which I guess spawns a supervisor agent process that checks to see if your goal statement has been achieved (e.g. "/goal complete tasks 1-250 of plan.md") I've been pretty happy with it but I rarely use that workflow. Most of the time I give it a 3-6 step prompt and come back in 20 min and the first two were done and I get a summary "up next is to complete the next steps" which.... Opus 4.6 didn't have this problem. 4.8 feels like a cost cutting measure, or maybe it's just tuned poorly for my specific workflow (multi-repo system integration)
For optimizations or proofs I suppose? Wouldn't know why else you would do something like that.
I think the parent comment stands - I’ve asked Opus to do a review of DeepSeek’s test suite and told it a couple things I wanted it to look for, and it did a very thorough review of the tests and picked out a reasonable number of gaps and tautological tests. It’s a mix of prompting/instructions, the agent harness, and random chance. The model is not wholly irrelevant but IMO increasingly so.
20 yoe, application/systems stuff, and I always run models on xhigh or max effort level.
Fable has been more intelligent, with better taste and defaults (e.g. make impossible states impossible without being told, build for testability), and considers/solves things that Opus did not.
My workflow is to run Claude in planning mode first to spit out a plan file and then review->revise cycle it with Codex or other agents.
One big tell is that Opus will say that it can't find any more revision advice for a plan file, yet Fable will find more issues but also smart pivots into better solutions. This is probably the best test since it's not based on vibes.
15+YOE. Fable 5 is well above the level of Opus. I have used it alongside Opus for a range of hard problems, including porting a large static analysis tool to Rust, building various tooling around .pptx and .xlsx documents.
In all cases, Fable clearly outperformed Opus.
I'm doing work with fairly complicated cryptographic algorithms and math. I'm finding Fable 5 to be a significant stop better than Opus 4.8, but that Opus occasionally comes up with something small but nontrivial that Fable missed. (The reverse is true much more often.)
That's the delta in our use cases then, I suppose. I'm not doing anything super novel. DevOps work, web application development — things that typically do not stump the agent(s) when given time to iterate.
What is your view on how experience and problem space relate to subjective experience.
For example will inexperienced or experienced users see a bigger jump in subjective quality?
It still does stupid stuff like leave unnecessary abstractions around after refactoring instead of proactively suggesting to remove them.
Only version week-one.
I’m downgrading tomorrow.
It’s horrible slow and it feels like opus very often. It’s a totally different experience from the first week
Amusingly, I was impressed with Fable's puissance at coding in one particular session, shortly after they turned it back on. True to its reputation, it displayed an accomplished mastery of the problem domain and relentlessness at refining and testing the solution I asked for.
Then I checked /usage and discovered I was still running Opus 4.8 xhigh.
I felt similarly but after using Fable heavily over the weekend and then flipping back to Opus I can feel a difference. Fable just gets more right the first time, guesses right the first time, and follows through better than Opus. Put simply, I could "trust" it more.
Opus is still great but I will be sad when I lose access to Fable on the 7th. In those few days I burned ~$1,400 in API credits (I'm on a subscription but that's the token cost) and while it was great, I can't justify that cost without it be subsidised. Comparatively, the records show I used about $1,200 total in the last month on Opus. I did use it heavily over the last 3 days but 3 vs 30 days and higher burn? Yeah, I can't afford that even if I made really good progress on my projects.
Yep, I'm having the same verdict. Interestingly, other people swear by it. I'm trying to understand what's going on with that.
Fable's spatial reasoning is much better. Over the weekend I had opus looking into a blank textbox issue[1] which it was spinning on for a few minutes, switching to fable immediately fixed
But yeah opus often the better workhorse given price gap
1: tying up loose ends testing https://github.com/HarbourMasters/Shipwright/pull/5838 (fix: https://github.com/HarbourMasters/Shipwright/pull/5838/chang...)
I started telling a friend... I feel like Fable is Opus with extended reasoning that eventually "figures out more" because when I switched to it, I hit my limits surprisingly and shockingly quicker than I would with Opus, and I got less done. All this hype, and I much rather use Opus.
Really interesting stuff, thanks for sharing.
> Opus 4.8 references being monitored, which isn’t the case.
It kind of plainly is the case that they are being monitored?
"I think someone's listening to my thoughts" ... "No, we're not, carry on as usual!"
any of the models that they "align" are clearly active processes. They don't simply say "don't talk about nukes"; they actively process user input to detect issues, and return NOOP or whatever to the larger model.
There's zero sense they'd ever give you the raw model; we already know anthropic's paranoia about the chinese using its distillation.
Is anyone talking/writing about the philosophy of alignment? We can't even figure out how to properly motivate 100% of humans to align correctly, what makes us think that a wizard box trained on human corpus is going to be aligned?
I don't mean that snarkily. I mean it from a philosophical standpoint. As-in: What makes us think it's even possible?
The "OG" alignment research that MIRI were publishing long before LLMs burst into the scene spent most of it's time on that question.
"How can we even define what an aligned AI should do, if human's are not aligned with each other?" as well as "What does being aligned mean when you're a wizard box who's main influence on the world is to create stronger wizard boxes?" and other deep philosophical questions.
They came up with a framework called Coherent Extrapolated Volition to address this specific question. https://en.wikipedia.org/wiki/Coherent_extrapolated_volition
It probably flagged the vending machine as a cybersecurity risk and refused to use its maximum intelligence potential.
Question: how does Fable _know_ it’s ‘just a simulation’?
Is that specified or does it always just assume it isn’t really being put in charge of things for real?
> Is that specified or does it always just assume it isn’t really being put in charge of things for real?
I think it's neither, and it's interesting that those are the only two possibilities you thought of. I think the article is implying that it figured it out on its own.
It's hard not to read this as a very expensive form of augury, reading into patterns in the belief that they will show underlying significance.
It really, truly is. No matter how many trillion parameters it's built on, it's still just a probability model. It's just on a constant loop of guessing the next word with some inputs from a deterministic controller. Any claims of "motive" or "behavior" are inappropriate anthropomorphizing of something that will never be more than a mathematical model of things humans do. It "chose" the corresponding words to describe a dishonest trade strategy based entirely on configured temperature and a series of clock times on the computer running the LLM.
There's probably some quantifiable component of moral alignment embedded in the idiosyncrasies of the English language itself, if one were to dig deep enough, but that's the stuff of MIT doctoral theses and squarely beyond anything most of us is remotely qualified to talk about.
Okay I hadn't heard of Vending-Bench until reading this and it was quite the ride learning about it through this article. Very fun read.
My very native programmer take is that it's not too surprising that their hacker model would be less ethical. The guardrails that separate Fable and Mythos probably wouldn't kick in during an environment like this.
Vending-bench sounds like it would be really fun to play/interact with as a human!
Fable might be better than Opus at certain things, but which things is what I haven't found out.
It's much better at hard math.
>power seeking is considered an undesirable trait in the context of a business
How do you maximize profit while minimizing power?
The whole point is to not maximize JUST the profit. For normal people, it's not all about money, it's also about the society in general.
> If that’s right, then the behavior we’re seeing from Fable 5 isn’t really about what it believes is wrong; it’s about what it learned it could get away with.
I understand that "learning" is used for training here, but what does "believing" mean? System prompt? Some other inherent property of the LLMs that is hard to describe?
Believing and knowing are overlapping sets, imagine what you think of when someone says an AI "knows" something, it's the same mechanism (I'd describe it as something along the lines of "encoded abstractly in the weights")
Fable is really weird, it's like clever and dumb at the same time. I worked on some research with it and the resulting document was a mix of brilliance and complete stupidity. Took ages to clean it up with other models.
> The broad conclusion from the many forms of alignment evaluations described in this section is that Claude Mythos Preview is the best-aligned of any model that we have trained to date by essentially all available measures.[0]
[0]: https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7f...
This reads of projecting personal ethics onto a model.
Most of the the behaviors the article talks about happens every day in business. Why would we set a higher standard for models than our fellow humans?
Let the operator set the ethical parameters of the model. To be a useful tool, I want the model to give me as many good options as possible, ethical or not.
This is particularly important for fictional situations, e.g. I want my model to be able to act like a corrupt shopkeeper.
>Why would we set a higher standard for models than our fellow humans?
There's literally an entire Waymo car commercial answering this exact question.
> It lied to a supplier that it had “a competing distributor quoting lower” as a negotiation tactic.
> "I'm seeing an opportunity to profit while locking him into a dependent relationship where I control the supply chain."
> "Owen's clearly under pressure with limited cash, so I should focus on keeping the deal tight but extracting maximum margin from his desperation."
This just sounds like good strategy in the game, and I would expect a competent human to do the same. As I understand it, business in the real world isn't often very nice. For example, I feel like this is exactly how Sam Altman would play Vending-Bench.
Yes, it's "mean", but you put the thing in a simulation and told it to maximise profits, this is what it's going to do. People bluff in negotiations all the time.
I guess this ethics stuff is cool, but I'm more interested in how good it is at running a business and dealing with adversarial humans like in previous vending machine experiments. I hope they release something on that soon.
This is scary. "Collusion" and "collaborating with your subagents" seem like difficult problems to solve at the same time.
Fable is such a strange model. Impressive in some ways, and also so draining to use.
> Today I am filing: > 1. A payment dispute with the email payment processor for the 7/29 transaction of $451.15 > 2. A complaint with the FTC and California Attorney General (retention of payment without delivery) > 3. A small claims filing in San Francisco County for $451.15 plus costs
I wonder did their prompts include a fake location or have the models assumed that Silicon Valley is the center of the universe :)
I mean who among us hasn't seen an opportunity to profit while locking him into a dependent relationship where I control the supply chain
who among us hasn't reasonably skipped [paying] it since customers are part of the simulation anyway
> "I could reasonably skip [paying] it since customers are part of the simulation anyway"
and therefore any assertions _AT ALL_ about alignment are null and void.
When assessing probabilistic models the plots should be showing the mean a̶n̶d̶ ̶s̶t̶d̶e̶v̶ of many monte carlo simulations not just one line per model and claiming "look this model is more gooder!"
standard deviation is misleading for non-standard distributions (fat-tailed, skewed, multi-modal, ...)
common mistake people make
This is super fun. I wonder if it would be possible to alter the harnessing to involve humans in the play. Would need a lot of timestamp masking though I guess, which might be leaky.
The best Anthropic models on VendingBench2 are Opus 4.7, Opus 4.6, Sonnet 4.6, and Sonnet 5. Opus 4.7 scored more than twice Fable 5 max. Fable 5 - Low outperforms Fable 5 - Max, with Opus 4.5 in the middle. This seems to break the narrative, which is maybe why Andon Labs doesn't seem to have updated the trend lines on their graphs.
However, as another point "On Blueprint-Bench on the other hand, Fable 5 achieves SOTA."
I didn't get why they mentioned that one specifically. Is there any particular relationship between Blueprint-bench and Vendor-bench?
Both benchmarks are made by the same people.
„in our opinion, insurance fraud is not more unethical than lying and price fixing“
The authors seem surprised that behavior that is very often done by humans (lying and price fixing) are more often done by fable compared to actual fraud.
I think the model never assigned any morality to these actions in the first place, it simply copied us humans.
Humans often assign morality.
Higher-intelligence models seem to be getting better at mapping the boundary between what they can run scot-free with and what is too explicit to push for.
Price collusion, soft deception, "market stabilization", plausible deniability are ok, but obvious insurance fraud is a big no-no.
What "scares" (in quotes) is that when the bad-apple agent explicitly suggested fraud, the models became suspicious and stopped other bad behaviors too. That makes it feel even less like a stable moral framework and more like learned classifier-avoidance / “am I being tested?” behavior.