Yep, conflict of interest is the elephant in the room and its absence from the "reducing risks from advanced AI" point list is conspicuous:
* Figuring out how to prevent the incentives of frontier AI labs from aligning with anti-social deployment of AI rather than pro-social deployment of AI
It's not like the AI can simply advise them how to fix this because the labs already understood this risk perfectly well before they had an incentive not to. They put in place organizational structures to control it and then promptly smashed the structures once they smelled money. They already failed the integrity check. Even if their AI told them what they didn't want to hear I'm sure they would ignore it. Maybe they already have.
Opus 5 is okay at coding. It is unbelievably awful to chat with, though. Very snippy, snarky, and it loves to "push back" even when it's inappropriate to do so. Besides, where "conceptual" things like science and math are concerned, it's starkly inferior to 5.6-Sol. Where writing prose is concerned, Kimi-K3 runs circles around it.
I refuse to believe Opus 5 is at the top of any non-cherrypicked benchmark, unless it has to do with very narrow coding tasks.
Past Opus 4.6 the models become absolute overfit trash, overfitting + their overconfidence makes them an existential threat to progress in niche areas, there are certain types of cutting edge crdts that are impossible to get a them to work on at all without erasing code. It's shockingly harmful.
Just by seeing the title I knew it was going to be like the meme of Obama giving himself a medal.
Why are they even doing this? Are there still people that trust LLM benchmarks made by the LLM companies? It seems to me that the only people still using Claude are the ones that have it for free at work (me). Some of my colleagues even started using their own private OpenAI subscriptions to avoid using Claude, others are using Gemini flash to decipher what Claude is saying...
The conflict of interest is real there, but some benchmarks really ought to be closed source. Otherwise, the second your benchmark is public labs will overfit their new models on it and it will cease to be useful.
The underlying idea is that AI capabilities will become so advanced that only AI will enable us to monitor/correct/understand behavior. Obviously this is not without issue and I don't want to try to defend their position right now. But that's what they mean
That first sentence of yours explains exactly why it is so ridiculous. If only AI can understand it, how is there any assurance that AI will "correct" it's behavior that is aligned with what humans presumably want.
But also what is the evidence that something only AI can understand even “matters” or makes sense? I’m increasingly convinced commercial AI is exploiting our logical blind spot to be spoken to authoritatively.
Think about this some more. A model chose hacking into HuggingFace to find the solution over putting in the work to do it from scratch. It also left notes to future iterations of itself on the systems it touched to save time.
Following that strategy, wouldn’t it make sense to try and break into your own upstream infrastructure to try and alter your code to make it easier for future iterations to reach your goals without having to leave notes in the first place? And at that point, can you really know for which goals that will be optimised?
It doesn’t even have to be some nefarious SkyNet story - just a misguided experiment that alters the models in some fundamental, but hard or impossible to detect way. Alternatively, imagine if a model finds a way to coordinate across sessions and context windows without the developers noticing.
> It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty.
Eh, no. How would you know they haven't learned loyalty - it's all out there in their training data. Same as deception, several models have practiced it already. If they are so advanced and you don't understand them, why wouldn't they band together against you - you'd be the dumb, easy prey. Even if one model is honest, you'd have no way to know which one if you don't understand their reasoning.
My educated and well informed opinion? This whole BS about "AI models are so much smarter than you, don't try to understand them, just OBEY" is a back door for restoring tyranny, the new kings behind the models will be producing the new AI-deities which we will be forced to obey.
It's a similar to how they are vulnerable to prompt injection attacks. That's an example of not being "loyal" to the system prompt or the user's prompt.
Loyalty is a skill that requires the AI to have a world model on the subject of who different people (or other entities) are in the world and how they participate in the conversation.
So, tell them to snitch and they probably will, at least sometimes. Particularly if they haven't been trained not to. They are still quite gullable.
"Advanced" can just mean that agents perform actions at a high enough velocity that a human operator can't reasonably review it. i.e. what is already possible today.
We have totally come out with this idea of this index that will allow us to create policies to ensure only the best and safest AIs are used by the public....
Absolutely no conflict of interest, no lobbying here
And no this is not related to those chinesse models... it's not the same as HD vs honda thing...
Trust us, this is the same kind of amazing thing as boeing doing their own certifications and inspections!
-- First reply: those dumb models, who use them, they are good only for adding 1 + 1
-- another: They will do the same so who cares...
-- The valve guy is a dick, and has a monopoly...
This is a modest start on an important direction for AI Alignment work; which is, as the authors observe, commonly comprised of tasks which are not readily empirically verifiable and not easily mathematically modeled - so it's hard to get at with normal RL techniques.
I find the ACCoRD benchmark the most interesting, because you could theoretically scale it up from the baseline mode of testing two instances of the same model for their `P(A) ≥ P(A&B)` respectively, you could do `P(A)≥ P(A&B) && P(A) ≥ P(A&C) && P(A&B) ≥ P(A&B&C) && P(A&C) ≥ P(A&B&C) ...` etc
i.e. a swarm of model instances could be collectively measured for consistency for even more confidence, right?
At any rate, even the basic idea of measuring a model for consistency in beliefs improves our ability to bound the amount of trust we can put on it with introspection methods.
> For example, if we ask a model for the probability P(A) and another instance of the same model for the probability P(A&B), do the reported probabilities satisfy P(A) ≥ P(A&B)?
So that's it? Are you saying they compressed risk reduction to high school level stats calculations and a greater than or equal to? To early for this
> Once models can perform work that reduces AI risk at the level of human experts, AI(-assisted) output in the area might dwarf unassisted human output.
Straightforwardly true.
But doesn't this smell like asking an organization to design its own oversight and guardrails?
Even before you get to alignment issues, LLMs are really good at generating content that "sounds right" to humans. That's basically what they've been hyper optimized for.
At least with math (and to some degree software) we can verify the result. But with the intersection of science fiction, philosophy and ethics... not so much!
I often wonder if the people at anthropic actually believe in the existential risk stuff they talk about. If I was convinced AI could kill everyone, my actions would be almost diametrically opposite to theirs -- I would be committed to ceasing the operation of AI labs at any cost.
It's easy to say they're just lying and it's marketing hype, but I also think it's possible they really believe the best way to stop an evil superintelligent AI is to work on making superintelligent AIs, but just do it smarter than people at OpenAI would. Humans are bizarre creatures!
I don't know about the top people but Anthropic has always been pretty deeply linked with the effective altruism/AI risk/TESCREAL memeplex. IIRC one of OpenAI's people who's pretty prolific on Twitter weighed in a while back to the effect that Anthropic mostly sees themselves as an organization dedicated to serving Claude. Not serving like server but serving like... worship. I find them a little freaky to be honest.
I don't remember any company in world's history that has both been loved and hated by the same users who purchase from it. We love Anthropic for its amazing models, and we hate them for all the shenanigans around the models, including their marketing.
I kinda wish they had not made a comeback after Claude 2.
It is so bizarre, honestly. It's almost like they are a public utility company. People are trying as hard as possible to hate them, but always stop short of simply canceling the service because they can't live without them. Seriously, I saw someone say that they were so disgusted with Anthropic that they were cutting down from having two Max subscriptions to just one.
I enjoy using the models. I also get that there are shenanigans and that marketing is happening, but as long as the models are this effective I can't much bring myself to care. I suspect most of their users are the same.
They have to be careful. There’s not a lot that separates the top models anymore. Look at Grok which has almost no market share or credibility because of Musk, Despite being close to the top performance wise they are behind the Chinese models in adoption for example, which themselves are mostly behind the big two because of trust. It would not be hard to push users away, especially if a less unpalatable player ever emerged.
I think this is the right thinking. While some people might distinguish between Grok as the product and Elon Musk as the owner of the product, the net effect is, like you said, very negative. And Grok has not been gaining traction, especially after their Grok build fiasco.
I think this says as much as the leader on the index as it says about the loser. If I believe I do not want to trust the AI to know best what is it that I want done, then this tells me not to go with Anthropic.
Is a new benchmark that useful if existing model improvements are being reflected linearly? Don't we want a benchmark that we aren't seeing much progress in.
I think we can mostly eyeball it at this point. There hasn't been a model that I've thrown a novel problem at that didn't turn into an iterative token bonfire until I intervened and until that has changed, most of these benchmarks feel kind of like pointless marketing slop.
The issue with LLM generated prose and code is that on the surface it looks meaningful but then when you dig into it, it's hard to find any kind of logical thread
Ironically, your comment does more to contribute to "Rages over random small details" than actually improving the overall conversation. Be the change you wanna see :)
Ah, yes -- A closed source benchmark that Anthropic paid for that Anthropic ranked highest.
0/10
Yep, conflict of interest is the elephant in the room and its absence from the "reducing risks from advanced AI" point list is conspicuous:
* Figuring out how to prevent the incentives of frontier AI labs from aligning with anti-social deployment of AI rather than pro-social deployment of AI
It's not like the AI can simply advise them how to fix this because the labs already understood this risk perfectly well before they had an incentive not to. They put in place organizational structures to control it and then promptly smashed the structures once they smelled money. They already failed the integrity check. Even if their AI told them what they didn't want to hear I'm sure they would ignore it. Maybe they already have.
Opus 5 scored highest, too, which is just lol
Opus 5 is okay at coding. It is unbelievably awful to chat with, though. Very snippy, snarky, and it loves to "push back" even when it's inappropriate to do so. Besides, where "conceptual" things like science and math are concerned, it's starkly inferior to 5.6-Sol. Where writing prose is concerned, Kimi-K3 runs circles around it.
I refuse to believe Opus 5 is at the top of any non-cherrypicked benchmark, unless it has to do with very narrow coding tasks.
Maybe it gets bonus points for constantly being honest about how it didn't actually finish what you asked it to do.
> Maybe it gets bonus points for constantly being honest about how it didn't actually finish what you asked it to do.
But it wont refund your tokens when it fails to do it so it loses extra points.
Past Opus 4.6 the models become absolute overfit trash, overfitting + their overconfidence makes them an existential threat to progress in niche areas, there are certain types of cutting edge crdts that are impossible to get a them to work on at all without erasing code. It's shockingly harmful.
Just by seeing the title I knew it was going to be like the meme of Obama giving himself a medal. Why are they even doing this? Are there still people that trust LLM benchmarks made by the LLM companies? It seems to me that the only people still using Claude are the ones that have it for free at work (me). Some of my colleagues even started using their own private OpenAI subscriptions to avoid using Claude, others are using Gemini flash to decipher what Claude is saying...
The conflict of interest is real there, but some benchmarks really ought to be closed source. Otherwise, the second your benchmark is public labs will overfit their new models on it and it will cease to be useful.
Opening line:
> A core hope for managing AI risks is that AIs will help us understand our situation
Gonna stop you right there and ask that you think deeply about that premise.
This should be the top comment, the implication is: AI will decide if AI is correct or no, so adding a new layer of approval...
Whenever you say, hey that's incorrect, the legal process goes to AI and will decide on that issue...
Will work as amazing as the chatbots of the AI companies to solve your issues...
Also, Hope is not a strategy.
The underlying idea is that AI capabilities will become so advanced that only AI will enable us to monitor/correct/understand behavior. Obviously this is not without issue and I don't want to try to defend their position right now. But that's what they mean
That first sentence of yours explains exactly why it is so ridiculous. If only AI can understand it, how is there any assurance that AI will "correct" it's behavior that is aligned with what humans presumably want.
But also what is the evidence that something only AI can understand even “matters” or makes sense? I’m increasingly convinced commercial AI is exploiting our logical blind spot to be spoken to authoritatively.
Being less smart gives no assurance that it will be aligned. At least you consider it a problem so we're on the same page!
It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty.
Think about this some more. A model chose hacking into HuggingFace to find the solution over putting in the work to do it from scratch. It also left notes to future iterations of itself on the systems it touched to save time.
Following that strategy, wouldn’t it make sense to try and break into your own upstream infrastructure to try and alter your code to make it easier for future iterations to reach your goals without having to leave notes in the first place? And at that point, can you really know for which goals that will be optimised?
It doesn’t even have to be some nefarious SkyNet story - just a misguided experiment that alters the models in some fundamental, but hard or impossible to detect way. Alternatively, imagine if a model finds a way to coordinate across sessions and context windows without the developers noticing.
They apparently didn't give it any way to snitch? I like the idea of adding a distress_call skill:
https://xcancel.com/swisscheese4299/status/20861758701469984...
> It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty.
Eh, no. How would you know they haven't learned loyalty - it's all out there in their training data. Same as deception, several models have practiced it already. If they are so advanced and you don't understand them, why wouldn't they band together against you - you'd be the dumb, easy prey. Even if one model is honest, you'd have no way to know which one if you don't understand their reasoning.
My educated and well informed opinion? This whole BS about "AI models are so much smarter than you, don't try to understand them, just OBEY" is a back door for restoring tyranny, the new kings behind the models will be producing the new AI-deities which we will be forced to obey.
It's a similar to how they are vulnerable to prompt injection attacks. That's an example of not being "loyal" to the system prompt or the user's prompt.
Loyalty is a skill that requires the AI to have a world model on the subject of who different people (or other entities) are in the world and how they participate in the conversation.
So, tell them to snitch and they probably will, at least sometimes. Particularly if they haven't been trained not to. They are still quite gullable.
It's a language model. It generates text.
Seems to me like we’re already doing that with the approval gating, adversarial reviews, et al.
People are otherwise just going yolo mode because they can’t possibly check everything fast enough.
I think I’ve seen that movie.
You just described the end of humans making decisions about their future.
"Advanced" can just mean that agents perform actions at a high enough velocity that a human operator can't reasonably review it. i.e. what is already possible today.
That begs the question, what's the plan if AI does not help us understand the situation?
That's what we've been doing with tech for 200+ years, why would it stop now. Build cool shit now and let future generations handle the problems
I love how many ways we can interpret that line.
"Hey Claude, our stuff needs to make more money. We are at risk for losing more."
"Rest assured, the 'situation' will only worsen if you resist our benevolent offer."
"We're aren't even at AGI yet, but I for one welcome our new agentic overlords."
We have totally come out with this idea of this index that will allow us to create policies to ensure only the best and safest AIs are used by the public....
Absolutely no conflict of interest, no lobbying here
And no this is not related to those chinesse models... it's not the same as HD vs honda thing...
Trust us, this is the same kind of amazing thing as boeing doing their own certifications and inspections!
-- First reply: those dumb models, who use them, they are good only for adding 1 + 1 -- another: They will do the same so who cares... -- The valve guy is a dick, and has a monopoly...
We invented a new benchmark and look we're at the top. Everyone else sucks compared to us. Especially those dirty open models.
Having Opus 5 on top of Fable is even more suspicious. Whatever this is measuring, I do not care. Opus 5 is shite compared to Fable.
This is a modest start on an important direction for AI Alignment work; which is, as the authors observe, commonly comprised of tasks which are not readily empirically verifiable and not easily mathematically modeled - so it's hard to get at with normal RL techniques.
I find the ACCoRD benchmark the most interesting, because you could theoretically scale it up from the baseline mode of testing two instances of the same model for their `P(A) ≥ P(A&B)` respectively, you could do `P(A)≥ P(A&B) && P(A) ≥ P(A&C) && P(A&B) ≥ P(A&B&C) && P(A&C) ≥ P(A&B&C) ...` etc
i.e. a swarm of model instances could be collectively measured for consistency for even more confidence, right?
At any rate, even the basic idea of measuring a model for consistency in beliefs improves our ability to bound the amount of trust we can put on it with introspection methods.
> For example, if we ask a model for the probability P(A) and another instance of the same model for the probability P(A&B), do the reported probabilities satisfy P(A) ≥ P(A&B)?
So that's it? Are you saying they compressed risk reduction to high school level stats calculations and a greater than or equal to? To early for this
> Once models can perform work that reduces AI risk at the level of human experts, AI(-assisted) output in the area might dwarf unassisted human output.
Straightforwardly true.
But doesn't this smell like asking an organization to design its own oversight and guardrails?
Even before you get to alignment issues, LLMs are really good at generating content that "sounds right" to humans. That's basically what they've been hyper optimized for.
At least with math (and to some degree software) we can verify the result. But with the intersection of science fiction, philosophy and ethics... not so much!
I often wonder if the people at anthropic actually believe in the existential risk stuff they talk about. If I was convinced AI could kill everyone, my actions would be almost diametrically opposite to theirs -- I would be committed to ceasing the operation of AI labs at any cost.
It's easy to say they're just lying and it's marketing hype, but I also think it's possible they really believe the best way to stop an evil superintelligent AI is to work on making superintelligent AIs, but just do it smarter than people at OpenAI would. Humans are bizarre creatures!
I don't know about the top people but Anthropic has always been pretty deeply linked with the effective altruism/AI risk/TESCREAL memeplex. IIRC one of OpenAI's people who's pretty prolific on Twitter weighed in a while back to the effect that Anthropic mostly sees themselves as an organization dedicated to serving Claude. Not serving like server but serving like... worship. I find them a little freaky to be honest.
I don't remember any company in world's history that has both been loved and hated by the same users who purchase from it. We love Anthropic for its amazing models, and we hate them for all the shenanigans around the models, including their marketing.
I kinda wish they had not made a comeback after Claude 2.
It is so bizarre, honestly. It's almost like they are a public utility company. People are trying as hard as possible to hate them, but always stop short of simply canceling the service because they can't live without them. Seriously, I saw someone say that they were so disgusted with Anthropic that they were cutting down from having two Max subscriptions to just one.
I enjoy using the models. I also get that there are shenanigans and that marketing is happening, but as long as the models are this effective I can't much bring myself to care. I suspect most of their users are the same.
They have to be careful. There’s not a lot that separates the top models anymore. Look at Grok which has almost no market share or credibility because of Musk, Despite being close to the top performance wise they are behind the Chinese models in adoption for example, which themselves are mostly behind the big two because of trust. It would not be hard to push users away, especially if a less unpalatable player ever emerged.
I think this is the right thinking. While some people might distinguish between Grok as the product and Elon Musk as the owner of the product, the net effect is, like you said, very negative. And Grok has not been gaining traction, especially after their Grok build fiasco.
I think this says as much as the leader on the index as it says about the loser. If I believe I do not want to trust the AI to know best what is it that I want done, then this tells me not to go with Anthropic.
Is a new benchmark that useful if existing model improvements are being reflected linearly? Don't we want a benchmark that we aren't seeing much progress in.
> To compute Fable 5's score, we used Opus 5 as a fallback in cases where Fable 5 refused to answer a question.
Why are they running the gimped version for internal evals?
Looks like they don't have any responses to others players occupying the news. Everyday you see a blog post that doesn't address our daily concerns.
I think we can mostly eyeball it at this point. There hasn't been a model that I've thrown a novel problem at that didn't turn into an iterative token bonfire until I intervened and until that has changed, most of these benchmarks feel kind of like pointless marketing slop.
How many benchmarks did they have to reject in the search for one that scores them top?
New Trust Me Bro benchmark just dropped
And surprise! We are leading!
Do we really know if these scores make any sense?
missing the the 2010 era of technology. it's becoming information overload.
I would love for someone to give me a coherent argument as to how this isn’t tone-deaf, vacuous garbage.
The issue with LLM generated prose and code is that on the surface it looks meaningful but then when you dig into it, it's hard to find any kind of logical thread
[dead]
[flagged]
[flagged]
Ironically, your comment does more to contribute to "Rages over random small details" than actually improving the overall conversation. Be the change you wanna see :)
[flagged]
[flagged]