> "We also haven't seen any bugs that couldn't have been found by an elite human researcher." In other words, it's like adding an automated security researcher to your team.
This is a really dumb take.
Having a security researcher you can spin up (and therefore an army of researcher you can spin up) is not a nothingburger.
That it hasn’t found a new class of vulnerabilities is little consolation if it can pump out vulnerabilities from known classes.
That was a really weird sentence that makes no sense, since when is an on demand "elite researcher" nothing? While the article appears to dismiss this model, it ends up giving it a very high praise.
Did Anthropic build the bug-finding machine thats a thousand times cheaper than an elite security researcher? Can others replicate it and make it ubiquitous?
We don’t know the answer to the first. But say it’s just as expensive, you can instantly 10x “headcount”. As they say, quantity has a quality all of its own.
As for the second question, I think our default stance should be “yes” given the history of every other model advancement.
> "Anthropic's marketing message for Mythos was effectively a challenge, not dissimilar to a capture-the-flag exercise, where success includes claims of unauthorized access to Mythos," Tim Mackey, head of risk strategy at supply chain security shop Black Duck, told The Register.
"We also haven't seen any bugs that couldn't have been found by an elite human researcher."
bro this is like the first gen, in two years they will iterate and get better. this is just like first gen video, text and image generation were crap but people saw the potential. i've been involved in natural language processing and TTS and in the span of 1-2 years we have seen some crazy innovation. now you can run really great expressive open source multi-lingual TTS on phones and edge hw like raspberry pi's. they have completely wrecked career prospects of voice actors and translators.
if i was a bad guy, i could spin up a 100 agents and find exploits faster than the number of elite security researchers on the planet could fix it. imagine getting the latest version of apple ios and have a new vulnerability within a few days to a week
Or the asymptotic curve is getting near the flatline stage. We simply don't know, because the first thing we'd need is evidence to support their claims.
Also, if you point any AI, doesn't have to be Mythos, at crap code it'll find stuff. I know of two carefully-written code bases that had a lot of AI analysis which found... nothing. Or at least no vulns, just minor issues in various places.
Upcoming IPO. Announce product is too powerful to release. No connection whatsoever.
> "We also haven't seen any bugs that couldn't have been found by an elite human researcher." In other words, it's like adding an automated security researcher to your team.
This is a really dumb take.
Having a security researcher you can spin up (and therefore an army of researcher you can spin up) is not a nothingburger.
That it hasn’t found a new class of vulnerabilities is little consolation if it can pump out vulnerabilities from known classes.
That was a really weird sentence that makes no sense, since when is an on demand "elite researcher" nothing? While the article appears to dismiss this model, it ends up giving it a very high praise.
Did Anthropic build the bug-finding machine thats a thousand times cheaper than an elite security researcher? Can others replicate it and make it ubiquitous?
We don’t know the answer to the first. But say it’s just as expensive, you can instantly 10x “headcount”. As they say, quantity has a quality all of its own.
As for the second question, I think our default stance should be “yes” given the history of every other model advancement.
At my company I let Claude run in the background for a couple of hours and it found several backdoors and data exfiltration mechanisms.
It’s irrelevant to say “I could have found those myself”. I could have but I didn’t, and Claude did.
And what would have been the effect if those had been exploited?
"We also haven't seen any holes that couldn't have been dug by an elite construction worker." Therefore excavators are nothingburger
It was beyond the obvious since the moment they announced it, but I guess people love miracles.
the first clue was all the claims without evidence, what a shock
> "Anthropic's marketing message for Mythos was effectively a challenge, not dissimilar to a capture-the-flag exercise, where success includes claims of unauthorized access to Mythos," Tim Mackey, head of risk strategy at supply chain security shop Black Duck, told The Register.
Beautiful quote
"We also haven't seen any bugs that couldn't have been found by an elite human researcher."
bro this is like the first gen, in two years they will iterate and get better. this is just like first gen video, text and image generation were crap but people saw the potential. i've been involved in natural language processing and TTS and in the span of 1-2 years we have seen some crazy innovation. now you can run really great expressive open source multi-lingual TTS on phones and edge hw like raspberry pi's. they have completely wrecked career prospects of voice actors and translators.
if i was a bad guy, i could spin up a 100 agents and find exploits faster than the number of elite security researchers on the planet could fix it. imagine getting the latest version of apple ios and have a new vulnerability within a few days to a week
Or the asymptotic curve is getting near the flatline stage. We simply don't know, because the first thing we'd need is evidence to support their claims.
Also, if you point any AI, doesn't have to be Mythos, at crap code it'll find stuff. I know of two carefully-written code bases that had a lot of AI analysis which found... nothing. Or at least no vulns, just minor issues in various places.