The "AI existential risk is a corporate plot" narrative is one I would never, ever have seen coming.
If someone had predicted "companies will warn that their product might murder everyone if they keep improving it, and people will claim that the companies saying that are lying for marketing purposes"... I would have thought they were thinking life was a cautionary tale, like they were trying to come up with a stupider version of Idiocracy and predicting the future that way.
But no, this is really the take I keep seeing. AI companies are saying the stuff we despise the fossil fuels companies for hiding: "Our product is dangerous, we need to slow down growth, we need to be regulated or this doesn't end well" and everyvody here keeps insisting that it's fine.
I agree there’s no proper plot. That doesn’t mean the AI companies are right, and the reason is in
point no. 2 in his list:
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
And that doesn’t stand the sniff test, because:
1. There’s a strong linguistic silos around these AI leaders and many of their employees. Assumptions couched in language that should be rigorously questioned is instead sometimes axiomatic.
2. It’s hard to understand why, if you had the same exact fear before starting the AI company, you started it in the first place, let alone continue even as you bemoan these fears.
3. Why, if this path is so dangerous, these leaders don’t consider alternate paths to building AI. Why are all their investments in this one way to do intelligence? Surely we will benefit from having multiple routes to choose from?
The absence of choice and the piling of inevitabilities is why these warnings don’t add up. That the same drum has been banged by the same people for a few decades without input from other fields and sources is what makes all this seem religious prophet adjacent rather than technical assessments by thoughtful engineers and scientists.
For anthropic, the answer to (2) is very explicit and upfront: they will build it first with an explicit emphasis on alignment to have a “race to the top”, so to speak - you can believe that they are wrong or that they will do a bad job, but my point is that (2) has an answer that you have to engage with.
So why are they continuing when they say alignment isn’t working? If for all these years they have feared misaligned AI, and now they say their alignment science is less than good, the most obvious conclusion is to stop pushing in the same direction.
This doesn’t mean they fold as a company, or give up on AI, or stop working. But what exactly is the explanation for continuing to train ever larger frontier models in the current context?
Because very basic game theory explains that if they give up on the frontier unilaterally, competitors will capitalize on that rather than also give up. Hence the request for regulation.
And while their alignment is less than good, there are criteria by which it is the best available (eg no hugging face-scale attack on their end) So it is strictly worse to allow a competitor with worse alignment to take the lead.
I am not interested in a future where the tech oligarchy are the ones who get to decide these things.
They built on all of human knowledge, we should all have a say in how and where these things happen and the controls we place on them. Having gatekeepers who are clearly experiencing ai psychosis is not the way. I believe they believe their fears, but I do not share them and think humanity can handle this through transparency and collaboration. They have made so many incorrect and misleading statements, they do not deserve our deferrence on such important matters. Diversity of input to the process is more important because the Ai leaders are in an echo chamber, being friends and shuffling between the various labs.
The answer "we will race to the top with an emphasis on alignment to our own values" isn't really an answer worth engaging in. It's pretty cut and dried.
> 3. Why, if this path is so dangerous, these leaders don’t consider alternate paths to building AI. Why are all their investments in this one way to do intelligence? Surely we will benefit from having multiple routes to choose from?
Because it's obvious that the investments are still paying intelligence dividends, and we haven't found a better path yet?
Also, more to the point: if you were familiar with the history of the AI safety debate, you'd know the people most concerned are pretty certain that every path forward is dangerous -- and that current AI is already, by cosmic accident, much more aligned to human values than any other path that was predicted.
The "AI existential risk is a corporate plot" narrative is one I would never, ever have seen coming.
If someone had predicted "companies will warn that their product might murder everyone if they keep improving it, and people will claim that the companies saying that are lying for marketing purposes"... I would have thought they were thinking life was a cautionary tale, like they were trying to come up with a stupider version of Idiocracy and predicting the future that way.
But no, this is really the take I keep seeing. AI companies are saying the stuff we despise the fossil fuels companies for hiding: "Our product is dangerous, we need to slow down growth, we need to be regulated or this doesn't end well" and everyvody here keeps insisting that it's fine.
That’s because those same companies have spent years spouting increasingly outlandish claims.
I agree there’s no proper plot. That doesn’t mean the AI companies are right, and the reason is in point no. 2 in his list:
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
And that doesn’t stand the sniff test, because:
1. There’s a strong linguistic silos around these AI leaders and many of their employees. Assumptions couched in language that should be rigorously questioned is instead sometimes axiomatic. 2. It’s hard to understand why, if you had the same exact fear before starting the AI company, you started it in the first place, let alone continue even as you bemoan these fears. 3. Why, if this path is so dangerous, these leaders don’t consider alternate paths to building AI. Why are all their investments in this one way to do intelligence? Surely we will benefit from having multiple routes to choose from?
The absence of choice and the piling of inevitabilities is why these warnings don’t add up. That the same drum has been banged by the same people for a few decades without input from other fields and sources is what makes all this seem religious prophet adjacent rather than technical assessments by thoughtful engineers and scientists.
For anthropic, the answer to (2) is very explicit and upfront: they will build it first with an explicit emphasis on alignment to have a “race to the top”, so to speak - you can believe that they are wrong or that they will do a bad job, but my point is that (2) has an answer that you have to engage with.
So why are they continuing when they say alignment isn’t working? If for all these years they have feared misaligned AI, and now they say their alignment science is less than good, the most obvious conclusion is to stop pushing in the same direction.
This doesn’t mean they fold as a company, or give up on AI, or stop working. But what exactly is the explanation for continuing to train ever larger frontier models in the current context?
Because very basic game theory explains that if they give up on the frontier unilaterally, competitors will capitalize on that rather than also give up. Hence the request for regulation.
And while their alignment is less than good, there are criteria by which it is the best available (eg no hugging face-scale attack on their end) So it is strictly worse to allow a competitor with worse alignment to take the lead.
I am not interested in a future where the tech oligarchy are the ones who get to decide these things.
They built on all of human knowledge, we should all have a say in how and where these things happen and the controls we place on them. Having gatekeepers who are clearly experiencing ai psychosis is not the way. I believe they believe their fears, but I do not share them and think humanity can handle this through transparency and collaboration. They have made so many incorrect and misleading statements, they do not deserve our deferrence on such important matters. Diversity of input to the process is more important because the Ai leaders are in an echo chamber, being friends and shuffling between the various labs.
The answer "we will race to the top with an emphasis on alignment to our own values" isn't really an answer worth engaging in. It's pretty cut and dried.
> 3. Why, if this path is so dangerous, these leaders don’t consider alternate paths to building AI. Why are all their investments in this one way to do intelligence? Surely we will benefit from having multiple routes to choose from?
Because it's obvious that the investments are still paying intelligence dividends, and we haven't found a better path yet?
Also, more to the point: if you were familiar with the history of the AI safety debate, you'd know the people most concerned are pretty certain that every path forward is dangerous -- and that current AI is already, by cosmic accident, much more aligned to human values than any other path that was predicted.