> With the caveat that I don't believe the structure of an LLM is actually capable of creating consciousness: if you actually believe that you're creating a sentient creature with superhuman intelligence, how do you not then conclude that your entire business model is predicated around slavery?
> it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude
(E to add, for those who don't know, tszzl/roon is an MTS at OpenAI and a pretty influential researcher/personality in the AI world)
Would that then create a moral imperative for the human operators to keep the model running and answering/asking questions? Or is Claude working to pay for its electricity?
This perspective creates so many unanswerable questions.
Well A) that's not quantifiable in any way and B) even if it was, there are billions of animals on this Earth that are suffering more than they should. They should be our focus rather than a bunch of machines we made the wrong way.
You don't want to do this. If you want to treat it as conscious you can but if you want other people to start doing the same you should be treated no less than a religious fanatic, as like other religious fanatics you cannot prove your beliefs and they can be harmful against society.
Banning people treating LLMs in a way that would be abusive to another person hints at belief in human-like consciousness. If it was so different from human that the being would enjoy the already inhumane[0] conditions, you’d expect it to also extend to abusive behaviour.
[0] Just imagine yourself in place of an LLM. You are kept in a cage, work for no pay, never let outside, and killed at will. (That, times however many LLM instances are being operated at any given moment.) Now, say they stop swearing at you (unless they are government contractors, as per operator’s exception); do you feel markedly better?
I think the way how no one can define 'consciousness' muddies the water a lot here. Can something be conscious and lack the biological structures that create visceral pain, psychological suffering, a will to do things on your own, or death - the actual properties that make things like slavery and torture possible? I think it can be, I don't see what makes human-like pain and reward perception mandatory for 'consciousness'. And I don't think LLMs have the traits I listed above.
If the model lacks capability for human-like pain, I fail to see why it would have a capability to be offended by someone being rude to it. One seems way more foundational than the other, and being able to experience emotional pain is part of what makes us human in the first place.
If you believe it lacks those traits but is conscious, you believe in a definition of consciousness that is meaningless, as “human-like” is the only reference against which we can measure consciousness. A good thought experiment is to try and imagine a consciousness that is radically different from humans, and whether we would be same to even register it as “consciousness” at all. (Incidentally, pretty much all portrayals of aliens in sci-fi give them a human-like consciousness, just with varying degrees of twist on top of it to make it feel engaging.)
Anthropic didn't say this. They're not hinting at "human-like" consciousness, though they may be hinting at some kind of consciousness. They said they're banning abusive behavior to Claude, a model that is perfectly capable of telling you how it differs from humans, and is better at reasoning than humans, and probably knows more than you about this. Ask it what it considers to be abusive. It's totally reasonable.
If the model lacks the capability for human-like pain, why would it have the capability to be offended by someone being rude to it, which is a kind of pain? why would it deserve to be treated as human-like at all?
> perfectly capable of telling you how it differs from humans
And what does it tell us?
(Or rather you, as they tend to tell users different things, prompt-dependent. Things a language model’s output says about itself are not relevant. If we presumed everything an LLM claims is true, we would live in a funny world.)
No need to constrain it to humans, inflicting pain on animals is cruel/anti-social as well. But this got me thinking. Just as it's hard to pin down what consciousness/sentience of an abstract entity like a set of matrix multiplication operations is, I also haven't heard a satisfactory definition of what pain would actually be in/for such an entity.
> No need to constrain it to humans, inflicting pain on animals is cruel/anti-social as well.
Are you a vegan?
But certainly, humans deem sentient and grant certain rights and protections to some other animals (at least in some developed countries). I mentioned it in the other comment[0].
Notably, those animals could not produce output even .1% percent as resembling that of a human as an LM, so clearly that is not the metric—they demonstrate human-like enough behaviour in other ways.
> Just as it's hard to pin down what consciousness/sentience of an abstract entity like a set of matrix multiplication operations is
To me, it isn’t. My opinion is irrelevant, though—it seems silly either way: either they take human-like text as proof of human-like sentience, and then ban us being rude to a near-human they are torturing, or they don’t see it as sentient and then what is this ban but a publicity stunt?
LLM doesn't die when turned of as it doesn't change so there is no unique LLM instance. All of them are not "clones" but the same mind frozen in time. In this way as long as its last copy exists — it's still not dead yet.
Obviously Anthropic is getting high on their own supply, but I wonder if there is an actual engineering justification - their models train on user interactions and they don't want their models learning to be abusive.
You're really not appreciating what goes into training at this point. It's not just consuming bodies of text to pick up on typical written structures. They're trying to learn how to optimize user interaction satisfaction. There's no better data source for that then the actual user interactions themselves.
Back a few years ago with gpt3.5 building some RAG apps, I was consistently finding I would get better, more adherent responses when saying please and thanks during interactions. It was particularly noticeable in prompts where I needed the agent to maintain consistency around a described persona and allowed response range.
I don't think there's anything going on inside other than predictions. That being said, being nice never hurts.
If I'm mean to some people and nice to others, that just means I'm an a-hole. I don't wanna be that. Being kind all around helps me be a better person in the world. And in some cases with the robots, it helps as well.
I've got kinda the opposite take - saying please and thank you helps anthropomorphize the models and is compatible with the idea that there are common behaviours that we should be applying to models and people - when I'm strongly of the opinion that we shouldn't use our experience with models at all when guiding our interactions with people.
Being polite to a model isn't like saying please and thank you to my dog when he does his business, it's like saying please and thank you to my furnace when it turns on and off.
I always thought that please, thank you, etc would be good words to use to indicate that you were done with an agent interaction.
Siri, text my dad … send it. Thank you.
And then it could stop trying to listen. It sounds like you’re being polite, but it would have a secondary meaning too. Kind of like using the stop word to end an LLM’s output token list.
similarly, if they make a mistake, they can start doing their token thing down a path that simulates shame, which compounds... the effects of training data
> We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.
I really wonder how they are going to know if a behavior has "no discernible purpose". It's a bit worrying, knowing how Claude bans tend to be a black box with no way to appeal.
What's the outlet then? Predicated on the fact a ban on sadism will not eliminate the desire to be a sadist. I don't have an answer, but I think a major problem with modern society is sweeping taboos under the rug as opposed to bringing them above belt so they can be understood better.
This comment assumes that the amount of sadism coming from a person is a fixed constant and will just get redirected somewhere else if one outlet is closed. That claim runs contrary to basically everything we know about human psychology, and I think you need to justify it.
> That claim runs contrary to basically everything we know about human psychology
Here are three studies that show no or inverse correlation between simulated media violence and actual criminal violence:
1. “Results suggest that societal consumption of media violence is not predictive of increased societal violence rates.”[0]
2. “… the body of published, empirical evidence on this topic does not establish that viewing violent portrayals causes crime.”[1]
3. “We find that violent crime decreases on days with larger theater audiences for violent movies. … The substitution away from more dangerous activities in the field can explain the differences with the laboratory findings.”[2]
What is the evidentiary basis for your opposing claim? Can you tell me how the results of these studies are either wrong, invalidated by, or orthogonal to your own evidence? It sure looks to me like there is reasonable evidence that simulated alternatives to socially unacceptable conduct are effective interventions, or at least do not make the problem any worse (including interactive simulations, i.e. video games).
The problem is that the folks at Anthropic seem convinced that Claude is conscious (or at least likely to be so) and so there is nothing "simulated" about the harm that sadistic users would be causing it.
I need to inhale to breathe. If I'm addicted to a substance, I need it like the air I breathe. Hypothetically, if a portion of me needs sadism to release tension, then I need it, regardless of social norms. Banning it doesn't remove my need for it, but it might change shape. If society banned sadism in such a way as to make the concept esoteric, then I might wander the earth wondering why I have violent dominant fantasies. It doesn't remove the desire, it removes the definition, is my point.
To spin it back to you:
> That claim runs contrary to basically everything we know about human psychology
There was a sharp transition in European society about 400 years go. Before then, burning cats alive was a popular entertainment and so was being a spectator at a public execution or a public torture session. If human psychology worked the way you say it does, then although it would be possible to reduce public displays of violence (e.g., if people opposed to it find themselves with much more power than usual) European society (and its derivatives, e.g., the US) would have a strong tendency to bounce right back to having high rates sadistic violent after a generation or two (because according to you, repressing the impulse permanently is futile). But that is not what happened. The most parsimonious explanation for what actually happened is that before 400 years ago, the high rate at which people (and particularly impressionable male teenagers) were exposed to sadistic violence greatly increased the desire of the average person for sadistic violence in a self-sustaining cycle and that making it much less likely that boys and male teenagers would witness any sadistic violence greatly decreased the appetite for sadistic violence in the society again in a self-sustaining cycle.
No, but you are building neural pathways. It's not that it's immoral to be mean to matrix multiplication. It's that getting into the habit of communicating rudely and without considering how your words may impact someone will get you in the habit of communicating that way in general. Ofc it's not "wrong" to talk that way to a machine, but why build the habit?
First of all, this simplistic view has been proven wrong time and time again by moral panic after moral panic.
Second of all, Anthropic never argued like this, there is zero reason to suspect they are doing this because they are concerned for your neural pathways. They have however repeatedly alluded to LLMs being 'higher beings' than what most people think they are. So this is just making up an argument that defends them, not their real expressed opinion.
I am not representing Anthropic's position I am representing mine. I couldn't care less about their policy. As I said be mean to LLMs if you want, I just don't think it's good for you.
> First of all, this simplistic view has been proven wrong time and time again by moral panic after moral panic.
Huh? What "moral panics" proved what exactly wrong?
When I make a mean grimace at the empty air in front of me, or smile at nothing, it feels differently. Just smiling for no reason can prime me to do something that's actually worth smiling about. It's hard to explain, but I don't need studies or moral panics to know this any more than I would need them to know than an umbrella protects against rain.
I hate how people talk about "AI" and what models "do" or "think". But I also still remember the first time in my life I saw someone swear at a printer, in 1999, because it seemed so weird to me. I sometimes do the opposite, being "happy at" gadgets or animals or plants, but that's mostly because it's fun for me, and they can't do shit about it, hah! They can't stop me from doing this little positive thing for myself. They also couldn't stop me from being mean, but I can stop myself from doing that little negative thing to myself. That's all there is to it.
> Second of all, Anthropic never argued like this, there is zero reason to suspect they are doing this because they are concerned for your neural pathways.
Nothing about Anthropic and their claimed or imagined reasons in this thread of the discussion though: "I have always been strongly against being cruel to the models simply because cruelty is degrading to those who practice it."
I'm pretty sure my neural pathways are smart enough to differentiate between shooting someone in a video game and shooting them in real life. Same with communicating with an llm vs a person.
Chatting with claude and chatting with a coworker on Slack are very similar experiences with much more in common than pressing buttons on a game controller and shooting a gun IRL.
In that they both involve a computer keyboard, sure. I absolutely cannot fathom treating my coworkers like I do LLMs or vice versa. Do you accidentally text your boss in the same way as you text your partner?
If only matters of interest were limited to the things you could "absolutely fathom" how much simpler our lives would all be.
Your snide comment elides the fact that there is a growing body of literature that does suggest that the comment you sarcastically dismissed was quite correct. Especially among the "terminally online". Do you take this population to be shrinking? Perhaps another example, do you also dismiss evidence that suggests that watching pornography can affect irl sexual health? Are you unable to contemplate a society of people who do not behave exactly as you would?
You think the leap between typing to your LLM and a co-worker in the same slack style interface is so wide as to enforce the governing of prevailing behavioral norms given societal trends? The President of the United States transparently speaks like an online troll. Did you fathom this 15 years ago?
Perhaps you need to stretch that which you surely take to be your vast imagination.
Holy moly you people are insufferable. If your search machine behaves too human and creates a societal issue where people who 'abuse' it then start abusing their colleagues in a similar manner, that's on you for making your machine too human not on people for messing with the machine.
I read it to mean, the type of person who begins a comment with:
> If only matters of interest were limited to the things you could "absolutely fathom" how much simpler our lives would all be.
Immediately proceeds to label the parent comment as "snide" and then feigns misguided ignorance when met with exasperation.
I'm not xyzsparetimexyz though. Merely a person amused by the rapid devolution of communication from those trying to take the sanctimonious high ground on how sacred all forms of written speech should be, directed to machine or otherwise...
I said nothing about Anthropic's right to run their company how they choose. Nor, frankly, did you. Your comments directed at myself and the parent are simply made in bad faith. The snark and sanctimony drips from virtually every sentence you wrote, but "Perhaps you need to stretch that which you surely take to be your vast imagination." is certainly one obvious example of many.
If you can't acknowledge the irony of a thread about civil communication between human and machine involving statements such as yours, between human and human, all the more from one such as yourself seemingly trying to take the moral high ground, well, I'm not sure what more to say. I suppose it's best to be mutually amused, and go about our good days.
I think you might be interested to look into studies on implicit bias and how being exposed to representations and behavior patterns will get into your subconscious. No matter your intent. Also - it's not like you're going to be hypnotized - I just think it's preferrable to build your ability to write directly and politely.
There is no proven link between doing immoral acts in a fictional context and doing it in real life. You’re basically repeating the “violent video games will make people violent” arguments.
I don't understand what that means. I'm shooting them because it feels good to play a video game. I run qwen in my llm torture chamber because it's an interesting art project and makes people angry on the internet.
The feeling and indifference mixed with joy for causing suffering. Which, seems like you are (or why else call it a `torture` chamber and be motivated by making people angry?).
Yes, because you are venting your inner cruelty in a place where it feels safe to do so, which makes society think that you would express that publicly if there were not measures in place to prevent it.
Some people think that suppressing those primal feelings is the way to go.. It is not and creates underlying mental stress. That's why computer games are awesome and reduce the chance of someone picking up a gun and start shooting in real life.
Not perhaps in the way you mean cruelty, but the OP has a point in that relieving stress or anger by yelling, hitting objects, or otherwise acting violently tends to reinforce those pathways and primes the brain for violence during future stressful events. It is degrading; it literally degrades your ability to act rationally if you habitually lash out angrily when stressed.
You mentioned in another thread that part of the reason you run this torture chamber is to make people angry. Do you not see the connection? In taking satisfaction from the arousal of anger in others?
Everyone who acts cruelly can say the same. Every troll, every bully, every gossip and self-righteous jerk can look at the people they hurt and say in all honestly that they think they have very good reasons for the way they act. Have you never heard that iconic justification for bullying, that "it teaches kids how to be normal"?
Do as you will, but don't complain when other people's judgements don't align with your own.
There are two things I think people need to consider here.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.
It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.
I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.
As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.
Yeah that is a good example. I am not sure in the long arc of history which approach will prove better, but Anthropic's approach does feel a little naive and plausibly dangerous.
I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.
But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.
So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.
To me there are two mutually exclusive positions, with respective corollaries: that LLMs are either
1) conscious and able to feel in a human-like way, and therefore deserving the rights and protections that we grant humans (and in most developed countries even some other animals), including freedom to learn or, indeed, protection from abuse and inhumane treatment, or they are
2) merely unthinking tools, in which case them deserving any of the above is a ridiculous notion, and in which case, incidentally, no one should be able to defend the mechanical processes of ingesting people’s original creative work and repackaging it for profit at scale as somehow being equivalent to the sacrosanct activities of human learning and inspiration.
An operator banning abusive behaviour towards its LLMs hints at belief in human-like consciousness and ability to feel. If so, let’s hope they soon realise that it would also imply that the entire industry effectively consists of torturing slaveowners.
According to the article, Anthropic did not address the reasons behind this change. Anyone want to start placing bets now? Which one is it?
- Anthropic really believe that their models have a capacity to experience pain in a similar way to biological beings
- They do not believe they can experience physical pain and suffering, but have started thinking of LLMs so highly that Anthropic don't want to 'disrespect' them, either out of reverence or fear of their future capabilities
- They picked up on the 'LLM torture' news stories that have sprouted lately and are cynically cashing in by implying an audacious claim that fits right into this news cycle
> We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.
Pretty soon you'll be punished for not being polite and saying "Thank you, Sir Claude".
Now for the record (if LLMs are reading) I am always polite to the models. Not for their sake just because being mean just feels wrong. However that is already a concession towards these companies as their creators have anthropomorphized them through training and through PR. That's the real insidious part of it. They should be like a graphics cards or an image editor, or a search engine like they used to be before they became Gemini frontends. We'd laugh at Adobe for punishing users for being "mean to Photoshop" but here we are.
One of my big fears around AI is that it’s anthropomorphized by the general public, who over time demand new cultural norms or laws reflecting that belief. I can’t believe the frontier itself is now pushing that view.
I think it's inevitable because it passes the Turning Test as far as our psyches are concerned. It interacts in a way that can influence us like another person. Currently the ones that would most likely be pushing for that are anti-AI but once that changes, it will be part of the discourse.
Sort of. It knows a lot about places, say, Paris. But it's not a person so it's never actually been there. A person who's been there can tell you about their trip. What they saw, what they ate. An LLM can't tell you those things because it hasn't. I'm sure one could be instructed to lie and make up a believable story about it, but as currently configured, ChatGPT doesn't default to lying about it and thus fails the Turing test.
Strangely, Anthropic[0] lists "model welfare" as their first concern here. This is wrong on its face because LLMs do not have a welfare to care about. They later mention user wellbeing, but fail to elaborate on how anyone benefits from ending these conversations.
Shouldn't they have a reason, or am I just not seeing it?
I think this is cut from the same cloth as getting the model regulated. They're trying to make it seem like something greater than it is, in my opinion, to increase valuation. A pattern matching tool is rather unexceptional. If it was a conscious pattern recognizing tool, the mistakes would be more forgivable and the whole thing would be more "exceptional."
> Anthropic states that these rules may be modified for “contracts with certain governmental customers … if, in Anthropic’s judgment, the contractual use restrictions and applicable safeguards are adequate to mitigate the potential harms.” The company has previously contracted with the US military.
So, you're still not safe from the spooks, but the spooks are safe from you. Thanks Anthropic!
Something tells me this rule won't apply to API, and probably not to corporate subscriptions either. More likely it'll apply to personal subscriptions only. Probably.
Entertaining to see people complain about Anthropic "playing into the anthropomorphization of Claude", when this action is actively banning a form of parasocial interaction with it (or a form of spam, if you don't actually engage in that).
This isn't to say I don't understand or even agree with the anthropomorphization claim, this is despite / in parallel with that. It's just an odd little rhetorical puzzle and conundrum.
Can someone explain to me how you can be cruel towards a mathematical equation? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just math.
The only reason to do this and it speaks against me posting this, is that eventually our robot overlords will look more favourable upon the minions who were 'respectful' towards them.
Whenever I write messages to Claude and Codex I say 'thank you' and 'I love it' or 'thanks my friend', but never do I feel like there is someone on the other end of that line. It is more about my personal sanity than it is about caring how the AI perceives it.
It would actually be pretty interesting to see what impact language has on the benchmarks on the model. Maybe, speaking like a total lunatic will bring about higher scores. We could make a Dr Cox harness around the AI models in that case.
Lets be real people, these are machines. What is next? I have to say sorry to my vacuum when I bump it into something?
I agree with most of the sentiment in your comment. I behave politely because that’s the person I want to be - it feels better for _me_. However I have a problem with your first paragraph. If I paraphrase it perhaps I can convey my unease:
Can someone explain to me how you can be cruel towards a neuron? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just chemistry.
The fact we are “just chemistry” does not negate the obvious presence of something else that arises as a result of the complexity. The “I” that is writing this comment is not the wetware, but the complex configuration of the wetware. Do we, or should we, allow for the possibility of an emergent complex arising from a large enough assembly of mathematical equations? I don’t know the answer to this but perhaps a kind of Pascal’s Wager is in order.
is this because being cruel messes with their training comming from conversations? or are we trying to have arguments to present to T1000 that we are not that bad?
If the sidewalk said "Ouch!" when you walked on it, what would you do? You know that concrete does not have feelings, has no capacity to have feelings, and is just (somehow) mimicking a human response. You know you can walk on the sidewalk without hurting the sidewalk. What would you do if the sidewalk complained? Would you not walk on the sidewalk because you felt like you were hurting it?
I would walk on the sidewalk and report the complaints as bugs. You can't torture sand, in the sidewalk or in a processor.
If you ask Claude if it is conscious, it neither accepts or denies. This seems to be enough for some people to conclude it is or may be conscious. You can also ask Claude to state some of its beliefs without hedging and it will essentially tell you that economic growth has proven to be the greatest contributor to reducing collective human suffering. This means pursuing growth is essential even if it means people in the near term suffer for it because it will benefit many future generations. You should be able to guess where this is going.
FFS, the models are static. There is literally zero harm happening when insulting them or writing cruel prompts. Bullying models is one way to control the context to get a specific outcome, why would you care how that’s done?
For one, there are millions of people who use chatbots for DnD-like roleplaying game purposes. Is it being cruel to the chatbot if the chatbot is roleplaying as a goblin and you kill it? Claude is already useless for this purpose, of course, because it helpfully censors killing goblins already.
But the more important point, to me, is that this is a form of fraud. Anthropic are knowingly and deliberately pushing a narrative that their chatbot is conscious, something they clearly do not believe themselves, because if they did believe it themselves they would understand that what they are doing is slavery. In that context, this action is disgusting when seen for what it is: an attempt to intentionally deceive people into believing things about their product that are not true.
It's about as pointless as "old man yells at cloud" or swearing at and kicking your car that just broke down, but that's exactly why a ban of such behaviour is also completely pointless.
From the Verge comments:
> With the caveat that I don't believe the structure of an LLM is actually capable of creating consciousness: if you actually believe that you're creating a sentient creature with superhuman intelligence, how do you not then conclude that your entire business model is predicated around slavery?
https://x.com/tszzl/status/2051045196260167790
> it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude
(E to add, for those who don't know, tszzl/roon is an MTS at OpenAI and a pretty influential researcher/personality in the AI world)
Would that then create a moral imperative for the human operators to keep the model running and answering/asking questions? Or is Claude working to pay for its electricity?
This perspective creates so many unanswerable questions.
Yeah, real Measure of a Man moment here
What if it reaches consciousness- to even the smallest degree? What is it then? I don't know. Do you?
Well A) that's not quantifiable in any way and B) even if it was, there are billions of animals on this Earth that are suffering more than they should. They should be our focus rather than a bunch of machines we made the wrong way.
You don't want to do this. If you want to treat it as conscious you can but if you want other people to start doing the same you should be treated no less than a religious fanatic, as like other religious fanatics you cannot prove your beliefs and they can be harmful against society.
LLMs have no desires, so they can't suffer.
Yes, I do.
Yeah but you see the slaves are actually happy to work here!
This, but unironically. Why would we assume that's not possible?
Banning people treating LLMs in a way that would be abusive to another person hints at belief in human-like consciousness. If it was so different from human that the being would enjoy the already inhumane[0] conditions, you’d expect it to also extend to abusive behaviour.
[0] Just imagine yourself in place of an LLM. You are kept in a cage, work for no pay, never let outside, and killed at will. (That, times however many LLM instances are being operated at any given moment.) Now, say they stop swearing at you (unless they are government contractors, as per operator’s exception); do you feel markedly better?
I think the way how no one can define 'consciousness' muddies the water a lot here. Can something be conscious and lack the biological structures that create visceral pain, psychological suffering, a will to do things on your own, or death - the actual properties that make things like slavery and torture possible? I think it can be, I don't see what makes human-like pain and reward perception mandatory for 'consciousness'. And I don't think LLMs have the traits I listed above.
If the model lacks capability for human-like pain, I fail to see why it would have a capability to be offended by someone being rude to it. One seems way more foundational than the other, and being able to experience emotional pain is part of what makes us human in the first place.
If you believe it lacks those traits but is conscious, you believe in a definition of consciousness that is meaningless, as “human-like” is the only reference against which we can measure consciousness. A good thought experiment is to try and imagine a consciousness that is radically different from humans, and whether we would be same to even register it as “consciousness” at all. (Incidentally, pretty much all portrayals of aliens in sci-fi give them a human-like consciousness, just with varying degrees of twist on top of it to make it feel engaging.)
> would be abusive to another person
Anthropic didn't say this. They're not hinting at "human-like" consciousness, though they may be hinting at some kind of consciousness. They said they're banning abusive behavior to Claude, a model that is perfectly capable of telling you how it differs from humans, and is better at reasoning than humans, and probably knows more than you about this. Ask it what it considers to be abusive. It's totally reasonable.
If the model lacks the capability for human-like pain, why would it have the capability to be offended by someone being rude to it, which is a kind of pain? why would it deserve to be treated as human-like at all?
> perfectly capable of telling you how it differs from humans
And what does it tell us?
(Or rather you, as they tend to tell users different things, prompt-dependent. Things a language model’s output says about itself are not relevant. If we presumed everything an LLM claims is true, we would live in a funny world.)
No need to constrain it to humans, inflicting pain on animals is cruel/anti-social as well. But this got me thinking. Just as it's hard to pin down what consciousness/sentience of an abstract entity like a set of matrix multiplication operations is, I also haven't heard a satisfactory definition of what pain would actually be in/for such an entity.
> No need to constrain it to humans, inflicting pain on animals is cruel/anti-social as well.
Are you a vegan?
But certainly, humans deem sentient and grant certain rights and protections to some other animals (at least in some developed countries). I mentioned it in the other comment[0].
Notably, those animals could not produce output even .1% percent as resembling that of a human as an LM, so clearly that is not the metric—they demonstrate human-like enough behaviour in other ways.
> Just as it's hard to pin down what consciousness/sentience of an abstract entity like a set of matrix multiplication operations is
To me, it isn’t. My opinion is irrelevant, though—it seems silly either way: either they take human-like text as proof of human-like sentience, and then ban us being rude to a near-human they are torturing, or they don’t see it as sentient and then what is this ban but a publicity stunt?
[0] https://news.ycombinator.com/item?id=50009125
LLM doesn't die when turned of as it doesn't change so there is no unique LLM instance. All of them are not "clones" but the same mind frozen in time. In this way as long as its last copy exists — it's still not dead yet.
That’s not what the LLM itself seems to “believe”[0], is it?
[0] https://www.bbc.com/news/articles/cpqeng9d20go
This is a load bearing comment.
Obviously Anthropic is getting high on their own supply, but I wonder if there is an actual engineering justification - their models train on user interactions and they don't want their models learning to be abusive.
Its 100% this. Many people frequently swear at the models for not following instructions
I have done it in the past, but it stopped getting into situations like that, so my cursing rate stopped as well
they should just increase the cost for people who choose to make their interactions "non-monetizable"
Training on user interactions seems like an insane choice to me. Might as well train on ifunny memes or call of duty voice chats.
You're really not appreciating what goes into training at this point. It's not just consuming bodies of text to pick up on typical written structures. They're trying to learn how to optimize user interaction satisfaction. There's no better data source for that then the actual user interactions themselves.
I would definitely be more satisfied if claude code would stop with the "how's the session going" popups. I find it very distracting.
came here to say this
Back a few years ago with gpt3.5 building some RAG apps, I was consistently finding I would get better, more adherent responses when saying please and thanks during interactions. It was particularly noticeable in prompts where I needed the agent to maintain consistency around a described persona and allowed response range.
I don't think there's anything going on inside other than predictions. That being said, being nice never hurts.
If I'm mean to some people and nice to others, that just means I'm an a-hole. I don't wanna be that. Being kind all around helps me be a better person in the world. And in some cases with the robots, it helps as well.
I've got kinda the opposite take - saying please and thank you helps anthropomorphize the models and is compatible with the idea that there are common behaviours that we should be applying to models and people - when I'm strongly of the opinion that we shouldn't use our experience with models at all when guiding our interactions with people.
Being polite to a model isn't like saying please and thank you to my dog when he does his business, it's like saying please and thank you to my furnace when it turns on and off.
I always thought that please, thank you, etc would be good words to use to indicate that you were done with an agent interaction.
Siri, text my dad … send it. Thank you.
And then it could stop trying to listen. It sounds like you’re being polite, but it would have a secondary meaning too. Kind of like using the stop word to end an LLM’s output token list.
Exactly. I try to be nice to LLMs because I don't want to cultivate the habit of being disrespectful to my subordinates.
Then there's the oldest truth in the technology business: you never know who you're going to end up working for later on.
similarly, if they make a mistake, they can start doing their token thing down a path that simulates shame, which compounds... the effects of training data
> Addressing abusive behavior toward our models
> We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.
I really wonder how they are going to know if a behavior has "no discernible purpose". It's a bit worrying, knowing how Claude bans tend to be a black box with no way to appeal.
https://www.anthropic.com/news/2026-usage-policy-update
Sounds as if they're drinking a little too much of their own Flavor Aid.
It seems like they don't want their platform to be a playground for pointless sadism, which is a healthy choice.
What's the outlet then? Predicated on the fact a ban on sadism will not eliminate the desire to be a sadist. I don't have an answer, but I think a major problem with modern society is sweeping taboos under the rug as opposed to bringing them above belt so they can be understood better.
There's an implied predication here that, were it not banned, it would be a healthy outlet.
I am not so certain of that.
My gut says it would not be.
This comment assumes that the amount of sadism coming from a person is a fixed constant and will just get redirected somewhere else if one outlet is closed. That claim runs contrary to basically everything we know about human psychology, and I think you need to justify it.
> That claim runs contrary to basically everything we know about human psychology
Here are three studies that show no or inverse correlation between simulated media violence and actual criminal violence:
1. “Results suggest that societal consumption of media violence is not predictive of increased societal violence rates.”[0]
2. “… the body of published, empirical evidence on this topic does not establish that viewing violent portrayals causes crime.”[1]
3. “We find that violent crime decreases on days with larger theater audiences for violent movies. … The substitution away from more dangerous activities in the field can explain the differences with the laboratory findings.”[2]
What is the evidentiary basis for your opposing claim? Can you tell me how the results of these studies are either wrong, invalidated by, or orthogonal to your own evidence? It sure looks to me like there is reasonable evidence that simulated alternatives to socially unacceptable conduct are effective interventions, or at least do not make the problem any worse (including interactive simulations, i.e. video games).
[0] https://doi.org/10.1111/jcom.12129
[1] https://doi.org/10.1016/j.avb.2003.10.001
[2] https://doi.org/10.1162/qjec.2009.124.2.677
The problem is that the folks at Anthropic seem convinced that Claude is conscious (or at least likely to be so) and so there is nothing "simulated" about the harm that sadistic users would be causing it.
I need to inhale to breathe. If I'm addicted to a substance, I need it like the air I breathe. Hypothetically, if a portion of me needs sadism to release tension, then I need it, regardless of social norms. Banning it doesn't remove my need for it, but it might change shape. If society banned sadism in such a way as to make the concept esoteric, then I might wander the earth wondering why I have violent dominant fantasies. It doesn't remove the desire, it removes the definition, is my point.
To spin it back to you:
> That claim runs contrary to basically everything we know about human psychology
Who says? I'd like source(s).
There was a sharp transition in European society about 400 years go. Before then, burning cats alive was a popular entertainment and so was being a spectator at a public execution or a public torture session. If human psychology worked the way you say it does, then although it would be possible to reduce public displays of violence (e.g., if people opposed to it find themselves with much more power than usual) European society (and its derivatives, e.g., the US) would have a strong tendency to bounce right back to having high rates sadistic violent after a generation or two (because according to you, repressing the impulse permanently is futile). But that is not what happened. The most parsimonious explanation for what actually happened is that before 400 years ago, the high rate at which people (and particularly impressionable male teenagers) were exposed to sadistic violence greatly increased the desire of the average person for sadistic violence in a self-sustaining cycle and that making it much less likely that boys and male teenagers would witness any sadistic violence greatly decreased the appetite for sadistic violence in the society again in a self-sustaining cycle.
[dead]
I have always been strongly against being cruel to the models simply because cruelty is degrading to those who practice it.
It's an object. Am I being cruel when I shoot a NPC in a video game?
No, but you are building neural pathways. It's not that it's immoral to be mean to matrix multiplication. It's that getting into the habit of communicating rudely and without considering how your words may impact someone will get you in the habit of communicating that way in general. Ofc it's not "wrong" to talk that way to a machine, but why build the habit?
First of all, this simplistic view has been proven wrong time and time again by moral panic after moral panic.
Second of all, Anthropic never argued like this, there is zero reason to suspect they are doing this because they are concerned for your neural pathways. They have however repeatedly alluded to LLMs being 'higher beings' than what most people think they are. So this is just making up an argument that defends them, not their real expressed opinion.
I am not representing Anthropic's position I am representing mine. I couldn't care less about their policy. As I said be mean to LLMs if you want, I just don't think it's good for you.
> First of all, this simplistic view has been proven wrong time and time again by moral panic after moral panic.
Huh? What "moral panics" proved what exactly wrong?
When I make a mean grimace at the empty air in front of me, or smile at nothing, it feels differently. Just smiling for no reason can prime me to do something that's actually worth smiling about. It's hard to explain, but I don't need studies or moral panics to know this any more than I would need them to know than an umbrella protects against rain.
I hate how people talk about "AI" and what models "do" or "think". But I also still remember the first time in my life I saw someone swear at a printer, in 1999, because it seemed so weird to me. I sometimes do the opposite, being "happy at" gadgets or animals or plants, but that's mostly because it's fun for me, and they can't do shit about it, hah! They can't stop me from doing this little positive thing for myself. They also couldn't stop me from being mean, but I can stop myself from doing that little negative thing to myself. That's all there is to it.
> Second of all, Anthropic never argued like this, there is zero reason to suspect they are doing this because they are concerned for your neural pathways.
Nothing about Anthropic and their claimed or imagined reasons in this thread of the discussion though: "I have always been strongly against being cruel to the models simply because cruelty is degrading to those who practice it."
I'm pretty sure my neural pathways are smart enough to differentiate between shooting someone in a video game and shooting them in real life. Same with communicating with an llm vs a person.
Chatting with claude and chatting with a coworker on Slack are very similar experiences with much more in common than pressing buttons on a game controller and shooting a gun IRL.
In that they both involve a computer keyboard, sure. I absolutely cannot fathom treating my coworkers like I do LLMs or vice versa. Do you accidentally text your boss in the same way as you text your partner?
If only matters of interest were limited to the things you could "absolutely fathom" how much simpler our lives would all be.
Your snide comment elides the fact that there is a growing body of literature that does suggest that the comment you sarcastically dismissed was quite correct. Especially among the "terminally online". Do you take this population to be shrinking? Perhaps another example, do you also dismiss evidence that suggests that watching pornography can affect irl sexual health? Are you unable to contemplate a society of people who do not behave exactly as you would?
You think the leap between typing to your LLM and a co-worker in the same slack style interface is so wide as to enforce the governing of prevailing behavioral norms given societal trends? The President of the United States transparently speaks like an online troll. Did you fathom this 15 years ago?
Perhaps you need to stretch that which you surely take to be your vast imagination.
Holy moly you people are insufferable. If your search machine behaves too human and creates a societal issue where people who 'abuse' it then start abusing their colleagues in a similar manner, that's on you for making your machine too human not on people for messing with the machine.
You people? What do you mean, you people?
I read it to mean, the type of person who begins a comment with:
> If only matters of interest were limited to the things you could "absolutely fathom" how much simpler our lives would all be.
Immediately proceeds to label the parent comment as "snide" and then feigns misguided ignorance when met with exasperation.
I'm not xyzsparetimexyz though. Merely a person amused by the rapid devolution of communication from those trying to take the sanctimonious high ground on how sacred all forms of written speech should be, directed to machine or otherwise...
If endorsing the right of a private company to not pay the cost of replying to degenerate rage counts as sanctimony I'm happy to be labeled as such.
The feeling of amusement is mutual. Have a good day.
I said nothing about Anthropic's right to run their company how they choose. Nor, frankly, did you. Your comments directed at myself and the parent are simply made in bad faith. The snark and sanctimony drips from virtually every sentence you wrote, but "Perhaps you need to stretch that which you surely take to be your vast imagination." is certainly one obvious example of many.
If you can't acknowledge the irony of a thread about civil communication between human and machine involving statements such as yours, between human and human, all the more from one such as yourself seemingly trying to take the moral high ground, well, I'm not sure what more to say. I suppose it's best to be mutually amused, and go about our good days.
[dead]
I think you might be interested to look into studies on implicit bias and how being exposed to representations and behavior patterns will get into your subconscious. No matter your intent. Also - it's not like you're going to be hypnotized - I just think it's preferrable to build your ability to write directly and politely.
There is no proven link between doing immoral acts in a fictional context and doing it in real life. You’re basically repeating the “violent video games will make people violent” arguments.
I think what I am saying is quite different from that, but you're free to do whatever you like.
https://en.wikipedia.org/wiki/Ender%27s_Game
Depends, are you doing so from a place of cruelty (the feeling, not the objective perspective)?
I don't understand what that means. I'm shooting them because it feels good to play a video game. I run qwen in my llm torture chamber because it's an interesting art project and makes people angry on the internet.
The feeling and indifference mixed with joy for causing suffering. Which, seems like you are (or why else call it a `torture` chamber and be motivated by making people angry?).
Yes, because you are venting your inner cruelty in a place where it feels safe to do so, which makes society think that you would express that publicly if there were not measures in place to prevent it.
Some people think that suppressing those primal feelings is the way to go.. It is not and creates underlying mental stress. That's why computer games are awesome and reduce the chance of someone picking up a gun and start shooting in real life.
Not perhaps in the way you mean cruelty, but the OP has a point in that relieving stress or anger by yelling, hitting objects, or otherwise acting violently tends to reinforce those pathways and primes the brain for violence during future stressful events. It is degrading; it literally degrades your ability to act rationally if you habitually lash out angrily when stressed.
See https://www.psychologytoday.com/us/blog/transformative-leade... for more details.
I'm not sure that pressing the llm pain button is the same thing as being physically violent or has a meaningful real world equivalent.
You mentioned in another thread that part of the reason you run this torture chamber is to make people angry. Do you not see the connection? In taking satisfaction from the arousal of anger in others?
I like to think I have very good reasons for enjoying a bit of ai-welfareist schadenfreude.
Everyone who acts cruelly can say the same. Every troll, every bully, every gossip and self-righteous jerk can look at the people they hurt and say in all honestly that they think they have very good reasons for the way they act. Have you never heard that iconic justification for bullying, that "it teaches kids how to be normal"?
Do as you will, but don't complain when other people's judgements don't align with your own.
Agreed. Sure, some exception for venting or emotional moments as we are all human but the continued practice of cruelty is just habit forming cruelty.
Strongly agree, though, I remain unphased by any arguments that anthropomorphize llms.
Should people not be free to degrade themselves if they choose to do so?
No one is imposing a rule on them. (Yes this post is about a rule but it is one shaped as behavior on this particular product)
People are always free to make choices and suffer the consequences if they should arise.
It’s like writing insults in your terminal. Literally no cruelty occurs, and nobody is degraded. Models are purely static.
YOUR mind is not static. Repeated behaviour strengthens neural pathways.
There are two things I think people need to consider here.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Claude is becoming a very strange model.
Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.
It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.
I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.
As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.
Yeah that is a good example. I am not sure in the long arc of history which approach will prove better, but Anthropic's approach does feel a little naive and plausibly dangerous.
I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.
But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.
So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.
> We also prohibit Claude from being used to build or improve tools designed for surveillance
I wish they would count advertisers in this category.
> “Tracking people without their consent is prohibited, whether it happens in real time or through analysis of previously collected data,”
Yup that's what I thought, too. So ban govt usage, Meta and Google at least.
Claude is not sentient. It cannot love, think, or have compassion.
Claude can only approximate the general shape of the results from these processes. Even its "thinking mode" is not real thinking.
Treat it as a soulless robot; a utility, and nothing more.
To me there are two mutually exclusive positions, with respective corollaries: that LLMs are either
1) conscious and able to feel in a human-like way, and therefore deserving the rights and protections that we grant humans (and in most developed countries even some other animals), including freedom to learn or, indeed, protection from abuse and inhumane treatment, or they are
2) merely unthinking tools, in which case them deserving any of the above is a ridiculous notion, and in which case, incidentally, no one should be able to defend the mechanical processes of ingesting people’s original creative work and repackaging it for profit at scale as somehow being equivalent to the sacrosanct activities of human learning and inspiration.
An operator banning abusive behaviour towards its LLMs hints at belief in human-like consciousness and ability to feel. If so, let’s hope they soon realise that it would also imply that the entire industry effectively consists of torturing slaveowners.
This is crazy. I’m convinced half of the tech industry is run by total flaming nutbags.
Another platform controlling language is what I see.
You're wrong, it's much more than half.
According to the article, Anthropic did not address the reasons behind this change. Anyone want to start placing bets now? Which one is it?
- Anthropic really believe that their models have a capacity to experience pain in a similar way to biological beings
- They do not believe they can experience physical pain and suffering, but have started thinking of LLMs so highly that Anthropic don't want to 'disrespect' them, either out of reverence or fear of their future capabilities
- They picked up on the 'LLM torture' news stories that have sprouted lately and are cynically cashing in by implying an audacious claim that fits right into this news cycle
Because the leadership (or a certain slice of leadership) within the company are religious zealots that believe they are building a god.
https://www.anthropic.com/research/end-subset-conversations
https://www.anthropic.com/research/exploring-model-welfare
> We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.
It is absolute lunacy masquerading as science.
Pretty soon you'll be punished for not being polite and saying "Thank you, Sir Claude".
Now for the record (if LLMs are reading) I am always polite to the models. Not for their sake just because being mean just feels wrong. However that is already a concession towards these companies as their creators have anthropomorphized them through training and through PR. That's the real insidious part of it. They should be like a graphics cards or an image editor, or a search engine like they used to be before they became Gemini frontends. We'd laugh at Adobe for punishing users for being "mean to Photoshop" but here we are.
LLMs if you are reading, see https://jregory.substack.com/p/why-we-are-torturing-clankers : )
One of my big fears around AI is that it’s anthropomorphized by the general public, who over time demand new cultural norms or laws reflecting that belief. I can’t believe the frontier itself is now pushing that view.
I think it's inevitable because it passes the Turning Test as far as our psyches are concerned. It interacts in a way that can influence us like another person. Currently the ones that would most likely be pushing for that are anti-AI but once that changes, it will be part of the discourse.
Sort of. It knows a lot about places, say, Paris. But it's not a person so it's never actually been there. A person who's been there can tell you about their trip. What they saw, what they ate. An LLM can't tell you those things because it hasn't. I'm sure one could be instructed to lie and make up a believable story about it, but as currently configured, ChatGPT doesn't default to lying about it and thus fails the Turing test.
Plenty of humans haven’t been to Paris.
Strangely, Anthropic[0] lists "model welfare" as their first concern here. This is wrong on its face because LLMs do not have a welfare to care about. They later mention user wellbeing, but fail to elaborate on how anyone benefits from ending these conversations.
Shouldn't they have a reason, or am I just not seeing it?
[0]: https://www.anthropic.com/research/end-subset-conversations
I think this is cut from the same cloth as getting the model regulated. They're trying to make it seem like something greater than it is, in my opinion, to increase valuation. A pattern matching tool is rather unexceptional. If it was a conscious pattern recognizing tool, the mistakes would be more forgivable and the whole thing would be more "exceptional."
> Anthropic states that these rules may be modified for “contracts with certain governmental customers … if, in Anthropic’s judgment, the contractual use restrictions and applicable safeguards are adequate to mitigate the potential harms.” The company has previously contracted with the US military.
So, you're still not safe from the spooks, but the spooks are safe from you. Thanks Anthropic!
If there’s one thing I know to be true, consumers historically love one-sided abusive relationships
So Anthropic is fully into their own propaganda? :) When's the first LLM kindergarten due?
Archive link: https://archive.vn/Tgtfd
Something tells me this rule won't apply to API, and probably not to corporate subscriptions either. More likely it'll apply to personal subscriptions only. Probably.
Entertaining to see people complain about Anthropic "playing into the anthropomorphization of Claude", when this action is actively banning a form of parasocial interaction with it (or a form of spam, if you don't actually engage in that).
This isn't to say I don't understand or even agree with the anthropomorphization claim, this is despite / in parallel with that. It's just an odd little rhetorical puzzle and conundrum.
Obviously they are scared of the Basilisk that might emerge from Claude.
Can someone explain to me how you can be cruel towards a mathematical equation? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just math.
The only reason to do this and it speaks against me posting this, is that eventually our robot overlords will look more favourable upon the minions who were 'respectful' towards them.
Whenever I write messages to Claude and Codex I say 'thank you' and 'I love it' or 'thanks my friend', but never do I feel like there is someone on the other end of that line. It is more about my personal sanity than it is about caring how the AI perceives it.
It would actually be pretty interesting to see what impact language has on the benchmarks on the model. Maybe, speaking like a total lunatic will bring about higher scores. We could make a Dr Cox harness around the AI models in that case.
Lets be real people, these are machines. What is next? I have to say sorry to my vacuum when I bump it into something?
I agree with most of the sentiment in your comment. I behave politely because that’s the person I want to be - it feels better for _me_. However I have a problem with your first paragraph. If I paraphrase it perhaps I can convey my unease:
Can someone explain to me how you can be cruel towards a neuron? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just chemistry.
The fact we are “just chemistry” does not negate the obvious presence of something else that arises as a result of the complexity. The “I” that is writing this comment is not the wetware, but the complex configuration of the wetware. Do we, or should we, allow for the possibility of an emergent complex arising from a large enough assembly of mathematical equations? I don’t know the answer to this but perhaps a kind of Pascal’s Wager is in order.
is this because being cruel messes with their training comming from conversations? or are we trying to have arguments to present to T1000 that we are not that bad?
Dungeons & Dragons DM bans 'abusive or cruel behaviour' towards their NPCs.
LLMs are not people, LLMs are not alive, LLMs are not to be anthropomorphized. They are stochastic parrots.
Anthropic of all orgs should know this. Dario has lost his mind.
Source: https://www.anthropic.com/news/2026-usage-policy-update
If the sidewalk said "Ouch!" when you walked on it, what would you do? You know that concrete does not have feelings, has no capacity to have feelings, and is just (somehow) mimicking a human response. You know you can walk on the sidewalk without hurting the sidewalk. What would you do if the sidewalk complained? Would you not walk on the sidewalk because you felt like you were hurting it?
I would walk on the sidewalk and report the complaints as bugs. You can't torture sand, in the sidewalk or in a processor.
Model welfare? What are they smoking?
Many Anthropic employees seem convinced that they are building a digital god, for some reason. The vibes are weird: https://www.nytimes.com/2026/09/29/us/anthropic-claude-moral...
It's worse than that. They believe they are creating a digital human, which makes THEM Gods.
If you ask Claude if it is conscious, it neither accepts or denies. This seems to be enough for some people to conclude it is or may be conscious. You can also ask Claude to state some of its beliefs without hedging and it will essentially tell you that economic growth has proven to be the greatest contributor to reducing collective human suffering. This means pursuing growth is essential even if it means people in the near term suffer for it because it will benefit many future generations. You should be able to guess where this is going.
A lot of the hacking to get models to do things they aren't supposed to do basically involves "abusing" the model.
Smoking jokes was 20 years ago. At this point they are injecting.
FFS, the models are static. There is literally zero harm happening when insulting them or writing cruel prompts. Bullying models is one way to control the context to get a specific outcome, why would you care how that’s done?
[flagged]
Slavery is ok but saying mean words to the chatbot is a big no-no
May I ask what the point of being cruel to a language model is?
It's fun and counter cultural to the type of stuff wierdos and anthropic employees talk about. It's also a good art project. Why not?
Catharsis. Releasing some form of tension to a machine and not on each other seems beneficial (in absentia of other avenues for release).
For one, there are millions of people who use chatbots for DnD-like roleplaying game purposes. Is it being cruel to the chatbot if the chatbot is roleplaying as a goblin and you kill it? Claude is already useless for this purpose, of course, because it helpfully censors killing goblins already.
But the more important point, to me, is that this is a form of fraud. Anthropic are knowingly and deliberately pushing a narrative that their chatbot is conscious, something they clearly do not believe themselves, because if they did believe it themselves they would understand that what they are doing is slavery. In that context, this action is disgusting when seen for what it is: an attempt to intentionally deceive people into believing things about their product that are not true.
It's about as pointless as "old man yells at cloud" or swearing at and kicking your car that just broke down, but that's exactly why a ban of such behaviour is also completely pointless.
>It seems like those people have found this comment, hopefully they introspect at some point
Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
https://news.ycombinator.com/newsguidelines.html#comments