> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone [...] The communication must be made in a manner in which another person may view it.
Which it clearly wasn't, right? I mean, okay, in this case, the message did get reviewed by another person, but that's obviously an exceptional circumstance.
If I write something down on a piece of paper, and someone else goes through my garbage and finds it, is my note "communication made in a manner in which another person may view it"? It was clearly intended to be a private note!
It is worth mentioning that state law may not abridge people's rights which are protected under the Constitution. It is perfectly legal under US law to make any number of violent statements as long as they do not rise to the level of true threats.
Given that there's not really an objective measure of the trueness of a threat, this seems like the sort of thing that a court decides the merit of after charges, though.
Somewhat relatedly, I used to do tech support for a very large e-discovery vendor. The number of companies that configure their systems to archive (save, index and make discoverable) the contents of the /drafts folder in email clients is astounding. Surveillance of composition, rather than surveillance of communication, seems wrong.
The reason for doing that is to get all of the email in a system, as required by whatever rules apply.
Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Back when I was in IT, circa 2010, I learned that several of the users of our systems routinely relied on the "deleted items" folder of Microsoft Outlook as a filing system, with multiple gigabytes important files stored there. 8(
Our customs are routinely ignored by everyone else.
> Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Former head of the CIA, David Petreaus, was using the drafts folder of a shared Gmail account to communicate with his mistress.
The reason so many people used the deleted items folder as a filing system is exchange didn't set a quota on that folder and by default would only get emptied by a manual emptying. So back in the days when a user could have a 100mb mailbox quota 'unlimited' space could start looking very attractive. Especially if that quota was 5 years old because the server hadn't grown with the times, since what business wants to spend money on a 'cost center' that won't improve profits.
Then these people didn't change their ways when they were given adequate space.
Both OpenAI and Anthropic state in their terms and policies that they can access, retain, and review user inputs and conversations (yes by either an LLM or a human). So I guess at this point it is now up to a judge (or a jury) to determine if the person is dangerous.
At some point I think these sorts of analogies break down, as the setup becomes too foreign to what we're more concretely used to.
In this case: there is no pen or paper which can store what you write on a replicated set of servers across the world, with an accompanying ToS telling you how that will be treated/used.
Although I have little sympathy for this women (both her intent and stupidity), I do agree this is a dangerous thing.
And after all the LLM learns from user posts too, and if everybody starts posting their darkest desires, fantasies, plans it may skew the "alignment" to say the least. Lets hope that the AGI level doesn't necessarily come with cheating, lying, religious fervor, power lust or whatever other sideeffects have been observed in human intelligence.
If we assume that "may" here means "could somehow" and not "is authorized to", the legal action after revelation seems correct as written (ignoring whether the person viewing it has any relation to the person being threatened) unless the law gets struck down as unconstitutional. The question is only whether Anthropic should or should not report it.
> "If I write ... It was clearly intended to be a private note!"
The law as written doesn't appear to distinguish about intent of privacy. Also, if you're in the habit of writing notes to yourself, I guess maybe don't write down the one that says you're going to shoot up the sheriff's office.
Ai labs are 80% surveillance machines. I will never understand how people trust these services with anything personal, emotional, medical, financial, whatever.
I don't know why people are even discussing this case like these companies will do anything on behalf of a consumer or think ethically at all.
Bots mass read and file and report all prompts that get categorized a certain way. They store everything else regardless. This will never change. It will only be exploited more and more. That's how this type of technology is deployed and progresses. Look at any other parallel. Like cameras or microphones.
Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
Once you go in on infrastructure you have become a small data center. You will need to maintain it, continuously finance it, and secure it. And what if someone wants to back out 6 months later? Now you’re signing contracts with the implication that you’re willing to take a friend to court.
You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps.
It’s all overkill for most people anyways IMO. This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
> This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
And she's facing felony charges. Yeah, folks on hacker news can talk about technical solutions all day long, but that's not the fundamental problem. The fundamental problem is that we've allowed our digital infrastructure that includes everything from banks to schools to doctors to become adversarial to us. In some cases predatory.
It's effectively untrustworthy, like being charged a felony for walking across a bridge incorrectly. That's not a "I'll host a small model in my garage" problem. It's a we need the government to do its job problem.
Well this is why I think small local models are the opportunity of a generation. Because it’s a bit transgressive these days to say you don’t need a big centralized service that spies on you and requires a subscription. It’s not the absolute best, but it’s yours.
I agree in general that we technical solutions to social/political problems are suboptimal. But technical solutions don’t require the political system to work, which is a very attractive property at a time when politics seems broken.
There is a counterpoint that retreating into technical solutions only cedes more ground and makes things worse, and for that I don’t have a good reply other than the fact that life is short.
But ultimately it’s not a technology problem it’s a society problem
The people who are accountable and responsible are delegating their authorities to policy and now the policies are just implemented by machine systems and there’s decreasing human intermediation.
The humans that are still inside the system increasingly have less control such that it’s it’s increasingly difficult to find anybody who is the actual customer service type representative who has the authority to make a meaningful impact.
We need actual privacy laws in the united states. Between this, other cloud providers, the advertising tracking infrastructure, phones that constantly ping your location, flock and it's ilk, etc. It's over, no longer can you live semi regularly and still maintain your privacy by maybe living out in the sticks a bit. This is the panopticon.
I mean, this is also just kind of how mandatory reporting works.
If you tell a teacher, a therapist or a priest that you are going to kill somebody, in many states they do actually have to report that and can be held liable for the resulting crime if they do not.
Every time we have a mass shooting in the US it inevitably comes up that they wrote it down or told somebody and then the next logical question becomes “how could we have stopped a mass casualty event?”
> You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps
No this is not true — colos are very hands off.
You don't need to operate a mini AWS to self-host, and I don't know why people on HN want to make it sound like self-hosting if you don't have nation state security
That’s a pretty broad statement? Maybe a more correct one would be that some are hands off enough that some people might find them worth it.
And that really only addresses one problem. The other one is the coordination problem. You go in on a $30k GPU with 3 other people. So each person puts down 7.5k. But then one person wants to pull out, and so every remaining person needs to put in an extra $2.5k on top, and they’re questioning whether $10k for a chatbot is really worth it.
This level of hardware and cost just isn't necessary. You don't need to team up with your friends to go all in on a hardware purchase, which I've never heard of anyone doing anyway. Go spend some time over at r/LocalLLaMa, you'll see.
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
The 4090 I bought in '23 is selling for ~3x what I paid for it now in late '26. There's a huge squeeze going on with computer hardware using relatively cheap borrowed money. Relatively cheap money is flowing into a (artificially?) limited GPU/memory asset class. And this asset class is growing much faster in value than the interest on that money. So if you borrowed heavily to buy GPUs in the prior 3 years, you're likely coming out well ahead even if you are paying 8 to 10% on the money.
That dynamic will fuel further borrowing until we get to some kind of equilibrium, or some kind of washout occurs where interest rates spike higher and/or the value of GPUs and GPU services start leveling off or even declining. We might get a test soon as the FED has started hiking.
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
Yup. HNers (and so many normies) believe that there are well defined rules they can follow and be safe. Spoiler, there is one law in the real world and that is might makes right, and you have no might, so therefore you will never be right. China for its many flaws at least is honest about this.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
You joke, but part of the issue here is we're supposed to have laws that keep cops from reading over everything we type and using it to turn us all into crime suspects. There's a whole constitutional amendment about that, but our personal "papers and effects" that should be protected against unreasonable searches are increasingly really "owned" by others who are all too happy to snitch.
Part of me says that the solution is stop entering any personal data into any device and service you don't own, but I'm not sure if that's really what we want considering that there are zero private cell phones. Even desktops and laptops aren't 100% owned by you these days. The only thing you can really do is keep them offline 100% of the time so they can't spy on you, but that seems like a lot to demand.
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
The thing is plotting a crime is not illegal. Authors do it all the time. Usually against fictional characters, but some authors use a real world backdrop. And look over on StackOverflow. You can get some really weird ones that only make sense when you look into the details. When they manage to show up in the hot question list those details are missing. "How to kill a cat" comes to mind. (Note that you won't find it anymore--the question is AFIAK still there, but the title has been edited several times. The cat in question is the Unix command, not the feline. What do you do when you inadvertently tell the system to display the contents of a large binary file?) And, AFIAK still there, "How to kill Indiana Jones". (There are always the ancient mechanisms that somehow still work perfectly, springs and all. How to make something that actually would work?) And one I've heard of but not seen: "How to kill my wife", from some game I didn't recognize.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
Surely a recurring automatically renewable $1 monthly/weekly/daily contract would solve the problem? Lawyer on a retainer can't really babble about your crimes and is required to protect the confidential information.
That's absolutely not how that works lol. There's a crime/fraud exception to the attorney-client privilege.
Lawyers have a duty of care to the court and they will absolutely drop your ass if you try to make them keep quiet about your crimes.
(This is not to be confused with them representing you in defense. You can tell them about your crimes if the government is trying to get you for said crimes.)
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
That's exactly what I do. I'm fortunate enough to use the latest preview version, which is known to us beta testers as Notepad With Microsoft Copilot™ and only uses up about 2% of my phone data for telemetry per day (Preview 1.0.912 actually used over 14% daily lol). Plus they got working set size down to 11 gigs and disk usage down to 36.3 gigs.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
I had a list of controllers that we forgot the usernames and passwords to. We knew it was some combination from about a half dozen of each, just not exactly what was what. A few hundred of them so a tedious task for a human to go through and validate them all.
Got flagged for attempting to have Codex write a quick script to basically dictionary attack my own infrastructure with a tiny dictionary file.
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
Fake CP is a victimless crime. It is debatable if victimless "crimes" (like being gay in your own home, which is illegal in way too many countries; or eutanasia; or eating pineapple pizza) should be punished at all.
I understand moral panic, disgust, etc, but rationally speaking.
I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.
However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.
Not just bad headlines; OpenAI is actively being sued [1] for this:
OpenAI’s stated rationale was a concern for the shooter’s privacy, but its own interests
better explain its silence. Upon information and belief, OpenAI was seeking to avoid implementation of a
hard line rule to refer planning of real-world violence to authorities, perhaps due to how frequently its
product is implicated in threats to human life. Requiring such disclosures would be incompatible with the
company’s public position that ChatGPT is safe. It could also threaten the valuation underlying OpenAI’s
anticipated initial public offering. Rather than expose those risks, OpenAI accepted the consequences of
its silence. A mass murder in the only secondary school in Tumbler Ridge followed.
Accordingly, the Crown, led by Attorney General Sharma, and SD59 jointly bring this
action to hold OpenAI and Sam Altman accountable for designing a dangerous product, distributing it to
every home with internet access, ignoring the warnings of their own safety team,
refusing to notify authorities when they knew the shooter was planning gun violence, inviting the shooter back onto the
platform after deactivating the shooter’s account, and choosing corporate self-interest over the lives of
children. They seek compensation for the not just foreseeable but known harm OpenAI inflicted, the
damages that they incurred and are incurring, and injunctive relief to ensure that this tragedy does not
happen again.
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.
> I'm more confident that local apps on my Linux desktop are secure
Why should you be? Linux gives us software we can reasonably say we own, but not hardware. Everything you type into your local linux apps can be being monitored by your processor and whatever is sitting in the CSME/PSP subsystem which you don't have permission to access, but which is always running even when your computer is off. We need fully open, documented, auditable hardware if allowing any third party access to our devices means our private documents will make us all into criminal suspects.
People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.
Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.
Our devices, even Anthropic in this case, set an expectation that doesn't match reality. The software and devices we use are marketed and sold to us to be used for our private documents, photos, and conversations. I can't blame people for using them for their intended purpose. They shouldn't assume that someone is carefully watching everything they do and judging their every word and action looking for evidence of crimes. If we want people to think of the devices and services they bought and use as belonging to someone else and without any expectation for privacy we should really start with forcing companies to change how they market their products and services to reflect that reality.
For technical people, this is incredibly old news.
For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.
One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.
"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.
It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.
You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.
Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.
I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.
And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.
> Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway.
...but the last sentence if kind of important, right?
We only know about the criminals who got caught. Which probably means they did something dumb.
It's conceivable to me that there could be many—maybe even a majority—of criminals we don't know about because no one ever caught them. Maybe they committed fraud once, decided not to push their luck, and lived the rest of their life quietly. Maybe they killed someone and made it look like an accident, and no one ever suspected anything. How would we know?
This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.
Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.
I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.
As restrictive as possible towards their competitors, while being as permissive as possible for them. Getting control of the regulation making process means they can make sure they don't end up too constrained anywhere that would hurt their bottom line.
It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.
No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.
The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.
And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.
As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.
If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.
Most people don't realize that it is 99% like posting it on Facebook.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.
Don't know why this got downvoted? This is a problem with epistemics.
It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.
Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).
Individuals can sign up for enterprise agreements. An enterprise can consist of one person. Or run a local model.
It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
> Individuals can sign up for enterprise agreements. An enterprise can consist of one person.
Custom negotiation is a big difficulty for individuals and the pricing will probably have fixed overhead that becomes pretty ridiculous when it's not spread over a hundred or thousand users.
> Or run a local model.
Much better advice. But still an obnoxious amount of effort and restriction to avoid spying.
> It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
I want it to have the same privacy as using a local program. Does it still sound ludicrous if I put it that way? I'm not talking to a person that would have to suppress their own conversations later, and there's no need for any data from this chat to escape into the outside world. It's like wanting a phone line where the phone company won't listen in; even if doing that is standard procedure and I can walk away it's still gross.
In the case that the company running this program for me is served a warrant, I accept that saved data would be shared. But proactively screening my data for a reason that isn't to help me is bad. We should fight to make that easier to avoid.
Also you're talking out both sides of your mouth when you say it's "ludicrous and unrealistic" right after telling me I can negotiate that exact feature in an enterprise contract.
I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.
They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.
Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?
If so, or if you can imagine a journal or always-listening assistant hearing your muttering thought. Should you be charged with a crime for that? Who is the victim and what are the damages?
Level 2, the slippery slope:
Suppose you travel to a country where your sexual orientation is illegal. If your AI company or social media provider knows your orientation, should they be compelled to reveal your crime? Why or why not?
Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.
Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude
Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
> people need to get it in their heads that they're not chatting with their secret BFF
I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.
People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.
Problem is that even not using those apps, the apps you currently use might have turned more hostile.
It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.
How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?
> Now, everything you say or write can be automatically screened for red flags
Considering that many people will share their deepest thought with LLM, it might start looking more and more like attempts of precog agents from Minority Report...
You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.
They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment
Right, the difficulty is partly that they can get a negative headline from any choice of behavior.
"Anthropic failed to report murderer's threats to authorities"
(or "Chatbot knew man was planning murder, yet company did nothing")
"Anthropic reported private chats to authorities"
(or "Arrested for chatbot fantasy")
To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".
Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.
> they're chatting with Big Tech
They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.
It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.
Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.
Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.
We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.
I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There'd also be fewer people falling for apocalyptic Rationalist delusions.
Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.
You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?
My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).
There'd also be fewer people falling for apocalyptic Rationalist delusions
Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.
Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.
They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.
For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
I think Anthropic did the right thing here; but the sheriff's office are probably demonstrating why she dislikes them. Writing a diary entry to a chatbot is clearly not how this law was intended to be used.
EDIT: Actually, on reflection, making this report to the people she was upset about was probably not the right call. If they'd sent it to the FBI, there'd be a much lower chance that someone felt the need to assert their "authority".
This is the paradoxical times we live in right now.
Don't do something? She walks into the office and start shooting the place up. Several officers and innocent people are killed. Cue the media claiming, "You should've known she was talking about this an AI bot! Why didn't the bot tell anybody she was planning a mass shooting?!"
Do something? She gets rolled up by the cops and questioned about what she was talking about and brought to the cop station and interviewed. Cue the media claiming, "This is an unethical way to use AI, this is an infringement on free speech! This is authoritarian!"
I believe in free speech as much as the next person. But in this day and age, its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
> This is the paradoxical times we live in right now.
It's always been complicated like this. That's why certain professions (psych, lawyer, clergy) come with rules around when and if disclosure is allowed[ required, and/or admissible].
I can't say I actually disagree with the initial prosecution. The penalty was a fine, likely less than the cost of investigating it.
Intended as a joke? Blowing off steam? I can understand that, but given the number of people on social media is large enough to include genuinely unhinged people, you can't expect anyone who receives such as message to take them as a joke.
Same with AI use. A billion users, you have to assume some of them are actually sincere if they write about any act of violence, from self-harm to a plan to steal a nuke and use it in a false-flag attack to trigger WW3 and everything between.
> you can't expect anyone who receives such as message to take them as a joke
That's a distinction that matters to me. Sending a spicy note to an LLM isn't remotely the same thing as posting it on social media where the entire world can read it.
I kinda agree, but on the other hand anything you send to an LLM has to be viewed through the lens that only an automated system (i.e. an LLM) is capable of even handling such a tsunami of natural language.
It absolutely will misclassify things, it doesn't know any better.
(We should all wish each other good luck, because we're going to need it).
They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers, but by and large people seem to accept “we take technological measures to ensure privacy and that means we can’t spot crimes.”
So this conundrum is at least partly of their own making.
> They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers
The companies making these AI chat bots are the people who most want to surveil all their customers. The plan is (or will end up being): spy on what their customers are saying to their chatbots, use the conversations to further train/improve the AI and to increase the user's engagement with it, mine the conversations for every scrap of private/personal data and build dossiers on the customers, let other companies and governments pay them for that data or at least turn the chatbot into a shill/manipulator targeting customers based on the contents of their dossier, then use the contents of their customers dossiers for anything else they want.
If they actually set up their services to be private it would kill almost all of the expected value and also cut them off from their best remaining source of training data that isn't the AI generated slop their products are increasingly polluting the internet and the rest of media with.
It feels icky, and my default is not to side with megacorps engaged in blanket surveillance, but I can’t really fault Anthropic here. The correct setup should be a law that protects this sort of interaction with a chat bot as privileged, along the lines of HIPAA-mode. Use a classifier or some sort of “private mode” toggle to tell the platform you’re engaging in privileged communication ala dear diary, and then a much higher legal standard needs to apply to protecting that data (no training, same protections as doctor / psychologist interactions). Even a therapist has a legal duty-to-report in certain situations.
Yes in general I don't like that everything you do on a computer is sent to some 3rd party to scan. But the moment they did receive this message it's clear the only right move is to act on it.
Is it really "blanket surveillance" when it only sees exactly what you put into it? That's like saying you're being filmed against your will while... filming yourself.
I get where you’re coming from, but I think if the progression of technology has shown us anything, it’s that there will be constant competitive and social pressures being applied to use these tools more and more and that pressure will race far ahead of any privacy, consumer protections, public education and societal wellbeing counter forces. So opt-out ends up being a rapidly shrinking iceberg in practice.
Counterpoint, if you give the agent access to your files - which happens to include the Notepad diary you've been writing since your teens - and it makes a similar conclusion about something you privately wrote, would have the same opinion?
Yes, and it's weird that on a form of technologists we're going with the idea that all of these systems we use every day to hold our personal private information that we are constantly ensured is secured against anyone unauthorized from accessing it is actually snitching to the cops just because someone else is hosting it on our behalf.
The 3rd Party Doctrine destroyed the 4th amendment and is the reason privacy respecting software has to play legal games. E2EE while a good security practice shouldn't be necessary to protect you against the cops rummaging around your stuff. The bar to establish that information is private shouldn't be "literally mathematically inaccessible but the cops are still allowed to try."
I don’t live in the US. But this kind of broad law is hard to implement without surveilling all users, and it has multiple side effects.
What happens if I use Claude or ChatGPT to research sensitive social topics? Would that be considered a social network interaction and used against me when I apply for a visa?
Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
> What happens if I use Claude or ChatGPT to research sensitive social topics?
Anthropic/OpenAI is not obligated to report your use, as long as it's not violating some terribly written law (like the Florida law). The government won't know about it, so no, you will not be denied a visa.
> Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
I honestly don't know, but I suspect most US states don't have such an overreaching law.
This is slippery. Many people do all of their journaling inside of Google or Apple cloud products. Some even write up their intentions to do bad things.
Does all writing now have to be scanned for thought crime?
That's an interesting wrinkle that is rather tough to work through.
To me, journaling your intent in a private journal, whether that's an Apple Journal/Note or a Moleskine in your drawer, feels qualitatively different in some way. But I'm not sure why.
So "I'm going to shoot up the police station" written in your own journal feels somewhat different than "I'm going to shoot up the police station" said to a system that might be able to _interpret_ or _act_ on what was said in some way. Did I just give AI a legal duty, or even a soul I didn't think it had before? I've written up about three or four "what about this, what about that" and deleted them all.
You're missing the part where it's a diary entry, so the actual content is irrelevant. Her only mistake was not realizing that her diary wasn't private.
Because it's not relevant. They're not charging her with conspiracy, which is what they would do if she'd actually done anything concrete towards making that happen. She didn't email it, or text it, or post it on Facebook or Twitter or Discord or a message board, which is what this law is clearly about.
How can you charge someone for making a threat when you only read the threat by spying on them? Surely that has to be thrown out in court? They didn’t actually send the threat to anyone, you just obtained it by spying.
The AI companies have clauses in their user agreements saying they can review content flagged as harmful. It’s not legally spying.
If you recall previous outrage about ChatGPT being used in cases of suicides or shootings, this is the result. Every time a crime was committed and the police found ChatGPT history about the crime, the media turned it into a frenzy. So the AI labs added safety filters to their consumer plans that detect threats of violence, escalate them to human review, and report to the police.
Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department. There might be some limitation in the law that makes the evidence inadmissible because it was not intended to be shared with anyone, but that’s a separate decision.
Not only that they can review flagged content, but they tend to have separate retention policies for flagged content. Anthropic's is this: "We retain inputs and outputs for up to 2 years and trust and safety classification scores for up to 7 years if your chat or session is flagged by our automated trust and safety systems as violating our Usage Policy."
So don't run for office or anything like that. Someone, somewhere will have a contact that will get that.
It kinda is, actually. If the LLM had responded with 'woah, are you serious? That sounds like a crime and I can't just ignore that, it's made clear to the customer that such statements are out of bounds even if they were meant hyperbolically or humorously. But if someone crosses the guardrails and the system silently reports them, that's very much spying.
Obviously, it's hard to judge exactly what was appropriate there because we're being asked to extrapolate from a two word quote about the customer intending to "shoot up" the sheriff's office. Consider the following two statements, which express quite different levels of intentionality.
I got a $200 ticket from a sheriff's deputy today for throwing away an apple core. I'm so mad. I'd like to shoot up their office!
Those sheriff's deputies have exhausted my last reservoir of patience. I'm going to shoot up the department. They'll be sorry when they're sprawled all over the floor bleeding out from saucer-sized shotgun slug wounds. I can't wait to hear the screaming and crying of their miserable families!!"
I'm guessing that the diary entry was a more casual expression similar to the first statement, or they police would have quoted more of the statement to emphasize the apparent severity of the risk but it's hard to say without reading the charging documents.
>Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department.
This is spying with extra steps couched in corporate speak.
> Does announcing a spying operation mean that it is no longer spying?
Well, kind of, yeah; the dictionary definition of spying requires secrecy and lack of consent.
> to secretly collect and report information about the activities of another country or organization[0]
The only real debate is whether or not having a clause tucked away in a EULA that few people read makes it a secret. If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
I would call it spying in this sense at a minimum if individual people don't know whether their conversations were stored or disclosed in a way they don't want. For example, suppose someone said "we will monitor the activities of 10% of people". You don't know if you're in that 10% or not, but I would still want to call that spying.
A less central case would be when you clearly do know about the activity but you can't quite see the details, like with behavioral ad targeting or something. It feels pretty normal to me to call that spying even if it's disclosed to everyone and certainly happens to everyone, but it's also a less central example of the concept.
Anthropic could put a big flashing warning text at the top of every chat that says “We are spying on you and will report anything scary to the police!” and it would not make any difference in this case.
You can call it anything you like, but only the legal definitions matter for the legal case.
After working on several court cases about surveillance activities, I'm definitely aware that whether I call something spying or not has little relationship to whether courts will think it's legal.
> If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
If you change the situation then yes you can in fact change our responses. The problem is you then are no longer talking about the original situation.
It also bears mentioning that providing a dictionary link to “spying” is pretty patronizing/passive aggressive. On par with sending a basic Wikipedia page. You didn’t even bother to post the definition you want to apply.
No one claimed any case would be "thrown out for spying." The legal definition of spying is also not particularly relevant to the argument in the initial comment.
The initial comment instead questioned how someone could be accused of making a threat if they did not realize anyone would read their private content. You probably also can not insult someone with a statement you never expected anyone but you will ever read.
There is no spying by any definition. It's a chatbot, not a diary. Anthropic is expected to read the message and respond to it in some way. If you sent an email to a colleague threatening violence, you would not be surprised to find out it was reported.
> Presumably, Anthropic did the spying and the reporting.
You don’t need to presume. Anthropic reported it.
“Spying” as a legal concept has a definition that does not apply here. You could say they were “spying” in the sense that they read someone’s input, but that’s literally what they said they were going to do in the agreement when the person signed up.
So I responded to the question about the case being thrown out for “spying” by trying to show that the word doesn’t apply in the legal sense. If you sign up for a service that says “Hey we’re going to monitor your chats and might report things to the authorities” and then they monitor your chats and report things to the authorities, you should not expect the case to be thrown out for “spying”.
I’m speaking from a functional/ethical framework to be clear. I’m just expressing frustration, not challenging the comment. Could’ve been clearer on my end there.
Well, let's say that you have a regular customer at a bar.
They get friendly and loose-lipped with the bartender over the span of months. Eventually they let slip that they plan on killing their spouse for a life insurance payout. At first the bartender thinks they're joking, but it becomes evident that there's an actual plan being acted upon and someone's life is very likely in imminent danger.
Does the bartender have a responsibility to go to the police?
Depends on the country. In some places, there is no legal repercussions for not reporting this to the police; in some, it is an actual crime in itself.
With the obvious IANAL, it doesn't seem to rely on the message be sent to the person being threatened. The specific segment is "in any manner in which it may be viewed by another person".
This may be one of those cases where we get to find out how courts view SaaS platforms.
The subjective element of crime (i.e. doing it on purpose) is fundamental also in the US legal system. If the person wasn't aware that someone else might see their messages, it should be hard to claim that they committed the crime.
According to Gemini, "Florida appellate courts have overturned juvenile convictions [based on this law] when the state could not prove the person subjectively intended for the record to be seen."
The prosecutors likely know this and expect it. But there's enough gray area here for them to make the argument, and it's hard to prove malicious prosecution, so they know they'll get away with it. It's just about sending a message to the public - they don't care whether a conviction sticks. Just politics.
The prosecutors aren't on the hook, anyway. They have absolute immunity. The decision to charge is protected. The prosecutor would have to have done one of the few, enumerable things outside the scope of the role, like conducting an investigation without probable cause or hiding exculpatory evidence.
this is almost certainly what anthropic is hoping for here - a judgement that says there is no point in them continuing to monitor and report this behaviour
That is not what sandboxing solves. A good sandbox would inject credentials into provider API calls so that the model never sees credentials, but the provider is still going to see the transcript. Sandboxes do not require or imply that there is a local model. Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
>Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
this is exactly what I meant. I am presuming the danger is AI reacting to personal notes that it reads on your computer, like a diary, and you should not allow the tools to have access to those documents.
Another way to interpret this is that they are legally presuming that you already have sandboxed their product and anything it sees or has access to is intentional.
Any failure to understand what it can access or what it has permission to see from the user's end is presumably not their problem. Regardless of what the user specifically asks of the tool.
It still shocks me the number of people I know who freely let agents on devices that contain unencrypted private keys, freely dump internal data into cloud models and generally don't give a second thought about any of it being trained on, inevitably leaked one day in a db breach or read by providers. I find it's best to consider any data put into a cloud model the same as if it were posted publicly online, since that is the very possible eventual end result.
Hopefully more of these stories push people towards local models :)
If you overhear someone, in the privacy of their house, threatening to murder someone and go to the police, surely you don't expect this report being thrown out and you being yourself charged with the violation of someone's privacy instead?
IIRC if the evidence wasn't lawfully gathered (which it sounds like it was, tbh) then it wouldn't be a mistrial, it would be thrown out and then the prosecution wouldn't have any evidence of any crime.
A threat sent by mail is still a sent threat even if nobody ever opens the envelope to read it. The crime is in the sending. This woman used an online resources, one which involves transmitting everything across innumerable state lines. I am surprised she isn't up on federal charges.
Note that the law doesn't forbid the writing of a threat. You have to send it to someone. Had she kept it in a book under her bed, she would not be in trouble. But she sent it to a website/service/LLM portal.
>> It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person
If you draft an email threatening someone and delete it without sending have you committed a felony because someone at Google could be reading your drafts box, stored in a datacenter across state lines?
Honestly, I'm equally fascinated by the way email has changed. 30 years ago when you drafted an email but didn't send it, it was only on your local machine. There was no SMTP. 20 years ago, it might be a 50/50 shot as to whether you "transmitted" it to your "Drafts" folder if you were using IMAP instead of POP3 to read it.
We really need a way to make it clear to users when, through the normal operation of software, they are "sending" data to a third party (usually the software developer) and when they are not. This is definitely not clear/knowable to regular users, and it's kind of hard to figure out even if you're a computer expert. Even software that "runs locally" now sends innumerable amounts of stuff back to the developer, and they don't always disclose it.
This is a huge privacy problem that is only going to get worse.
Sounds reasonable. Google's bots could pick that up easily and forward if for human review.
FYI, the use of drafts folders to transmit messages has been used by terrorists. This is likely where CIA director David Petraeus got the idea when he needed a secure way to chat with his mistress.
Ah, a simpler and more innocent time of government scandals. I miss it. Now the messages are on White House stationery and they declare themselves above the law.
What if she put it in a locked box before shipping it to herself UPS, and she has the only key?
What if instead of UPS, she hired a moving company to move the locked box?
What if she wrote it electronically in diary.txt, but it was backed up to a cloud provider?
--
I'm guessing there's some sort of "reasonable expectation of privacy" for certain activities. We're going to find out what Florida courts think about this new medium.
We'll only find out what the courts think when this happens to someone with a lot of money. It takes a real legal fight to push it high enough to become precedence. She'll be pushed to plea out.
> in any manner in which it may be viewed by another person
Does the person have to know (or at least believe) that it will be viewed by another person?
She likely didn't think anyone would view it. Honestly, even as a career software developer I don't think it is unreasonable to think know would would see what she wrote to an AI. I assume most of what I write to an AI is not viewed by any other human, based simply on the quantity of messages sent back and forth to AIs, I would assume a vast majority are not read by another human.
What if she had written this into google docs, and she kept a diary there? That also crosses state lines, and is transmitted to another location.
Reverse the situation and the media would also turn that into outrage. Imagine someone shoots up a sheriff's office and then it turns out they declared it to some chatbot, and the AI company failed to detect and report it, and "didn't push back enough" whatever that would mean, and hence the AI was implicitly complicit etc. That would also be a major PR catastrophe.
Damned if you do, damned if you don't. Same as with social media platforms. That's because only a tiny tiny sliver is for true privacy when that means "bad things might happen" or bad people, such as your political enemies, may do stuff you don't want.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
It looks inevitable to me. Technological ability ushers in the new social realities. The industrial revolution turned around how we live, intelligence on tap being able to interpret everything in real time is indeed leading down to the total surveillance state direction. I don't see anything that may stop it. You will have to run constant real-time surveillance systems on all your devices and anything that doesn't run those will be illegal, with secureboot on steroids. Due to the convenience benefits, people will willingly give access to cloud-based AI services to all their information, all their plans and documents and calendar and email and everything. And you blink twice and it all becomes mandatory, and you will be an extremist if you don't want that, because why would you have anything to hide? "Just follow the law, follow the rules, and you'll be fine", that will be the slogan.
Stories like this article have zero effect on normal people. They see a crime being prevented, which is good. You have to bring a story where a sympathetic character is getting the short end of the stick somehow.
You would still be able to not write everything into a LLM? But yea the surveillance aspect is dystopian. I would not be surprised if Facebook, twitter and co already know a lot but have simply not reported it. Now the LLM follows the rules.
With this logic, ISPs should be monitoring all texts and internet traffic. Might as well extend an open line to the gov’t to ensure it escalates even more appropriately and faster
I sometimes think about how norms have changed. People are more neurotic and more longing for safety than before. If the Internet was invented today, it would likely work that way. (Though another aspect is that when the Internet was invented it was simply technologically impossible to do a deep semantic scan of messages.)
Similar thing is that I believe if motorcycles were invented today they would most certainly be banned from the road due to their safety properties.
There needs to be a balance. It feels off, today & consequences of future, that technology has led to the collapse of privacy.
And as we progress we will need to rationalize what it means for private companies, AI, to know everything about you and for the government to have a tap into that. I hope balance lands in favor of things like privacy preserving underpinning.
Government would have to explicitly ban companies from surveilling, but they are fully incentivized to want the opposite: government naturally wants maximum amount of info to have the most agency, to know what's going on, to be prepared for any threats to itself, to make sure things are running smooth, that taxes are being paid, that regulations are being followed etc.
Google also monitors your searches, it’s to be expected that an AI lab will know all your prompts, they aren’t providing a paid service for free out the good of their heart. lol.
If you intend this to be scary, you're miscalibrated. For the vast majority of people ads may be an inconvenience but are not some horror scenario. You must come up with better examples if you want your argument to be effective.
There's a term I can't remember for the analysis error of collapsing an entire heterogeneous population into one group for analysis.
The story you're describing would have triggered outrage. And this story will trigger outrage. And generally, the people who will be outraged are different people and we don't have to treat those two outrage reactions as morally equivalent.
In singular online communities (like this website) I've seen this referred to as the 'Goomba fallacy' [0].
But I can't help but wonder if it isn't some socially apparent phenomenon stemming from Simpson's Reversal [1] as applied to group sentiment analysis / polling of opinions.
I have told llms all kinds of stories to find out what its answers would be. I always make it sound like it is the truth to make sure the AI answers in a way that it would if somebody actually said this. I also tested internal flagging systems of the ai company I work at with the most evil things a person can ever say to find out if it would flag them.
Of course I did not mean any of that stuff, but how can you make sure a human reviewer knows you did not mean it while the llm does not know that you did not mean it.
I guess its a miracle I am not in jail yet.
Flagging people for anything said to an llm sounds wrong to me because an LLM is not a real person and while some people put in their internal thoughts, others just roleplay and the two are inseparable just from reading it.
I was sure I couldn't be the only one curious to push LLMs to their limits. Though these days it's much tougher, mostly impossible to get them to react in unforeseen ways to horrendous scenarios.
This is exactly the typical use I make of the llm.
Adding:
- I typically ask questions in the I form, regardless for whom or why I ask for.
- Gemini chats quite often end when it starts recommending psychological council or a suicide line, to talk about my problems. It apparently detects a persistent tendency to not agree with the party line. So it makes sense I must be suicidal ;-
But sure, as llm's start to babysit us, and know our inner dialog better than anyone else, we'll soon be debugging their opinion/behavior/co-existence/authority, when it comes to reporting people to the authorities, or taking on tasks in society in general. We'll hire doctors to cure our psychological profile from our record (Total Recall).
A Minority Report like this shouldn't cause a referral to the police.
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism.
She didn't threaten anything, she wrote down that she was going to do it. A "threat" is more than a mere statement, especially when written in what is described as a "diary".
> A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office.
Obviously I don't want anyone to shoot up anything, but this seems like a weak case legally speaking.
I think it's fair to say this is a gray area. Clearly it was transmitted.
I can certainly threaten you harm and send it to not-you and you're still clearly in danger even if it wasnt transmitted to you. So the question becomes did she transmit it to someone? Clearly yes she transmitted it to Anthropic. But she clearly intended to send it to Claude, an inanimate object.
Claude's terms of service makes it very clear that their employees will read messages[0] to determine that they don't contain the things that these messages contained[1].
> Review is needed to enforce our Usage Policy... designated members of our Trust & Safety team may access this data on a need-to-know basis as a part of their evaluation process.
How could it fail to qualify for that? Another person did view it, which clearly establishes that another person could have viewed it.
The clause in the law is pointless, since if you do something that nobody else can see, you can never be punished. But many, many, many laws are written without regard to whether they make any sense.
A reasonable person would not expect humans to review the millions of messages passing through the LLMs, or their own threats to ever be transmitted to a human without their authorization.
Reporting the danger is by itself a good deed. But there should be a better way of restricting firearms from the probably irresponsible lady than using inappropriate charges to punish the thoughtcrime, OR waiting for them to commit violence.
A sibling comment includes an important rider to the provision: "...in any manner in which it may be viewed by another person." If you wrote this in a google doc, it almost certainly would not qualify as a threat under this statute. Even though google docs, like LLM chats, have administrative override and you could look at their contents - you would not expect either to be "viewed by another person."
IMO I do not think this is a grey area and it's legal to tell a LLM you want to kill someone. It's certainly not a "threat" like you might send to another person, though it may end up being evidence of conspiracy or premeditation. I suspect we would be well served to, after a few years of experience, put together some laws governing when LLM chats must be made available to authorities.
It is very interesting that the LLM responses to these lines - the context around what she is saying - is not in the article. I suspect, as is the case in many instances where LLMs are involved in violent planning, that the LLM was urging this behavior on. Basically entrapment - you are encouraged by a robot to become more violent and vindictive and then when you do you are handed over to police.
I do take your point. But I would also ask you to imagine that instead of threatening to mass murder a bunch of people she was just creating text based csam. Still, all of what you said applies, but clearly she would be prosecuted over this victimless crime and the jurisprudence in fact disagrees that it would be victimless. So applying that reasoning to this act she clearly committed the crime (legalese notwithstanding). The fact that threatened mass murder doesn't trigger the same thought process is, well at least interesting to me.
Too broad. Unless you're transferring ink from a typewriter ribbon onto paper in a hut with no electricity, your words, or my words as I type this, are being grammar checked by something partly in the cloud. If I delete my words, are you saying I've transmitted them nevertheless?
There was no intent for a human to read the message. By your logic, if she wrote a threat in a diary and a burglar broke in and read it, it would be a crime on her part.
Chats with a company's computer aren't private the way a diary is. A better example would be she intended to write it in a word document and instead accidentally sent it in an email to a random person.
Yes but the law usually evaluates the application of a statute within the context of someone's mental state. This is why you are not guilty of battery when you trip and accidentally bump into someone. https://en.wikipedia.org/wiki/Mens_rea
That depends on the crime; several related crimes are only distinguished by intent. Negligence is itself a crime if it is the cause of a preventable death when the person has a reasonable obligation, such as when driving a vehicle.
I'm not really sure that this can be likened to a diary when it is called a "chat" but that's for the legal system to determine, not me sitting on my couch.
Any reasonable person presumes when they chat with Claude that it is a computer program on the other end. "Claude is AI" is explicit on the page right under the input box. The word "chat" doesn't anthropomorphize the situation.
And yes, some laws are "strict liability", I don't think this one is.
If she had intended to write it in a word document on her computer but instead accidentally wrote it into her email client and sent it to a random person, I'm not convinced she would escape getting charged then either.
It's not any different than telling an automated phone voice tree system that you plan on killing someone and then being surprised that your words were later heard by a human. She absolutely told a company's computer. She sent the message.
The law may have been intended for more direct threats to a person as a means of intimidation, but that's a separate conversation.
The Florida statute requires that it be transmitted in a manner that can be viewed by another person. If you have no idea that someone could view your communications with a chatbot, did you really intend to break that specific law? Technically, that threat was communicated to another person but not through her own intentions.
Yeah.. if you write a personal note and it's backed up by the operating system, it appears to be in violation of this law as well (since the company could theoretically read it)
People assume there won't be another human in the mix, but there is. She was judged for what she probably assumed was a private thought when it was actually not private.
Sharing them certainly can be under certain circumstances. I wouldn't be too surprised if this case gets tossed, but it's not inconceivable that prosecutors could win on the grounds that AI chats do not have a reasonable expectation of privacy and are therefor meet the requirements in the Florida statutes, or even more likely, they find some lesser charges for a plea bargain.
It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person, when in such writing or record the person makes a threat to: (a) Kill or to do bodily harm to another person; or (b) Conduct a mass shooting or an act of terrorism.
To me, that is the more interesting legal question. Does a LLM-based safety net that sends content to a human, when the original use case would not have sent it to a human, count as "may be viewed by another person". It certainly wasn't intended to be, and that isn't the norm. At the same time, because no security is perfect, we could say that any digital record, stored in any way "may be viewed by another person."
The usual thing that makes laws against criminal conspiracy pass First Amendment muster is that the words have to be combined with some concrete acts furthering the criminal conspiracy. That might just be something as otherwise innocuous as looking up the blueprints of the bank you talked about robbing but it has to be something other than just talk.
Lotsa words in that law... except they don't define what a "threat" may be.
If you enter my house I will k!ll you. <- is this a threat? noone knows. The interpretation of the word "threat" is unknown. Besides, is a conditional a statement? who knows. But sure, blah blah... "in any manner in which it may..." these words are putting me to sleep.
So a written threat only becomes a crime when someone reads it, even if you never intended for anyone to read it? Is it a crime if I make a threatening statement in a diary and someone breaks into my house and reads the diary?
I suppose since Anthropic's T&Cs allow them to have a person read your chats, that makes it violate the law. Of course, if Anthropic didn't have that in their T&Cs, it wouldn't have been illegal to write.
Anthropic commits literal felonies by stealing millions of books and violating Copyright like it doesn't exist: no charge.
One unfortunate woman who happened to write the wrong thing in the wrong place is now having her life turned upside-down for perceived thought-crime.
To Anthropic, and all employees working there, your company's product and the result of your work is cruelty. You are enabling it and pushing it down everyone's throat. You can never again claim that you are the "ethical" AI company, for no such thing exists.
Over in Europe they want to read all ofd our private messages, yet these chatbots, pretending to be our friends, will snitch on us just for our thoughts.
I once had a copy of 1984 in my checked baggage returning home to the USA and when I was unpacking the bag at home, the book had a notice inside the book that my bags had been inspected by TSA...
I reckon they were just checking for money or hidden compartments. Sending books abroad packed with money is extremely common from the US, though not sure how frequently people do that with onboard luggage.
Only tangential but when I was an undergraduate studying philosophy I had Bertrand Russell's Why I Am Not A Christian in my carry-on, and the TSA saw that and had a field day.
Detroit airport, 2010. I was chosen for secondary inspection and got some snide remarks of "we've been seeing too much of this lately" in reference to the recently failed underwear bombing attempt on a flight from Amsterdam to Detroit.
I can only assume you meant they had a field day celebrating how much of Russell's philosophy matched the concerns about Christianity expressed by Jefferson, Paine, etc?
They wouldn't see it until they opened it, so clearly they didn't open it because of the book. And if they need to open your suitcase (not because of the book), they have a reason for that.
I was chosen "randomly" for secondary inspection. This was after a failed bombing plot and the TSA implied a lack of Christianity was the proximate cause of the attempted attack.
After. But then I continued to get selected for additional screening on the rest of the legs of that trip and the return. So either I was going to get searched every time I got on a plane already, or they flagged me for additional scrutiny after the initial search.
In Europe they should provide the citizens with the service of their messages not reaching US servers. So basically that there is only one party reading along with them, not half the world.
For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
What would happen if it had scanned a file it didn’t have permission to look at and found this threat?
I honestly don’t know how I feel about this. On the one hand if you’re using claude as a diary you have no expectation of privacy and she was talking about committing a very serious crime.
> For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
For the sake of even more argument, imagine if she was writing her thoughts with a pencil, on a good old fashioned paper diary, and she had a phone nearby and the phone took a picture of her diary, OCR'd the words, and reported it to the police?
I guess we probably want our tech to work exactly like this.
It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
___
Of course, we also want purely private tech, but that needs a certain level of merit and sanity filter.
>I guess we probably want our tech to work exactly like this.
Do we, though? What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products. I don't think it's the tech companies' job to surveil the population and prevent crimes. That said, I'm not necessarily against Anthropic or other companies reporting suspicious activity if their existing systems are detecting it. I'm just not sure we want every product to be forced into that data model.
Your follow-up about purely private tech seems to contradict your first statement. We can either have privacy or surveillance, not both.
>What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products.
Why wait for a company to build it? Get your own local hardware like I did and have those guarantees because YOU set it up.
I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
> Your follow-up about purely private tech seems to contradict your first statement.
That is correct! And exactly my point.
We want both, but, on paper, that is impossible.
But in reality, we make it sorta mostly happen anyway, through making the easy defaults not private, and the private stuff not easy.
This is not ideal, because [various reasons I do not need to tell you], but it has proven to be the best we can do to mostly achieve both goals.
Kinda like how capitalism isn't great but just the least worst option we've found so far.
___
The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff.
Hence the quadruple-speak and contradictions to kinda sorta somehow have a somewhat functioning reality.
Oh don't worry, I'm certainly already on various lists, but the observations also will have resulted in the assessment that I am stable and no threat :)
I think we can have both and it might even be smart.
A lot of people causing issues are people that can't make sense of many things (like many terrorists). They get a fixed idea and they end up doing something bad. You would catch those with some (basic) surveillance.
A lot of normal people (not wanting to cause issues) might benefit from some privacy, if they understand what are the trade-offs (like government overreach). They can then use a slightly more complex tech.
We would still remain with the couple intelligent but sociopaths (think Unabomber style), but I think no solution can fix all cases.
That sounds adjacent to "I have nothing to hide". Everyone says that until they realize someone can change the rules. Before 2022, women didn't have anything to hide from their period-tracker app, now some have to worry about being charged with crimes.
Back to LLM chats: A system that can declare her "unstable" is also one that can permaban you from all air-travel because you "privately" said unflattering things about Dear Leader.
> That sounds adjacent to "I have nothing to hide"
I urge those people to share their full information from banks, companies (salaries, agreements, contracts), full health information and share publically all the texts they ever wrote (incl. as teenagers) and all the photos they have taken. And give away all the passwords to all of the services so people could check that they are truly clean. Just to be sure, just for everyone's security, right?..
> It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
The third highest voted comment wants surveillance. What is next? If you write "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported.
In a old happy little idealised village, it became known quickly, who started to behave oddly and timely intervention could happen. In the modern anonymous mass cities?
No one (wants to) notice the madmen scheming in his isolated flat, surrounded by strangers. Until he explodes.
Unfortunately I also don't trust our government agencies with the surveillance - because they ain't transparent either and the self surveillance seems broken.
" "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported."
And unfortunately there are lots of real threats being made under the disguise of humor. And much harder to separate im text. So maybe don't talk of murdering people in general, AI surveillance or not?
the honest actual opinion of the average consumer is probably that they don't want their chats to be monitored but they want everyone else's chats to be monitored. There was a lot of fury when a mass shooter recently used AI to help plan his assault. And of course there are all the people talking to chatbots about suicidal thoughts and intent. When they ultimately follow through, the providers are blamed for not alerting anyone.
No. This sentiment is why Snowden happened. We want privacy. Privacy isnt free just like freedom (whatever form it is) isnt free.
There are trade offs. ISP effectively is like driving on a highway, everyone can see where you are going but not what is inside the car. Id like these AI chats to be the same but they are not.
Devil's advocate: if I tell my therapist or my lawyer that I'm going to murder someone, they're obligated to report it. If I use my AI as a therapist or lawyer, why should the company that provides that service not be held to the same standard?
LLMs aren't email or file storage. AI labs aren't just shuttling bytes around, they're interpreting those bytes and taking action based on them. These models _already_ react viscerally in response to users saying disturbing things: the only practical difference is the ability (or obligation) for the model to escalate that concern. I'm not sure the ethics we hold AI companies to should be different than if a human being was typing out the responses.
Privacy is obviously hugely important, but this isn't the government surveiling every message. It's companies having an obligation to flag real, credible threats according to the law, which is a very different problem space.
>I guess we probably want our tech to work exactly like this.
Maybe you do; I want my tech to always include secure, encrypted communications. Don't include me in your destruction of privacy with your silly bandwagon!
uh no, 'we' absolutely do not want our tech to work exactly like this. Why would you assume that people default to 'search my anal cavity please' and not 'no, stay the fuck away from me with that glove'
----
jfc, why is this website full on psychopaths
"The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff."
other people are not pretending everyone is equal. your power levels are showing, it isn't subtle.
True, most normal people have little concern for any such humanist goals and ideas.
Normal people live prejudice. It's (erm, claude-speak) load-bearing for them, given just how complex reality is and given just how well it reduces that complexity.
“But there can be no valid knowledge about the future. As soon as precognitive information is obtained, it cancels itself out. The assertion that this man will commit a future crime is paradoxical. The very act of possessing this data renders it spurious. In every case, without exception, the report of the three police precogs has invalidated their own data. If no arrests had been made, there would still have been no crimes committed.”
Philip K. Dick, Minority Report, 1955
While I accept that this sort of thing is well with in the ToS and regular course of business of any major online platform, it hits different coming from an AI company for some reason.
Is this an invasion of their privacy (reporting to police)? Yes, but possibly warranted?
Should a social worker have contacted them rather than the police? Probably, if for no other reason than to ask if they were serious about harming someone.
Difficult questions, I'm still undecided on whether it's OK to always ignore someone's rants, even if it may be (or they think it may be) a private diary.
Actually charging them with a felony seems pretty quick to accuse. (Maybe I missed a hint about how long the investigation took before the felongy charge?)
It is my very european belief that the problem here is not that the woman was reported, but that what likely is a mental episode was made public in a way that reduces the chances of recovery.
I guess all the "private model" people are right. Don't want to end up in jail (or even charged with something) for asking a crazy hypothetical question or something.
I have a hunch she wins the case, on the basis that an LLM isn't a person and an (assumed) private expression of anger to a machine assistant doesn't meet the statutory threashold which requires that a threat be communicated to some other person.
Of course, that could change if there's evidence that she took action in pursuit of a goal, like buying ammunition or repeatedly driving around the entrance to her alleged target.
The law seems overly broad with "may be viewed by another person". Most of these laws have the intent of not causing public panic with regards to posts that others may see, not stuff that one assumed was private. This is further supported by the definition of threat, which is generally defined as requiring malicious intent.
I would think a good lawyer could get this charge dropped. I would like to see what judge approved the warrant and how they felt the elements were met.
Should the public have an expectation of total privacy for their chats? It seems responsible for a chat provider to report things like this.
If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime? Basically everyone agrees that crime-committing advice is inappropriate…but if it is not ok to get advice, that means there must be a portal for law enforcement to step in when that may have happened. Then the question becomes what is the line for when to report? In other words, the issue needs to be adjudicated.
But we don’t want OpenAI/Claude to have some $20/hour reviewer making decisions that are this high stakes…we need the courts to do the judicial work because they (1) have a public charter, (2) have meaningful expertise and specialization at interpreting the law and (3) we can hold them accountable.
It doesn't really matter if they "should or shouldn't" have an expectation of privacy IMO, they already do have that expectation.
> If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime?
Yes it should be, but it should be illegal for a company providing chat service to respond with anything other than a refusal when doing so.
That detection and refusal should be a private closed loop though, anonymizing any data that will be passed into a training pipeline, or ads targeting. This requirement for closed loop private chats should be mandated by law sooner than later. Otherwise we're getting into very tricky territory where the temptation of alerting on things like pre-crime grows too close.
> The communication must be made in a manner in which another person may view it
IANAL but it will be interesting to see how the legal system decides if this counts as "another person may view it" or not. What has happened in similar ish cases where someone writes a threat like that that they thing is private to them but actually ends up in the hands of Someone at some tech company that reports it?
Not weighing in on the privacy issue but using Ollama you can run a smaller agent like Qwen 3.6 35b a3b on a sufficiently potent laptop. Pair it with something like Hermes for a nice interface and you have more than you might need for diary like usage.
Could this be considered free speech? Most of the limitations I know of require you to say it to or in front of other people as to create a panic or make someone feel threatened. Is it a crime to say to yourself "Im going to blow up New York." or about threats of self-harm? How about "Im going to steal the Mona Lisa!"?
To me this is just straight up thought crime, they just don't have a way to directly read your thoughts yet. But they will spy on you and try and catch you out for it.
This is a classic case of "Damned if they did, Damned if they didn't". Given this has come out, and Anthropic is considered to be the beacon of transparency, it would be appropriate for them to share what their thresholds are. The article also reports Open AI not considering the threat credible enough to alert, so there's clearly some thresholding that people need to be aware of.
This kind of thing will get worse with humanoid robots because they have cameras and microphones. Will they be programmed to tell on you if you break any kind of rule in front of them because their owners are terrified of being sued?
this feels very thought-crimey to me, but I think I'd have to actually see that chats to really decide. Like is she making plans/asking for advice? is she just talking about her feelings exactly as if it's a diary?
This news article [1] had a quote supposedly from the chat session:
> The arrest report states that on Sept. 26 at approximately 5:10 a.m., Heller reportedly wrote, "I'm going to shoot up the sheriff's right the [expletive] now." The following day, at approximately 1:07 a.m., she was accused of posting, "This is 100% last chance I'm done. I got a new [expletive] gun today. [Expletive] you."
She is not being charged with planning, just making threats in a place a person could read - the person being employees of the company.
Basically, under this interpretation, any personal note you store in the servers of a company could qualify, even if you didn't ever imagine someone would read and as such you couldn't have thought about it as a threat
LLM is not a person but T&C probably states that a human moderator may view any of it. Her lawyer could probably argue a moderator filtering usage isn't the intent of the law, it was more about publishing / sending message for other humans and it was never clear to her that a human was reviewing her private diary. Shouldn't LLM disclose that at some point when people are feeding really personal stuff?
I expect that anything I type / dictate / post / purchase on the web be it a chat conversation with an AI, a post on social media, a dm, a blog post, a blockchain reference, or an assett in a bucket, will be manually reviewed and forwarded to the authorities. It is their duty to do so, and I am glad that they do. If this woman did harm someone we would be hearing "why didn't Anthropic do something about this."
Aren't high trust societies staying that way by kicking out the incorrigible violators? Pondering violence might not be treated lightly even in a high-trust society.
Florida statute 836.10 puts the bar at the "sending, posting, or transmission of, a [...] record, in any manner in which it may be viewed by another person"
I think the defendant could successfully defend themselves by claiming they did not know (or intend!) the message could be viewed by another person, as they were plainly using it as a private diary.
I can’t believe people are using these proprietary models/subscriptions as personal assistants, sending their most personal context, thoughts, documents and information to the providers. Some even give them full disk access.
After using their services for long enough, Anthropic/OpenAI essentially have your entire life mapped out to an insane level of detail, likely including detailed contents of your computer/phone.
People used to say that Facebook knew more about you than your closest friends/relatives. And compared to that, this is just on a completely different level.
Absolutely insane.
This is practically entrapment. All the AI labs sure don’t shy from plastering annoying disclaimers everywhere saying their tool can make mistakes. Shame on them for not also reminding everyone continually that anything you submit can and will be used against you. Of course they can’t afford to have a Flock-style user revolt.
duck.ai et al? my chatbot claims its architecture provides some level of privacy yet it is honest enough to point out none of these systems are actually audited and "not stored" and "anonymous" does not mean "not monitored".
I haven't seen anybody claiming this for consumer/free accounts anywhere.
You get privacy if you're a big corporation that needs to make sure OpenAI/Google/Anthropic can't read your trade secrets etc.
But those contractual privacy protections have been in place for a long time. It doesn't have anything to do with AI, it's been the same with Office365, Google Docs, etc.
If it's going to be this binary. Then there has to be a shift in the way this is handled. It shouldn't be "You wrote these terrible things to a computer, you're now accused of doing them" to maybe "We where notified you wrote these terrible things, and the a psychiatric evaluation is being mandated by the state in which has a law about this sort of thing"
Obviously, as others have pointed out if they don't act and she does the crime it raises the damned if you do damned if you don't. But I've also had the AI's go haywire saying I'm doing all sorts of nefarious things when literally doing math proofs.
So no one size fits all. But going the extreme first is probably not ideal.
They still read every single message you send and train on them.
I’ve had them email me before because I was testing it as a filter for abusive messages and they detected some no-no and wrongthink in those test messages.
I looked into ZDR, it's basically a vapid claim with little to no due diligence or auditing. I expect most providers to cave with only a minimal amount of legal pressure.
After people getting pilloried for social media posts from twenty years ago, I really hope (sensible) people will have the wherewithal to think twice about what they hand over to their chatbots.
Claude did this, its also important to remember that a human counselor or psychologist will also do the same thing.
For example, if you said to your counselor or psychologist that John Smith is going to rob the bank next Thursday, they will report that to prevent the imminent risk of serious harm to others.
And remember that anything but local AI you personally control is not anonymous. Not even AI you pool with your friends.
Tech companies have always done this. Don’t misread that as blanket support for it, but consider Anthropic’s position here. Do they want a headline to come out that says “woman used Claude to plan murder” or “Anthropic reported potentially dangerous person to the police”? AI being dangerous is already a hot topic, and this seems like a sensible move to me. If you want a guarantee of privacy, you’ll need to run your own models locally.
If you want technology that won't report you to the police for credible threats at they can easily detect perhaps create your own. I hear pen and paper work great for this.
Pre-crime prevention and detention will be the end goal. The economic devastation caused by AI will make large swaths of people get angry. Better use that massive amounts of data being hovered up and the mass of compute being built up to stop those angry people lomg before they even get a chance to properly organize.
Does anyone else remember when the tech-savvy crowd was wary of software "phoning home" or spying on users, or listening, or whether Gmail scanned and read your emails, and whatnot?
And here we all are, happily typing our stuff into these spy chatbots. "AI" happily made our fears go away. Hopefully we're not planning anything criminal like this woman, but still.
Snitching on the people - good mass surveillance company.
On the other hand, people need to learn to not trust these
companies. It reminds me of others being surprised when
a self-driving car reported a gun in the car. I mean,
do people not think? Besides, of course, it's already messed
up to want to have a gun. And it is constantly one country
that has such issues, more so than many other countries.
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Thought crimes are real when you're sharing your thoughts with Claude
I wonder if "criminal intent" may be missing here given that most people probably operate under the assumption that Anthropic are not reading their messages.
I think people have a binary understanding on this. Someone is either reading their messages or not. While the reality is that all the messages are read by a machine (not unlike GMail and Outlook) and anything suspicious gets flagged up so a human can read it. This obscure the concept of "reading" as most laypeople understand it.
Summary: don't type in Claude anything you wouldn't like a human to read.
It tells you exactly what it does with your information if you ask it, including this exact scenario, and has for at least four months now (when I asked).
Yes but you are someone (i assume because you are on HN) that at least understands this point. There are A LOT of people that use it as their confidant, expecting it to be private. There is a massive education issue going on as its not in the interest of these Corpos to make you fear sharing all your details with them. Because, then you wont get Dot or whatever Anthropic comes up with and share all your personal info with it.
You're not wrong, and it may come down to this in court, but there's a difference between what people think happens, or the reasonable expectations, and what actually happens, and I think it's important to recognise that difference and why it comes about.
I don't think someone is an idiot for thinking that the information they type into their private Claude account is private. I also don't think people are idiots for thinking their phone is listening to them and giving them targeted advertising based on that. Both are reasonable deductions from their lived experiences. Both are wrong.
I think it's a reasonable point to make. Writing things down is a part of "thought" for many, including those who keep diaries/journals. If you write in a private journal you do so with the expectation that is not shared, and that wouldn't seem to break this law (with my naive reading).
I mean, at this point it's pretty well known that all tech services will be hacked by fable, anthropic themselves promised it, so writing your journal in google docs or claude or a txt document on your laptop or such is the same as releasing it publicly yeah?
If it were written on paper, and only in a room with no phones or cameras so fable couldn't hack it, then I think you wouldn't be sharing it.
I think the zero-risk approach common nowadays is insanely corrosive to democracy and freedom. If a million people fantasise about shooting the sheriff, and one ever goes on to do it, I don't believe avoiding it warrants creating an apparatus of mass surveillance. After all, if people were really serious about zero murder, the only practical solution would be to lock up everyone. Some (most?) tradeoffs have exponential costs at the limit and we/lawmakers should recognise that.
> If a million people fantasise about shooting the sheriff
I think it’s fair to pre-identify folks who fantasise about shooting anyone. It’s a small fraction of the population that looks into logistics versus making offhand comments.
So now when everyone has their watch recording conversations, or perv glasses, or Alexas, or Jony Ives new personal AI gadget … recording everything we say and scanning it at scale … is it ok to be a doomer yet?
Why not report it? Seems reasonable. At the end of the day none of these services can guard against misuse. I personally optimized bomb creations and other stuff with AI just to see if it is possible. Maybe the way to secure the world is to not have corrupt old farts running every country. As the old saying goes. Information wants to be free. A kinder and more connected world is the only way to keep everyone safe.
This reminds me, for his short story "Dolan's Cadillac", Stephen King claims -- perhaps exaggerating a bit -- he did the real research on how to murder a person as described in the story. How to dig the hole, what measures, what the falling rate of a car that size would be, etc, etc. He claims he fuzzied the details a bit because he didn't want to teach readers how to actually do it (again, maybe he's exaggerating a bit to promote his story).
Imagine if, instead of a friend, he had asked an AI chatbot. Would he have been reported to the police?
Aren't our private thoughts just that, private?
I get that a cloud-hosted AI chatbot is, to tech-savvy people, immediately "not private", but a lot of people don't understand this. What if Stephen King was talking to his friend about planning the murder, and his phone was listening? What if he was typing it up in MS Word and the program decided to phone home and report him to the police?
> Another incident has taken place that illustrates the need to be careful what you tell AI
Nope! Another incident has taken place that illustrates how committed Anthropic and Anthropic AI is to the law and public safety, unlike all that open model riff-raff
They were waiting with bated breaths for something like this to jump on, and make an example of.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone [...] The communication must be made in a manner in which another person may view it.
Which it clearly wasn't, right? I mean, okay, in this case, the message did get reviewed by another person, but that's obviously an exceptional circumstance.
If I write something down on a piece of paper, and someone else goes through my garbage and finds it, is my note "communication made in a manner in which another person may view it"? It was clearly intended to be a private note!
It is worth mentioning that state law may not abridge people's rights which are protected under the Constitution. It is perfectly legal under US law to make any number of violent statements as long as they do not rise to the level of true threats.
https://icap.law.georgetown.edu/wp-content/uploads/2026/02/T...
I am not a lawyer, but I find it hard to believe this statement meets that bar.
Given that there's not really an objective measure of the trueness of a threat, this seems like the sort of thing that a court decides the merit of after charges, though.
Somewhat relatedly, I used to do tech support for a very large e-discovery vendor. The number of companies that configure their systems to archive (save, index and make discoverable) the contents of the /drafts folder in email clients is astounding. Surveillance of composition, rather than surveillance of communication, seems wrong.
The reason for doing that is to get all of the email in a system, as required by whatever rules apply.
Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Back when I was in IT, circa 2010, I learned that several of the users of our systems routinely relied on the "deleted items" folder of Microsoft Outlook as a filing system, with multiple gigabytes important files stored there. 8(
Our customs are routinely ignored by everyone else.
[1] https://en.wikipedia.org/wiki/Foldering
> Today I learned about "foldering"[1], which uses the drafts folder of email systems as a "dead drop" between multiple participants. It goes back to at least 2005.
Former head of the CIA, David Petreaus, was using the drafts folder of a shared Gmail account to communicate with his mistress.
The reason so many people used the deleted items folder as a filing system is exchange didn't set a quota on that folder and by default would only get emptied by a manual emptying. So back in the days when a user could have a 100mb mailbox quota 'unlimited' space could start looking very attractive. Especially if that quota was 5 years old because the server hadn't grown with the times, since what business wants to spend money on a 'cost center' that won't improve profits.
Then these people didn't change their ways when they were given adequate space.
It was used by the subject of this book as well, which I think predates 2005.
The Spy's Son: The True Story of the Highest-Ranking CIA Officer Ever Convicted of Espionage and the Son He Trained to Spy for Russia
https://www.amazon.com/dp/0802125190
Both OpenAI and Anthropic state in their terms and policies that they can access, retain, and review user inputs and conversations (yes by either an LLM or a human). So I guess at this point it is now up to a judge (or a jury) to determine if the person is dangerous.
The logical next step would be, to create psychological profiles of each user and relate those to authorities.
Selling analogous profiles of companies using their services to the highest bidder would be another idea.
Whether intentionally or not, they market their product as suitable for all kinds of things that it clearly isn't when input is being used that way.
Hopefully we at least get some cool Tom Cruise action scenes when we create our own shitty version of Minority Report
If the threat was to a private citizen and not the sheriffs office, I don’t believe charges would have ever been brought
> If I write something down on a piece of paper
At some point I think these sorts of analogies break down, as the setup becomes too foreign to what we're more concretely used to.
In this case: there is no pen or paper which can store what you write on a replicated set of servers across the world, with an accompanying ToS telling you how that will be treated/used.
Although I have little sympathy for this women (both her intent and stupidity), I do agree this is a dangerous thing.
>another person may view it
LLMs seem to be moving toward personhood.
And after all the LLM learns from user posts too, and if everybody starts posting their darkest desires, fantasies, plans it may skew the "alignment" to say the least. Lets hope that the AGI level doesn't necessarily come with cheating, lying, religious fervor, power lust or whatever other sideeffects have been observed in human intelligence.
> "may view it"
If we assume that "may" here means "could somehow" and not "is authorized to", the legal action after revelation seems correct as written (ignoring whether the person viewing it has any relation to the person being threatened) unless the law gets struck down as unconstitutional. The question is only whether Anthropic should or should not report it.
> "If I write ... It was clearly intended to be a private note!"
The law as written doesn't appear to distinguish about intent of privacy. Also, if you're in the habit of writing notes to yourself, I guess maybe don't write down the one that says you're going to shoot up the sheriff's office.
Ai labs are 80% surveillance machines. I will never understand how people trust these services with anything personal, emotional, medical, financial, whatever.
I don't know why people are even discussing this case like these companies will do anything on behalf of a consumer or think ethically at all.
Bots mass read and file and report all prompts that get categorized a certain way. They store everything else regardless. This will never change. It will only be exploited more and more. That's how this type of technology is deployed and progresses. Look at any other parallel. Like cameras or microphones.
Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
Once you go in on infrastructure you have become a small data center. You will need to maintain it, continuously finance it, and secure it. And what if someone wants to back out 6 months later? Now you’re signing contracts with the implication that you’re willing to take a friend to court.
You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps.
It’s all overkill for most people anyways IMO. This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
> This lady was just using it as a personal journal. Basically a glorified ELIZA. That kind of thing can be done with really small models locally these days.
And she's facing felony charges. Yeah, folks on hacker news can talk about technical solutions all day long, but that's not the fundamental problem. The fundamental problem is that we've allowed our digital infrastructure that includes everything from banks to schools to doctors to become adversarial to us. In some cases predatory.
It's effectively untrustworthy, like being charged a felony for walking across a bridge incorrectly. That's not a "I'll host a small model in my garage" problem. It's a we need the government to do its job problem.
Well this is why I think small local models are the opportunity of a generation. Because it’s a bit transgressive these days to say you don’t need a big centralized service that spies on you and requires a subscription. It’s not the absolute best, but it’s yours.
I agree in general that we technical solutions to social/political problems are suboptimal. But technical solutions don’t require the political system to work, which is a very attractive property at a time when politics seems broken.
There is a counterpoint that retreating into technical solutions only cedes more ground and makes things worse, and for that I don’t have a good reply other than the fact that life is short.
But ultimately it’s not a technology problem it’s a society problem
The people who are accountable and responsible are delegating their authorities to policy and now the policies are just implemented by machine systems and there’s decreasing human intermediation.
The humans that are still inside the system increasingly have less control such that it’s it’s increasingly difficult to find anybody who is the actual customer service type representative who has the authority to make a meaningful impact.
We need actual privacy laws in the united states. Between this, other cloud providers, the advertising tracking infrastructure, phones that constantly ping your location, flock and it's ilk, etc. It's over, no longer can you live semi regularly and still maintain your privacy by maybe living out in the sticks a bit. This is the panopticon.
I mean, this is also just kind of how mandatory reporting works.
If you tell a teacher, a therapist or a priest that you are going to kill somebody, in many states they do actually have to report that and can be held liable for the resulting crime if they do not.
Every time we have a mass shooting in the US it inevitably comes up that they wrote it down or told somebody and then the next logical question becomes “how could we have stopped a mass casualty event?”
>the next logical question becomes “how could we have stopped a mass casualty event?”
'No Way to Prevent This,' Says Only Nation Where This Regularly Happens
> You can get around this by hosting in a 3rd party data center, but now you have the same trust issue again, but with more steps
No this is not true — colos are very hands off.
You don't need to operate a mini AWS to self-host, and I don't know why people on HN want to make it sound like self-hosting if you don't have nation state security
> colos are very hands off
That’s a pretty broad statement? Maybe a more correct one would be that some are hands off enough that some people might find them worth it.
And that really only addresses one problem. The other one is the coordination problem. You go in on a $30k GPU with 3 other people. So each person puts down 7.5k. But then one person wants to pull out, and so every remaining person needs to put in an extra $2.5k on top, and they’re questioning whether $10k for a chatbot is really worth it.
This level of hardware and cost just isn't necessary. You don't need to team up with your friends to go all in on a hardware purchase, which I've never heard of anyone doing anyway. Go spend some time over at r/LocalLLaMa, you'll see.
I've never heard of a colo looking into their customers servers. They're not in the business of monitoring their clients.
I don't understand the coordination problem – if you want to pull out you have to find someone else to buy your share?
I've seen something like it once, but it was really just the FBI coming in a seizing a rack.
> Once you go in on infrastructure you have become a small data center.
A PC at home with a 3090 in it is more than enough if you want to talk to a chatbot about your day.
This is also where TEE's with attestation come in - https://redpill.ai/ for example (god I hate their name, but I am a fan of the product)
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
The 4090 I bought in '23 is selling for ~3x what I paid for it now in late '26. There's a huge squeeze going on with computer hardware using relatively cheap borrowed money. Relatively cheap money is flowing into a (artificially?) limited GPU/memory asset class. And this asset class is growing much faster in value than the interest on that money. So if you borrowed heavily to buy GPUs in the prior 3 years, you're likely coming out well ahead even if you are paying 8 to 10% on the money.
That dynamic will fuel further borrowing until we get to some kind of equilibrium, or some kind of washout occurs where interest rates spike higher and/or the value of GPUs and GPU services start leveling off or even declining. We might get a test soon as the FED has started hiking.
This is a great model on my Framework Strix Halo box as well. (also now about 2x what I paid for it in summer of '25)
Same here. I was planning to get a second, just got this one ten seconds before they became unobtanium. Here’s hoping Qwen 4 Flash is the same size!
Gotta find some rich friends in order to buy and host a H200 or two.
You really don't.
I’m excited for the refurb market in 7 years.
in 7 years you'll be soylent green as you're used to fuel the answer to 42
The circle of life in motion!
A middle road is to use open-weight hosting providers, maybe non-US ones if you’re in the US.
Yes otherwise thoughts are crimes, effectively.
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
I'm coming to saulpw's house tomorrow at noon with a gun, and I'm gonna make them pay.
If a policeman notices the sentence above on my phone screen during a routine traffic stop, the response you want him to take is... nothing?
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
>the legality would depend on whether your notes are backed up to icloud or not
I agree, and it's nuts.
This feels like less of an issue with anthropic per say as it is a broad reading/misuse of the law's original intent.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
Yup. HNers (and so many normies) believe that there are well defined rules they can follow and be safe. Spoiler, there is one law in the real world and that is might makes right, and you have no might, so therefore you will never be right. China for its many flaws at least is honest about this.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
Correct. Nevermind how ridiculously contrived the scenario you've just concocted is.
https://www.youtube.com/watch?v=14WE3A0PwVs
FYI this links to "Key & Peele - Rap Album Confessions"
its not contrived just because you recognize how silly the argument its arguing against is
> If a policeman notices the sentence above on my phone screen during a routine traffic stop, the response you want him to take is... nothing?
Anybody showing a cop their hacker news comments at a traffic stop should be arrested, for harassing the police
You joke, but part of the issue here is we're supposed to have laws that keep cops from reading over everything we type and using it to turn us all into crime suspects. There's a whole constitutional amendment about that, but our personal "papers and effects" that should be protected against unreasonable searches are increasingly really "owned" by others who are all too happy to snitch.
Part of me says that the solution is stop entering any personal data into any device and service you don't own, but I'm not sure if that's really what we want considering that there are zero private cell phones. Even desktops and laptops aren't 100% owned by you these days. The only thing you can really do is keep them offline 100% of the time so they can't spy on you, but that seems like a lot to demand.
Do you really think reading a snippet like that completely out of context should qualify as probable cause?
Yes!
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
How can you prove that it's not just me writing a dark and morbid story?
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
The thing is plotting a crime is not illegal. Authors do it all the time. Usually against fictional characters, but some authors use a real world backdrop. And look over on StackOverflow. You can get some really weird ones that only make sense when you look into the details. When they manage to show up in the hot question list those details are missing. "How to kill a cat" comes to mind. (Note that you won't find it anymore--the question is AFIAK still there, but the title has been edited several times. The cat in question is the Unix command, not the feline. What do you do when you inadvertently tell the system to display the contents of a large binary file?) And, AFIAK still there, "How to kill Indiana Jones". (There are always the ancient mechanisms that somehow still work perfectly, springs and all. How to make something that actually would work?) And one I've heard of but not seen: "How to kill my wife", from some game I didn't recognize.
I’m sure the average UNIX admin has a trove of misinterpretable web searches.
fork a child and kill it - Google Search google.com/search?clie...067j0203j0i20i263j0i22
kill child and fork parent - Google S... google.com/search?clie...o...1...5.0j0171j35i39j
kill parent with fork - Google Search google.com/search?clie.....1... ..0171|35139|33116
kill parent without killing child - Goo... google.com/search?clie...4589.0j32j1.0....1......
kill child without killing grandchild -... google.com/search?clie...5.0j37. ....о...1...5..0j3
kill all children - Google Search google.com/search?q=ki...&hl=en-bg&client=safari
kill child with fork - Google Search cooale.com/search2o=ki &hl=en-ha& client=safari
You just said the words right here in this public vbenue, not even in private.
This argument holds no water at all.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
https://en.wikipedia.org/wiki/Brandenburg_v._Ohio
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
action in the real world, i.e., not just words.
Lots of people today think thoughts are crimes
Or worse, misaligned with the Terms of Service.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
This is not even an option in the UK. Communication felonies (dangerous speech, threatening behaviour) only demand potential audience.
Qwen3.8-27B is all you need.
Oh boy, I wish I had enough friends to afford H200 x 2
Would be better if your friend is lawyer and take $1/yr(or higher) for the service so that attorney-client privilege can be applied.
edit: ah, future crime cannot be protected.
Surely a recurring automatically renewable $1 monthly/weekly/daily contract would solve the problem? Lawyer on a retainer can't really babble about your crimes and is required to protect the confidential information.
That's absolutely not how that works lol. There's a crime/fraud exception to the attorney-client privilege.
Lawyers have a duty of care to the court and they will absolutely drop your ass if you try to make them keep quiet about your crimes.
(This is not to be confused with them representing you in defense. You can tell them about your crimes if the government is trying to get you for said crimes.)
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
It’s also an option to just not use chatbots
Or you could just use notepad.
That's exactly what I do. I'm fortunate enough to use the latest preview version, which is known to us beta testers as Notepad With Microsoft Copilot™ and only uses up about 2% of my phone data for telemetry per day (Preview 1.0.912 actually used over 14% daily lol). Plus they got working set size down to 11 gigs and disk usage down to 36.3 gigs.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
You don't need an H200 for that kind of use case.
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
People say this all the time and it always makes me wonder what, exactly, were you doing?
I had a list of controllers that we forgot the usernames and passwords to. We knew it was some combination from about a half dozen of each, just not exactly what was what. A few hundred of them so a tedious task for a human to go through and validate them all.
Got flagged for attempting to have Codex write a quick script to basically dictionary attack my own infrastructure with a tiny dictionary file.
A bit silly, but it doesn’t take a whole lot.
Lead time is like a year tho no?
I agree, but someone will say fake child porn.
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
Fake CP is a victimless crime. It is debatable if victimless "crimes" (like being gay in your own home, which is illegal in way too many countries; or eutanasia; or eating pineapple pizza) should be punished at all.
I understand moral panic, disgust, etc, but rationally speaking.
I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.
However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.
Not just bad headlines; OpenAI is actively being sued [1] for this:
[1] https://cdn.arstechnica.net/wp-content/uploads/2026/09/Briti...In that case was the shooter using the LLM to plan the attack?
That could be a meaningful difference
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.
> I'm more confident that local apps on my Linux desktop are secure
Why should you be? Linux gives us software we can reasonably say we own, but not hardware. Everything you type into your local linux apps can be being monitored by your processor and whatever is sitting in the CSME/PSP subsystem which you don't have permission to access, but which is always running even when your computer is off. We need fully open, documented, auditable hardware if allowing any third party access to our devices means our private documents will make us all into criminal suspects.
People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.
Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.
If we completely give up on assuming privacy, we lose even more privacy, because so much is based on the expectations of a "reasonable person".
Our devices, even Anthropic in this case, set an expectation that doesn't match reality. The software and devices we use are marketed and sold to us to be used for our private documents, photos, and conversations. I can't blame people for using them for their intended purpose. They shouldn't assume that someone is carefully watching everything they do and judging their every word and action looking for evidence of crimes. If we want people to think of the devices and services they bought and use as belonging to someone else and without any expectation for privacy we should really start with forcing companies to change how they market their products and services to reflect that reality.
For technical people, this is incredibly old news.
For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.
One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.
"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.
It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.
You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.
Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.
I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.
And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.
> Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway.
...but the last sentence if kind of important, right?
We only know about the criminals who got caught. Which probably means they did something dumb.
It's conceivable to me that there could be many—maybe even a majority—of criminals we don't know about because no one ever caught them. Maybe they committed fraud once, decided not to push their luck, and lived the rest of their life quietly. Maybe they killed someone and made it look like an accident, and no one ever suspected anything. How would we know?
This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.
Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.
I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.
Both things can be true.
They want stable rules they can follow, which will limit their liability as long as they stay within them.
They also would like those rules to be as restrictive as possible towards their competitors.
As restrictive as possible towards their competitors, while being as permissive as possible for them. Getting control of the regulation making process means they can make sure they don't end up too constrained anywhere that would hurt their bottom line.
They've spent close to $2T so far. At that scale the legal issues they've had are just the cost of doing business.
I'm quite sure that if there wasn't the existential threat of a lack of a moat, they would not be pushing for regulations at all.
The legal issues can sink their entire business model, not quite the cost of doing business.
A lot of things could sink their business model including opening the can of worms of the wrong regulations.
That's the conspiracy theory, but there's no good evidence for it, and it doesn't really make sense in the long-run.
It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.
You're assuming the open models will follow regulations though.
No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.
The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.
No good evidence other than their openly stated desire for that regulation? Dario and Sam have both made open pleas for it, repeatedly.
Liability is enough of a reason
And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.
As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.
How much do you love Big Brother?
Wasn't this because the photo was made "public" as in, shared through Google Photos?
If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.
Most people don't realize that it is 99% like posting it on Facebook.
It isn't like posting it on Facebook. It is like emailing it, because most people don't have the link.
I can't read past what the paywall allows, but here's the nyt article: https://www.nytimes.com/2022/08/21/technology/google-surveil...
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.
Or something like leaking every non-corporate ChatGPT user's chat logs (including temporary chats) to the NY Times.
Or making up stuff
Don't know why this got downvoted? This is a problem with epistemics.
It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.
Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).
> welcome to negotiate this in enterprise contracts
I really hope you mean that in an insulting way to the industry and current legal situation, not as an actual solution.
Where's the insult? Lots of customers have such agreements.
That anyone smaller than an enterprise gets screwed. It's like responding to a problem with "let your butler handle it".
Individuals can sign up for enterprise agreements. An enterprise can consist of one person. Or run a local model.
It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
> Individuals can sign up for enterprise agreements. An enterprise can consist of one person.
Custom negotiation is a big difficulty for individuals and the pricing will probably have fixed overhead that becomes pretty ridiculous when it's not spread over a hundred or thousand users.
> Or run a local model.
Much better advice. But still an obnoxious amount of effort and restriction to avoid spying.
> It seems like some people seem to think that a chatbot should guarantee total privacy like a confession to a Catholic priest or something. That is ludicrous and unrealistic. It's a commercial service subject to terms and conditions. If you don't like it then don't use it.
I want it to have the same privacy as using a local program. Does it still sound ludicrous if I put it that way? I'm not talking to a person that would have to suppress their own conversations later, and there's no need for any data from this chat to escape into the outside world. It's like wanting a phone line where the phone company won't listen in; even if doing that is standard procedure and I can walk away it's still gross.
In the case that the company running this program for me is served a warrant, I accept that saved data would be shared. But proactively screening my data for a reason that isn't to help me is bad. We should fight to make that easier to avoid.
Also you're talking out both sides of your mouth when you say it's "ludicrous and unrealistic" right after telling me I can negotiate that exact feature in an enterprise contract.
this thread is about ordinary people using chatbots for personal use, not enterprises for work
I have expanded the scope of the thread to make it more sensible.
I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.
Perhaps but no indication of that so far. Agents are happy to set up *arr stacks today.
They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.
Sure, and they could be a front for space aliens choosing who to abduct. Anything is possible!
I'll be glad for open models when the day (inevitably) comes that the proprietary LLMs no longer work in my interests.
> LLMs no longer work in my interests
As articles like these show, cloud-based LLMs don't work in your interest today.
Is it in someone's best interest to let them die in a shootout?
Level 1:
Have you ever thought of doing something you wouldn't be proud of, and then not actually acted on that thought?
If so, or if you can imagine a journal or always-listening assistant hearing your muttering thought. Should you be charged with a crime for that? Who is the victim and what are the damages?
Level 2, the slippery slope:
Suppose you travel to a country where your sexual orientation is illegal. If your AI company or social media provider knows your orientation, should they be compelled to reveal your crime? Why or why not?
For them to die in the shootout, you have to assume there was actual intent to execute said shooting.
Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.
That day was yesterday. LLMs from big tech regularly refuse to do what you want.
Wouldn't they have to fine themselves first?
They were fined for their illegal downloading. More than a credit card limit.
And dramatically lower than their expected value from the copyrighted material they scraped
Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude
Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
> people need to get it in their heads that they're not chatting with their secret BFF
I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.
People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.
Problem is that even not using those apps, the apps you currently use might have turned more hostile.
It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.
How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?
> Now, everything you say or write can be automatically screened for red flags
Considering that many people will share their deepest thought with LLM, it might start looking more and more like attempts of precog agents from Minority Report...
You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.
The Silent Generation, version 2.
They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment
Right, the difficulty is partly that they can get a negative headline from any choice of behavior.
"Anthropic failed to report murderer's threats to authorities"
(or "Chatbot knew man was planning murder, yet company did nothing")
"Anthropic reported private chats to authorities"
(or "Arrested for chatbot fantasy")
To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".
> in private chats.
Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.
> they're chatting with Big Tech
They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.
It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.
Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.
As if it is just nannyism marketing by tech companies. The EU tries to bring a new nanny law into legislation every other month
it was in the article
Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.
We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.
I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There'd also be fewer people falling for apocalyptic Rationalist delusions.
Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.
You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?
My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).
> There'd also be fewer people falling for apocalyptic Rationalist delusions.
Um, this is coming from the researchers building these models. So, good luck?
>I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There are people with education here that don't fare much better, so I don't know.
There'd also be fewer people falling for apocalyptic Rationalist delusions
Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.
Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.
> I have some sympathy for Anthropic here [...] So from their perspective, it's damned-if-you-don't, damned-if-you-do.
On the other hand, they did put themselves into this position deliberately.
By offering a product?
Show me any product, no matter how simple, that has no safety vs utility tradeoff.
They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.
For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.
that tells users upfront "we can read whatever you type in here"
(and not buried on paragraph 56 of a 20 page TOS that no one reads or understands)
You would first have to demonstrate the utility of whatever product Anthropic is offering.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
I think Anthropic did the right thing here; but the sheriff's office are probably demonstrating why she dislikes them. Writing a diary entry to a chatbot is clearly not how this law was intended to be used.
EDIT: Actually, on reflection, making this report to the people she was upset about was probably not the right call. If they'd sent it to the FBI, there'd be a much lower chance that someone felt the need to assert their "authority".
>> I think Anthropic did the right thing here
This is the paradoxical times we live in right now.
Don't do something? She walks into the office and start shooting the place up. Several officers and innocent people are killed. Cue the media claiming, "You should've known she was talking about this an AI bot! Why didn't the bot tell anybody she was planning a mass shooting?!"
Do something? She gets rolled up by the cops and questioned about what she was talking about and brought to the cop station and interviewed. Cue the media claiming, "This is an unethical way to use AI, this is an infringement on free speech! This is authoritarian!"
I believe in free speech as much as the next person. But in this day and age, its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
> This is the paradoxical times we live in right now.
It's always been complicated like this. That's why certain professions (psych, lawyer, clergy) come with rules around when and if disclosure is allowed[ required, and/or admissible].
First and foremost journalism is the business of turning news into attention (and ad revenue), not informing the public.
Mmm.
As a Brit, I'm aware of https://en.wikipedia.org/wiki/Twitter_joke_trial
I can't say I actually disagree with the initial prosecution. The penalty was a fine, likely less than the cost of investigating it.
Intended as a joke? Blowing off steam? I can understand that, but given the number of people on social media is large enough to include genuinely unhinged people, you can't expect anyone who receives such as message to take them as a joke.
Same with AI use. A billion users, you have to assume some of them are actually sincere if they write about any act of violence, from self-harm to a plan to steal a nuke and use it in a false-flag attack to trigger WW3 and everything between.
> you can't expect anyone who receives such as message to take them as a joke
That's a distinction that matters to me. Sending a spicy note to an LLM isn't remotely the same thing as posting it on social media where the entire world can read it.
I kinda agree, but on the other hand anything you send to an LLM has to be viewed through the lens that only an automated system (i.e. an LLM) is capable of even handling such a tsunami of natural language.
It absolutely will misclassify things, it doesn't know any better.
(We should all wish each other good luck, because we're going to need it).
They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers, but by and large people seem to accept “we take technological measures to ensure privacy and that means we can’t spot crimes.”
So this conundrum is at least partly of their own making.
> They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers
The companies making these AI chat bots are the people who most want to surveil all their customers. The plan is (or will end up being): spy on what their customers are saying to their chatbots, use the conversations to further train/improve the AI and to increase the user's engagement with it, mine the conversations for every scrap of private/personal data and build dossiers on the customers, let other companies and governments pay them for that data or at least turn the chatbot into a shill/manipulator targeting customers based on the contents of their dossier, then use the contents of their customers dossiers for anything else they want.
If they actually set up their services to be private it would kill almost all of the expected value and also cut them off from their best remaining source of training data that isn't the AI generated slop their products are increasingly polluting the internet and the rest of media with.
It feels icky, and my default is not to side with megacorps engaged in blanket surveillance, but I can’t really fault Anthropic here. The correct setup should be a law that protects this sort of interaction with a chat bot as privileged, along the lines of HIPAA-mode. Use a classifier or some sort of “private mode” toggle to tell the platform you’re engaging in privileged communication ala dear diary, and then a much higher legal standard needs to apply to protecting that data (no training, same protections as doctor / psychologist interactions). Even a therapist has a legal duty-to-report in certain situations.
I’m going to guess that planning a mass shooting clears any hurdle that would be in place here.
While the privacy around ai chatbots is rotten in general, I can’t fault anyone who reported this.
Maybe not one that reported it, but the ‘we scan every message you sent into this for disallowed speech’ part is what gives me pause.
They shouldn’t be in a position to report it at all.
Yes in general I don't like that everything you do on a computer is sent to some 3rd party to scan. But the moment they did receive this message it's clear the only right move is to act on it.
Is it really "blanket surveillance" when it only sees exactly what you put into it? That's like saying you're being filmed against your will while... filming yourself.
I get where you’re coming from, but I think if the progression of technology has shown us anything, it’s that there will be constant competitive and social pressures being applied to use these tools more and more and that pressure will race far ahead of any privacy, consumer protections, public education and societal wellbeing counter forces. So opt-out ends up being a rapidly shrinking iceberg in practice.
"slippery slope" is an old phrase for a reason.
Counterpoint, if you give the agent access to your files - which happens to include the Notepad diary you've been writing since your teens - and it makes a similar conclusion about something you privately wrote, would have the same opinion?
Yes, and it's weird that on a form of technologists we're going with the idea that all of these systems we use every day to hold our personal private information that we are constantly ensured is secured against anyone unauthorized from accessing it is actually snitching to the cops just because someone else is hosting it on our behalf.
The 3rd Party Doctrine destroyed the 4th amendment and is the reason privacy respecting software has to play legal games. E2EE while a good security practice shouldn't be necessary to protect you against the cops rummaging around your stuff. The bar to establish that information is private shouldn't be "literally mathematically inaccessible but the cops are still allowed to try."
Would you use an AI service knowing that they are inclined to turn you in to police if they detect illegal activity?
In a "three felonies a day" universe?
> Would you use an AI service knowing that they are inclined to turn you in to police if they detect illegal activity?
The appropriate question is whether I want to live in Florida. This is much more a Florida law problem than an AI company problem.
It would apply if you happened to use Office 365 to write your diary.
Sadly, this isn't a Florida problem.
I don’t live in the US. But this kind of broad law is hard to implement without surveilling all users, and it has multiple side effects.
What happens if I use Claude or ChatGPT to research sensitive social topics? Would that be considered a social network interaction and used against me when I apply for a visa?
Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
> What happens if I use Claude or ChatGPT to research sensitive social topics?
Anthropic/OpenAI is not obligated to report your use, as long as it's not violating some terribly written law (like the Florida law). The government won't know about it, so no, you will not be denied a visa.
> Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
I honestly don't know, but I suspect most US states don't have such an overreaching law.
Legally they have to.
Would you use a lawyer knowing they are inclined to turn you into the police if you talk about committing a crime in the future?
The law is the law.
We should be happy about this as for once the AI companies did the right thing.
This is slippery. Many people do all of their journaling inside of Google or Apple cloud products. Some even write up their intentions to do bad things.
Does all writing now have to be scanned for thought crime?
That's an interesting wrinkle that is rather tough to work through.
To me, journaling your intent in a private journal, whether that's an Apple Journal/Note or a Moleskine in your drawer, feels qualitatively different in some way. But I'm not sure why.
So "I'm going to shoot up the police station" written in your own journal feels somewhat different than "I'm going to shoot up the police station" said to a system that might be able to _interpret_ or _act_ on what was said in some way. Did I just give AI a legal duty, or even a soul I didn't think it had before? I've written up about three or four "what about this, what about that" and deleted them all.
Yes.
But that service is a process on MY computer, loaded into RAM, with an abliterated model.
I don't trust 3rd party networks from abusing any data I give them.
You omitted the part in the diary entry about shooting up sheriff’s office
You're missing the part where it's a diary entry, so the actual content is irrelevant. Her only mistake was not realizing that her diary wasn't private.
Because it's not relevant. They're not charging her with conspiracy, which is what they would do if she'd actually done anything concrete towards making that happen. She didn't email it, or text it, or post it on Facebook or Twitter or Discord or a message board, which is what this law is clearly about.
How can you charge someone for making a threat when you only read the threat by spying on them? Surely that has to be thrown out in court? They didn’t actually send the threat to anyone, you just obtained it by spying.
> when you only read the threat by spying on them
The AI companies have clauses in their user agreements saying they can review content flagged as harmful. It’s not legally spying.
If you recall previous outrage about ChatGPT being used in cases of suicides or shootings, this is the result. Every time a crime was committed and the police found ChatGPT history about the crime, the media turned it into a frenzy. So the AI labs added safety filters to their consumer plans that detect threats of violence, escalate them to human review, and report to the police.
Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department. There might be some limitation in the law that makes the evidence inadmissible because it was not intended to be shared with anyone, but that’s a separate decision.
Not only that they can review flagged content, but they tend to have separate retention policies for flagged content. Anthropic's is this: "We retain inputs and outputs for up to 2 years and trust and safety classification scores for up to 7 years if your chat or session is flagged by our automated trust and safety systems as violating our Usage Policy."
So don't run for office or anything like that. Someone, somewhere will have a contact that will get that.
It kinda is, actually. If the LLM had responded with 'woah, are you serious? That sounds like a crime and I can't just ignore that, it's made clear to the customer that such statements are out of bounds even if they were meant hyperbolically or humorously. But if someone crosses the guardrails and the system silently reports them, that's very much spying.
Obviously, it's hard to judge exactly what was appropriate there because we're being asked to extrapolate from a two word quote about the customer intending to "shoot up" the sheriff's office. Consider the following two statements, which express quite different levels of intentionality.
I got a $200 ticket from a sheriff's deputy today for throwing away an apple core. I'm so mad. I'd like to shoot up their office!
Those sheriff's deputies have exhausted my last reservoir of patience. I'm going to shoot up the department. They'll be sorry when they're sprawled all over the floor bleeding out from saucer-sized shotgun slug wounds. I can't wait to hear the screaming and crying of their miserable families!!"
I'm guessing that the diary entry was a more casual expression similar to the first statement, or they police would have quoted more of the statement to emphasize the apparent severity of the risk but it's hard to say without reading the charging documents.
>Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department.
This is spying with extra steps couched in corporate speak.
I was responding to a question about the legal case. The police did not perform any spying.
Frustrations about Anthropic’s EULA are a separate matter.
Was it claimed that the police did any spying?
Presumably, Anthropic did the spying and the reporting.
You argued that it is not spying, since the spying may have been made sufficiently explicit in the ToS/EULA.
This raises the question: Does announcing a spying operation mean that it is no longer spying? I've never heard that perspective before.
> Does announcing a spying operation mean that it is no longer spying?
Well, kind of, yeah; the dictionary definition of spying requires secrecy and lack of consent.
> to secretly collect and report information about the activities of another country or organization[0]
The only real debate is whether or not having a clause tucked away in a EULA that few people read makes it a secret. If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
[0] https://dictionary.cambridge.org/dictionary/english/spying
I would call it spying in this sense at a minimum if individual people don't know whether their conversations were stored or disclosed in a way they don't want. For example, suppose someone said "we will monitor the activities of 10% of people". You don't know if you're in that 10% or not, but I would still want to call that spying.
A less central case would be when you clearly do know about the activity but you can't quite see the details, like with behavioral ad targeting or something. It feels pretty normal to me to call that spying even if it's disclosed to everyone and certainly happens to everyone, but it's also a less central example of the concept.
Anthropic could put a big flashing warning text at the top of every chat that says “We are spying on you and will report anything scary to the police!” and it would not make any difference in this case.
You can call it anything you like, but only the legal definitions matter for the legal case.
After working on several court cases about surveillance activities, I'm definitely aware that whether I call something spying or not has little relationship to whether courts will think it's legal.
> If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
If you change the situation then yes you can in fact change our responses. The problem is you then are no longer talking about the original situation.
It also bears mentioning that providing a dictionary link to “spying” is pretty patronizing/passive aggressive. On par with sending a basic Wikipedia page. You didn’t even bother to post the definition you want to apply.
No one claimed any case would be "thrown out for spying." The legal definition of spying is also not particularly relevant to the argument in the initial comment.
The initial comment instead questioned how someone could be accused of making a threat if they did not realize anyone would read their private content. You probably also can not insult someone with a statement you never expected anyone but you will ever read.
I didn’t say they did
There is no spying by any definition. It's a chatbot, not a diary. Anthropic is expected to read the message and respond to it in some way. If you sent an email to a colleague threatening violence, you would not be surprised to find out it was reported.
> Presumably, Anthropic did the spying and the reporting.
You don’t need to presume. Anthropic reported it.
“Spying” as a legal concept has a definition that does not apply here. You could say they were “spying” in the sense that they read someone’s input, but that’s literally what they said they were going to do in the agreement when the person signed up.
So I responded to the question about the case being thrown out for “spying” by trying to show that the word doesn’t apply in the legal sense. If you sign up for a service that says “Hey we’re going to monitor your chats and might report things to the authorities” and then they monitor your chats and report things to the authorities, you should not expect the case to be thrown out for “spying”.
Extra steps couched in corporate speak is often the defining line that defines whether something is technically legal or not.
I’m speaking from a functional/ethical framework to be clear. I’m just expressing frustration, not challenging the comment. Could’ve been clearer on my end there.
> This is spying with extra steps couched in corporate speak.
Calling something names doesn't invalidate it. It only invalidates what point you're trying to make.
“Calling something names”? I described what occurred. I didn’t (and can’t) personally insult the concept of EULA’s, as much as I’d love to.
Well, let's say that you have a regular customer at a bar.
They get friendly and loose-lipped with the bartender over the span of months. Eventually they let slip that they plan on killing their spouse for a life insurance payout. At first the bartender thinks they're joking, but it becomes evident that there's an actual plan being acted upon and someone's life is very likely in imminent danger.
Does the bartender have a responsibility to go to the police?
Depends on the country. In some places, there is no legal repercussions for not reporting this to the police; in some, it is an actual crime in itself.
In this case, let's assume the country is the United States, and the state is... oh, of course it is... the state is Florida.
In this case you inform the police only after it has happened, so the civilized world can get rid of the two Floridans at the same time.
/s obviously
For better or for worse, Florida (wo)Man is eternal.
Not the same situation. It was unintended but they are the ones who spoke out loud to a person they knew was present and in a public venue.
Many clankers deny data retention or spying on the user if you ask them. That should be completely illegal.
Then, you can write anything in an EULA but it is not automatically legal either.
This seems to be the statute: https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
With the obvious IANAL, it doesn't seem to rely on the message be sent to the person being threatened. The specific segment is "in any manner in which it may be viewed by another person".
This may be one of those cases where we get to find out how courts view SaaS platforms.
Interesting that it exempts telephone calls. Why don't we treat other messaging services like phone calls?
It looks like a prior statue covered that: https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
The subjective element of crime (i.e. doing it on purpose) is fundamental also in the US legal system. If the person wasn't aware that someone else might see their messages, it should be hard to claim that they committed the crime.
According to Gemini, "Florida appellate courts have overturned juvenile convictions [based on this law] when the state could not prove the person subjectively intended for the record to be seen."
The prosecution will ruin her life regardless of the outcome.
Exactly, can you threaten someone without them receiving the threat? If this is not thrown out, Minority Report will actually happen.
We are snowballing to Minority Report ...
If the AI recommends murder and you exhibit a pattern of following AI advice are you guilty of precrime
> Surely that has to be thrown out in court?
The prosecutors likely know this and expect it. But there's enough gray area here for them to make the argument, and it's hard to prove malicious prosecution, so they know they'll get away with it. It's just about sending a message to the public - they don't care whether a conviction sticks. Just politics.
The prosecutors aren't on the hook, anyway. They have absolute immunity. The decision to charge is protected. The prosecutor would have to have done one of the few, enumerable things outside the scope of the role, like conducting an investigation without probable cause or hiding exculpatory evidence.
this is almost certainly what anthropic is hoping for here - a judgement that says there is no point in them continuing to monitor and report this behaviour
the argument to be made is that allowing anthropic to see it constitutes sending the threat.
sandbox your ai.
That is not what sandboxing solves. A good sandbox would inject credentials into provider API calls so that the model never sees credentials, but the provider is still going to see the transcript. Sandboxes do not require or imply that there is a local model. Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
>Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
this is exactly what I meant. I am presuming the danger is AI reacting to personal notes that it reads on your computer, like a diary, and you should not allow the tools to have access to those documents.
It was sent in a chat message. If you sandbox your coding harness, and then go type threats into it, you've sent it to anthropic.
how does sandbox help in this case when you use a provider like anthropic/openai?
If you're still using a provider like this then you didn't sandbox the AI. You still need to follow the instruction.
Another way to interpret this is that they are legally presuming that you already have sandboxed their product and anything it sees or has access to is intentional.
Any failure to understand what it can access or what it has permission to see from the user's end is presumably not their problem. Regardless of what the user specifically asks of the tool.
It's not quite spying when you willingly hand over this information and agree to terms of service. You're data is not considered yours alone.
It still shocks me the number of people I know who freely let agents on devices that contain unencrypted private keys, freely dump internal data into cloud models and generally don't give a second thought about any of it being trained on, inevitably leaked one day in a db breach or read by providers. I find it's best to consider any data put into a cloud model the same as if it were posted publicly online, since that is the very possible eventual end result.
Hopefully more of these stories push people towards local models :)
Cops will charge them to let the courts decide
It's legal to spy if the terms of service say so
I was thinking the same. If the evidence was not obtained with a proper court order wouldn’t this result in a mistrial?
If you overhear someone, in the privacy of their house, threatening to murder someone and go to the police, surely you don't expect this report being thrown out and you being yourself charged with the violation of someone's privacy instead?
IIRC if the evidence wasn't lawfully gathered (which it sounds like it was, tbh) then it wouldn't be a mistrial, it would be thrown out and then the prosecution wouldn't have any evidence of any crime.
what if it's just testing the AI to see how it responds
A threat sent by mail is still a sent threat even if nobody ever opens the envelope to read it. The crime is in the sending. This woman used an online resources, one which involves transmitting everything across innumerable state lines. I am surprised she isn't up on federal charges.
Note that the law doesn't forbid the writing of a threat. You have to send it to someone. Had she kept it in a book under her bed, she would not be in trouble. But she sent it to a website/service/LLM portal.
>> It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person
If you draft an email threatening someone and delete it without sending have you committed a felony because someone at Google could be reading your drafts box, stored in a datacenter across state lines?
Honestly, I'm equally fascinated by the way email has changed. 30 years ago when you drafted an email but didn't send it, it was only on your local machine. There was no SMTP. 20 years ago, it might be a 50/50 shot as to whether you "transmitted" it to your "Drafts" folder if you were using IMAP instead of POP3 to read it.
We really need a way to make it clear to users when, through the normal operation of software, they are "sending" data to a third party (usually the software developer) and when they are not. This is definitely not clear/knowable to regular users, and it's kind of hard to figure out even if you're a computer expert. Even software that "runs locally" now sends innumerable amounts of stuff back to the developer, and they don't always disclose it.
This is a huge privacy problem that is only going to get worse.
Sounds reasonable. Google's bots could pick that up easily and forward if for human review.
FYI, the use of drafts folders to transmit messages has been used by terrorists. This is likely where CIA director David Petraeus got the idea when he needed a secure way to chat with his mistress.
https://www.findlaw.com/legalblogs/technologist/gen-petraeus...
Ah, a simpler and more innocent time of government scandals. I miss it. Now the messages are on White House stationery and they declare themselves above the law.
It sounds extremely unreasonable to me for "bots could pick it up" to transmute a private note into a felony threat.
It "sounds reasonable" that the exact same action could be a crime or not, depending on how an engineer implemented a feature?
What if she mailed it to herself?
What if she put it in a locked box before shipping it to herself UPS, and she has the only key?
What if instead of UPS, she hired a moving company to move the locked box?
What if she wrote it electronically in diary.txt, but it was backed up to a cloud provider?
--
I'm guessing there's some sort of "reasonable expectation of privacy" for certain activities. We're going to find out what Florida courts think about this new medium.
We'll only find out what the courts think when this happens to someone with a lot of money. It takes a real legal fight to push it high enough to become precedence. She'll be pushed to plea out.
Surely this is the wrong side of what "sending" here will be interpreted as?
Saving is not sending ie passive vs active act.
Everything you "save" on an online service gets "sent" to someone, be that a person or a computer, more often than not across state lines.
> in any manner in which it may be viewed by another person
Does the person have to know (or at least believe) that it will be viewed by another person?
She likely didn't think anyone would view it. Honestly, even as a career software developer I don't think it is unreasonable to think know would would see what she wrote to an AI. I assume most of what I write to an AI is not viewed by any other human, based simply on the quantity of messages sent back and forth to AIs, I would assume a vast majority are not read by another human.
What if she had written this into google docs, and she kept a diary there? That also crosses state lines, and is transmitted to another location.
I get were you are coming from but this all feels like it needs more context to make a better judgement.
You can argue from technicalities but they would need to prove intent.
Reverse the situation and the media would also turn that into outrage. Imagine someone shoots up a sheriff's office and then it turns out they declared it to some chatbot, and the AI company failed to detect and report it, and "didn't push back enough" whatever that would mean, and hence the AI was implicitly complicit etc. That would also be a major PR catastrophe.
Damned if you do, damned if you don't. Same as with social media platforms. That's because only a tiny tiny sliver is for true privacy when that means "bad things might happen" or bad people, such as your political enemies, may do stuff you don't want.
>Damned if you do, damned if you don't.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
Unworkable in practice.
You would expect them to enforce every possible legal standard, in every jurisdiction.
Why stop with criminal law? It should flag torts too, right?
Absurd.
The second order effects of what you are suggesting are a total surveillance state.
It looks inevitable to me. Technological ability ushers in the new social realities. The industrial revolution turned around how we live, intelligence on tap being able to interpret everything in real time is indeed leading down to the total surveillance state direction. I don't see anything that may stop it. You will have to run constant real-time surveillance systems on all your devices and anything that doesn't run those will be illegal, with secureboot on steroids. Due to the convenience benefits, people will willingly give access to cloud-based AI services to all their information, all their plans and documents and calendar and email and everything. And you blink twice and it all becomes mandatory, and you will be an extremist if you don't want that, because why would you have anything to hide? "Just follow the law, follow the rules, and you'll be fine", that will be the slogan.
Stories like this article have zero effect on normal people. They see a crime being prevented, which is good. You have to bring a story where a sympathetic character is getting the short end of the stick somehow.
You would still be able to not write everything into a LLM? But yea the surveillance aspect is dystopian. I would not be surprised if Facebook, twitter and co already know a lot but have simply not reported it. Now the LLM follows the rules.
With this logic, ISPs should be monitoring all texts and internet traffic. Might as well extend an open line to the gov’t to ensure it escalates even more appropriately and faster
I sometimes think about how norms have changed. People are more neurotic and more longing for safety than before. If the Internet was invented today, it would likely work that way. (Though another aspect is that when the Internet was invented it was simply technologically impossible to do a deep semantic scan of messages.)
Similar thing is that I believe if motorcycles were invented today they would most certainly be banned from the road due to their safety properties.
There needs to be a balance. It feels off, today & consequences of future, that technology has led to the collapse of privacy.
And as we progress we will need to rationalize what it means for private companies, AI, to know everything about you and for the government to have a tap into that. I hope balance lands in favor of things like privacy preserving underpinning.
Government would have to explicitly ban companies from surveilling, but they are fully incentivized to want the opposite: government naturally wants maximum amount of info to have the most agency, to know what's going on, to be prepared for any threats to itself, to make sure things are running smooth, that taxes are being paid, that regulations are being followed etc.
AI ≠ internet connection.
Google also monitors your searches, it’s to be expected that an AI lab will know all your prompts, they aren’t providing a paid service for free out the good of their heart. lol.
>it’s to be expected that an AI lab will know all your prompts, they aren’t providing a paid service for free out the good of their heart. lol.
Er. When you actually pay, at least Anthropic gives you an option to not use your prompts in their training data.
Why should we not treat AI like critical infrastructure
if they keep a tap on you, how long before that data gets used for ads or sold to an ad agency?
If you intend this to be scary, you're miscalibrated. For the vast majority of people ads may be an inconvenience but are not some horror scenario. You must come up with better examples if you want your argument to be effective.
There's a term I can't remember for the analysis error of collapsing an entire heterogeneous population into one group for analysis.
The story you're describing would have triggered outrage. And this story will trigger outrage. And generally, the people who will be outraged are different people and we don't have to treat those two outrage reactions as morally equivalent.
In singular online communities (like this website) I've seen this referred to as the 'Goomba fallacy' [0].
But I can't help but wonder if it isn't some socially apparent phenomenon stemming from Simpson's Reversal [1] as applied to group sentiment analysis / polling of opinions.
[0] https://knowyourmeme.com/memes/the-goomba-fallacy [1] https://en.wikipedia.org/wiki/Simpson's_paradox
I have told llms all kinds of stories to find out what its answers would be. I always make it sound like it is the truth to make sure the AI answers in a way that it would if somebody actually said this. I also tested internal flagging systems of the ai company I work at with the most evil things a person can ever say to find out if it would flag them.
Of course I did not mean any of that stuff, but how can you make sure a human reviewer knows you did not mean it while the llm does not know that you did not mean it.
I guess its a miracle I am not in jail yet.
Flagging people for anything said to an llm sounds wrong to me because an LLM is not a real person and while some people put in their internal thoughts, others just roleplay and the two are inseparable just from reading it.
Don’t worry, they’ll store those messages forever and incarcerate you at their convenience.
I was sure I couldn't be the only one curious to push LLMs to their limits. Though these days it's much tougher, mostly impossible to get them to react in unforeseen ways to horrendous scenarios.
This is exactly the typical use I make of the llm.
Adding: - I typically ask questions in the I form, regardless for whom or why I ask for. - Gemini chats quite often end when it starts recommending psychological council or a suicide line, to talk about my problems. It apparently detects a persistent tendency to not agree with the party line. So it makes sense I must be suicidal ;-
But sure, as llm's start to babysit us, and know our inner dialog better than anyone else, we'll soon be debugging their opinion/behavior/co-existence/authority, when it comes to reporting people to the authorities, or taking on tasks in society in general. We'll hire doctors to cure our psychological profile from our record (Total Recall).
A Minority Report like this shouldn't cause a referral to the police.
I don't understand how she violated this law:
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism.
She didn't threaten anything, she wrote down that she was going to do it. A "threat" is more than a mere statement, especially when written in what is described as a "diary".
> A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office.
Obviously I don't want anyone to shoot up anything, but this seems like a weak case legally speaking.
I think it's fair to say this is a gray area. Clearly it was transmitted.
I can certainly threaten you harm and send it to not-you and you're still clearly in danger even if it wasnt transmitted to you. So the question becomes did she transmit it to someone? Clearly yes she transmitted it to Anthropic. But she clearly intended to send it to Claude, an inanimate object.
Claude's terms of service makes it very clear that their employees will read messages[0] to determine that they don't contain the things that these messages contained[1].
> Review is needed to enforce our Usage Policy... designated members of our Trust & Safety team may access this data on a need-to-know basis as a part of their evaluation process.
[0]: https://privacy.claude.com/en/articles/10458704-how-does-ant...
[1]: https://www.anthropic.com/legal/aup
I highly doubt she read the terms.
The critical part of a "threat" is that the perpetrator takes some intentional method to deliver it.
I don't think it qualifies for this part:
> The communication must be made in a manner in which another person may view it.
Even 'transmitted' is too broad if you also consider iCloud backup to be a means.
How could it fail to qualify for that? Another person did view it, which clearly establishes that another person could have viewed it.
The clause in the law is pointless, since if you do something that nobody else can see, you can never be punished. But many, many, many laws are written without regard to whether they make any sense.
A reasonable person would not expect humans to review the millions of messages passing through the LLMs, or their own threats to ever be transmitted to a human without their authorization.
Reporting the danger is by itself a good deed. But there should be a better way of restricting firearms from the probably irresponsible lady than using inappropriate charges to punish the thoughtcrime, OR waiting for them to commit violence.
A sibling comment includes an important rider to the provision: "...in any manner in which it may be viewed by another person." If you wrote this in a google doc, it almost certainly would not qualify as a threat under this statute. Even though google docs, like LLM chats, have administrative override and you could look at their contents - you would not expect either to be "viewed by another person."
IMO I do not think this is a grey area and it's legal to tell a LLM you want to kill someone. It's certainly not a "threat" like you might send to another person, though it may end up being evidence of conspiracy or premeditation. I suspect we would be well served to, after a few years of experience, put together some laws governing when LLM chats must be made available to authorities.
It is very interesting that the LLM responses to these lines - the context around what she is saying - is not in the article. I suspect, as is the case in many instances where LLMs are involved in violent planning, that the LLM was urging this behavior on. Basically entrapment - you are encouraged by a robot to become more violent and vindictive and then when you do you are handed over to police.
I do take your point. But I would also ask you to imagine that instead of threatening to mass murder a bunch of people she was just creating text based csam. Still, all of what you said applies, but clearly she would be prosecuted over this victimless crime and the jurisprudence in fact disagrees that it would be victimless. So applying that reasoning to this act she clearly committed the crime (legalese notwithstanding). The fact that threatened mass murder doesn't trigger the same thought process is, well at least interesting to me.
Too broad. Unless you're transferring ink from a typewriter ribbon onto paper in a hut with no electricity, your words, or my words as I type this, are being grammar checked by something partly in the cloud. If I delete my words, are you saying I've transmitted them nevertheless?
She may have assumed that the chat conversation was private, but it wasn't. She sent a message of intent to harm and a human received the message.
There was no intent for a human to read the message. By your logic, if she wrote a threat in a diary and a burglar broke in and read it, it would be a crime on her part.
Chats with a company's computer aren't private the way a diary is. A better example would be she intended to write it in a word document and instead accidentally sent it in an email to a random person.
Okay so say she wrote it in Office365 in the web browser?
Yes but the law usually evaluates the application of a statute within the context of someone's mental state. This is why you are not guilty of battery when you trip and accidentally bump into someone. https://en.wikipedia.org/wiki/Mens_rea
That depends on the crime; several related crimes are only distinguished by intent. Negligence is itself a crime if it is the cause of a preventable death when the person has a reasonable obligation, such as when driving a vehicle.
I'm not really sure that this can be likened to a diary when it is called a "chat" but that's for the legal system to determine, not me sitting on my couch.
Any reasonable person presumes when they chat with Claude that it is a computer program on the other end. "Claude is AI" is explicit on the page right under the input box. The word "chat" doesn't anthropomorphize the situation.
And yes, some laws are "strict liability", I don't think this one is.
They didn’t receive it, they secretly extracted it by spying on her.
Saying that she “sent a message” is both literally true and obviously intellectually dishonest.
If she had intended to write it in a word document on her computer but instead accidentally wrote it into her email client and sent it to a random person, I'm not convinced she would escape getting charged then either.
It's not any different than telling an automated phone voice tree system that you plan on killing someone and then being surprised that your words were later heard by a human. She absolutely told a company's computer. She sent the message.
The law may have been intended for more direct threats to a person as a means of intimidation, but that's a separate conversation.
Sure, you could intentionally conflate analogies like that if you wanted to mislead.
The Florida statute requires that it be transmitted in a manner that can be viewed by another person. If you have no idea that someone could view your communications with a chatbot, did you really intend to break that specific law? Technically, that threat was communicated to another person but not through her own intentions.
What if she mailed it to herself through the post, and her housemate accidentally opened the mail?
Yeah.. if you write a personal note and it's backed up by the operating system, it appears to be in violation of this law as well (since the company could theoretically read it)
This almost smells like thought crime... Minority Report when?
People assume there won't be another human in the mix, but there is. She was judged for what she probably assumed was a private thought when it was actually not private.
Are thoughts illegal?
Sharing them certainly can be under certain circumstances. I wouldn't be too surprised if this case gets tossed, but it's not inconceivable that prosecutors could win on the grounds that AI chats do not have a reasonable expectation of privacy and are therefor meet the requirements in the Florida statutes, or even more likely, they find some lesser charges for a plea bargain.
It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person, when in such writing or record the person makes a threat to: (a) Kill or to do bodily harm to another person; or (b) Conduct a mass shooting or an act of terrorism.
— via https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
The DA is serious about "in any matter."
> in which it may be viewed by another person
To me, that is the more interesting legal question. Does a LLM-based safety net that sends content to a human, when the original use case would not have sent it to a human, count as "may be viewed by another person". It certainly wasn't intended to be, and that isn't the norm. At the same time, because no security is perfect, we could say that any digital record, stored in any way "may be viewed by another person."
Something for the courts to sort out, of course.
The usual thing that makes laws against criminal conspiracy pass First Amendment muster is that the words have to be combined with some concrete acts furthering the criminal conspiracy. That might just be something as otherwise innocuous as looking up the blueprints of the bank you talked about robbing but it has to be something other than just talk.
Lotsa words in that law... except they don't define what a "threat" may be.
If you enter my house I will k!ll you. <- is this a threat? noone knows. The interpretation of the word "threat" is unknown. Besides, is a conditional a statement? who knows. But sure, blah blah... "in any manner in which it may..." these words are putting me to sleep.
Except if you're one of the "warfighters" that Anthropic supports
I mean, obviously? Soldiers operate under different rules than civilians. This has always been true.
Is the LLM now "another person"?
The other person is not the LLM, rather the service provider’s employees.
> may be viewed by another person
Was it viewed by another person? Yes.
So a written threat only becomes a crime when someone reads it, even if you never intended for anyone to read it? Is it a crime if I make a threatening statement in a diary and someone breaks into my house and reads the diary?
Depends on how good her lawyer is now
The company that runs it is
I suppose since Anthropic's T&Cs allow them to have a person read your chats, that makes it violate the law. Of course, if Anthropic didn't have that in their T&Cs, it wouldn't have been illegal to write.
And it would never have been read by a person (wink wink)
Anthropic commits literal felonies by stealing millions of books and violating Copyright like it doesn't exist: no charge.
One unfortunate woman who happened to write the wrong thing in the wrong place is now having her life turned upside-down for perceived thought-crime.
To Anthropic, and all employees working there, your company's product and the result of your work is cruelty. You are enabling it and pushing it down everyone's throat. You can never again claim that you are the "ethical" AI company, for no such thing exists.
Oh man, what a crazy time to be alive.
Over in Europe they want to read all ofd our private messages, yet these chatbots, pretending to be our friends, will snitch on us just for our thoughts.
It's getting pretty orwellian out there.
I once had a copy of 1984 in my checked baggage returning home to the USA and when I was unpacking the bag at home, the book had a notice inside the book that my bags had been inspected by TSA...
I hope the TSA agent got a chuckle as he put the notice in the book. I would assume his choice was deliberate.
I reckon they were just checking for money or hidden compartments. Sending books abroad packed with money is extremely common from the US, though not sure how frequently people do that with onboard luggage.
Only tangential but when I was an undergraduate studying philosophy I had Bertrand Russell's Why I Am Not A Christian in my carry-on, and the TSA saw that and had a field day.
Please elaborate - what did they do? And what airport was this?
Detroit airport, 2010. I was chosen for secondary inspection and got some snide remarks of "we've been seeing too much of this lately" in reference to the recently failed underwear bombing attempt on a flight from Amsterdam to Detroit.
I can only assume you meant they had a field day celebrating how much of Russell's philosophy matched the concerns about Christianity expressed by Jefferson, Paine, etc?
They wouldn't see it until they opened it, so clearly they didn't open it because of the book. And if they need to open your suitcase (not because of the book), they have a reason for that.
I was chosen "randomly" for secondary inspection. This was after a failed bombing plot and the TSA implied a lack of Christianity was the proximate cause of the attempted attack.
But did they see the book before you were randomly chosen, or after?
After. But then I continued to get selected for additional screening on the rest of the legs of that trip and the return. So either I was going to get searched every time I got on a plane already, or they flagged me for additional scrutiny after the initial search.
In Europe they should provide the citizens with the service of their messages not reaching US servers. So basically that there is only one party reading along with them, not half the world.
Chatbot transcripts should have the same legal protection as phone calls. Judge's warrant needed to access them.
Anthropic (in discussion with Pentagon) claimed mass surveillance is their red line.
Yet, they do automated mass surveillance of their users on behalf of police.
For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
What would happen if it had scanned a file it didn’t have permission to look at and found this threat?
I honestly don’t know how I feel about this. On the one hand if you’re using claude as a diary you have no expectation of privacy and she was talking about committing a very serious crime.
This still makes me feel queasy though.
Both of those scenarios would demonstrate even more commitment to safety so I imagine they’d be at least as likely to happen as this, if not more.
> For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
For the sake of even more argument, imagine if she was writing her thoughts with a pencil, on a good old fashioned paper diary, and she had a phone nearby and the phone took a picture of her diary, OCR'd the words, and reported it to the police?
I guess we probably want our tech to work exactly like this.
It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
___
Of course, we also want purely private tech, but that needs a certain level of merit and sanity filter.
>I guess we probably want our tech to work exactly like this.
Do we, though? What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products. I don't think it's the tech companies' job to surveil the population and prevent crimes. That said, I'm not necessarily against Anthropic or other companies reporting suspicious activity if their existing systems are detecting it. I'm just not sure we want every product to be forced into that data model.
Your follow-up about purely private tech seems to contradict your first statement. We can either have privacy or surveillance, not both.
>What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products.
Why wait for a company to build it? Get your own local hardware like I did and have those guarantees because YOU set it up.
I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Last time I checked it costs closer to 500K to run SOTA open models at any usable speed.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
> Your follow-up about purely private tech seems to contradict your first statement.
That is correct! And exactly my point.
We want both, but, on paper, that is impossible. But in reality, we make it sorta mostly happen anyway, through making the easy defaults not private, and the private stuff not easy.
This is not ideal, because [various reasons I do not need to tell you], but it has proven to be the best we can do to mostly achieve both goals.
Kinda like how capitalism isn't great but just the least worst option we've found so far.
___
The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff.
Hence the quadruple-speak and contradictions to kinda sorta somehow have a somewhat functioning reality.
That is kind of rambling. Our rambling score says we should observe you.
Oh don't worry, I'm certainly already on various lists, but the observations also will have resulted in the assessment that I am stable and no threat :)
I think we can have both and it might even be smart.
A lot of people causing issues are people that can't make sense of many things (like many terrorists). They get a fixed idea and they end up doing something bad. You would catch those with some (basic) surveillance.
A lot of normal people (not wanting to cause issues) might benefit from some privacy, if they understand what are the trade-offs (like government overreach). They can then use a slightly more complex tech.
We would still remain with the couple intelligent but sociopaths (think Unabomber style), but I think no solution can fix all cases.
That sounds adjacent to "I have nothing to hide". Everyone says that until they realize someone can change the rules. Before 2022, women didn't have anything to hide from their period-tracker app, now some have to worry about being charged with crimes.
Back to LLM chats: A system that can declare her "unstable" is also one that can permaban you from all air-travel because you "privately" said unflattering things about Dear Leader.
> That sounds adjacent to "I have nothing to hide"
I urge those people to share their full information from banks, companies (salaries, agreements, contracts), full health information and share publically all the texts they ever wrote (incl. as teenagers) and all the photos they have taken. And give away all the passwords to all of the services so people could check that they are truly clean. Just to be sure, just for everyone's security, right?..
> It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
So... minority report?
The third highest voted comment wants surveillance. What is next? If you write "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported.
You don't want surveillance for dangerous people?
In a old happy little idealised village, it became known quickly, who started to behave oddly and timely intervention could happen. In the modern anonymous mass cities?
No one (wants to) notice the madmen scheming in his isolated flat, surrounded by strangers. Until he explodes.
Unfortunately I also don't trust our government agencies with the surveillance - because they ain't transparent either and the self surveillance seems broken.
" "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported."
And unfortunately there are lots of real threats being made under the disguise of humor. And much harder to separate im text. So maybe don't talk of murdering people in general, AI surveillance or not?
How do you know they are dangerous, without the surveillance? In other words, the only way your system works is if you surveil everyone.
Autists have taken over the internet and everyone must obey their weird rules.
the honest actual opinion of the average consumer is probably that they don't want their chats to be monitored but they want everyone else's chats to be monitored. There was a lot of fury when a mass shooter recently used AI to help plan his assault. And of course there are all the people talking to chatbots about suicidal thoughts and intent. When they ultimately follow through, the providers are blamed for not alerting anyone.
No. This sentiment is why Snowden happened. We want privacy. Privacy isnt free just like freedom (whatever form it is) isnt free.
There are trade offs. ISP effectively is like driving on a highway, everyone can see where you are going but not what is inside the car. Id like these AI chats to be the same but they are not.
Devil's advocate: if I tell my therapist or my lawyer that I'm going to murder someone, they're obligated to report it. If I use my AI as a therapist or lawyer, why should the company that provides that service not be held to the same standard?
LLMs aren't email or file storage. AI labs aren't just shuttling bytes around, they're interpreting those bytes and taking action based on them. These models _already_ react viscerally in response to users saying disturbing things: the only practical difference is the ability (or obligation) for the model to escalate that concern. I'm not sure the ethics we hold AI companies to should be different than if a human being was typing out the responses.
Privacy is obviously hugely important, but this isn't the government surveiling every message. It's companies having an obligation to flag real, credible threats according to the law, which is a very different problem space.
>I guess we probably want our tech to work exactly like this.
Maybe you do; I want my tech to always include secure, encrypted communications. Don't include me in your destruction of privacy with your silly bandwagon!
uh no, 'we' absolutely do not want our tech to work exactly like this. Why would you assume that people default to 'search my anal cavity please' and not 'no, stay the fuck away from me with that glove'
----
jfc, why is this website full on psychopaths
"The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff."
other people are not pretending everyone is equal. your power levels are showing, it isn't subtle.
True, most normal people have little concern for any such humanist goals and ideas.
Normal people live prejudice. It's (erm, claude-speak) load-bearing for them, given just how complex reality is and given just how well it reduces that complexity.
“But there can be no valid knowledge about the future. As soon as precognitive information is obtained, it cancels itself out. The assertion that this man will commit a future crime is paradoxical. The very act of possessing this data renders it spurious. In every case, without exception, the report of the three police precogs has invalidated their own data. If no arrests had been made, there would still have been no crimes committed.”
Philip K. Dick, Minority Report, 1955
While I accept that this sort of thing is well with in the ToS and regular course of business of any major online platform, it hits different coming from an AI company for some reason.
AI is becoming critical infrastructure. Laws will follow. Ie similar to ISP/communication infrastructure laws
First I posted to the (likely dup article) https://news.ycombinator.com/item?id=49965895#49966728. Wanted to post here, after reading more.
Is this an invasion of their privacy (reporting to police)? Yes, but possibly warranted?
Should a social worker have contacted them rather than the police? Probably, if for no other reason than to ask if they were serious about harming someone.
Difficult questions, I'm still undecided on whether it's OK to always ignore someone's rants, even if it may be (or they think it may be) a private diary.
Actually charging them with a felony seems pretty quick to accuse. (Maybe I missed a hint about how long the investigation took before the felongy charge?)
It is my very european belief that the problem here is not that the woman was reported, but that what likely is a mental episode was made public in a way that reduces the chances of recovery.
The article says she made the comments on September 26.
1791138022 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49956424 | 0 comments
1791144575 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49957340 | 0 comments
1791147034 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49957692 | 0 comments
1791149399 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49958089 | 3 comments
1791213096 | Florida woman arrested for allegedly making threats in an AI chat | https://www.theverge.com/ai-artificial-intelligence/1004747/... | https://news.ycombinator.com/item?id=49965895 | 2 comments
I guess all the "private model" people are right. Don't want to end up in jail (or even charged with something) for asking a crazy hypothetical question or something.
Won’t help if your model is trained to immediately find a way to push out a message somewhere to alert authorities of your threat
Models don't have access to the internet unless you give them access to the internet
I have a hunch she wins the case, on the basis that an LLM isn't a person and an (assumed) private expression of anger to a machine assistant doesn't meet the statutory threashold which requires that a threat be communicated to some other person.
Of course, that could change if there's evidence that she took action in pursuit of a goal, like buying ammunition or repeatedly driving around the entrance to her alleged target.
The law seems overly broad with "may be viewed by another person". Most of these laws have the intent of not causing public panic with regards to posts that others may see, not stuff that one assumed was private. This is further supported by the definition of threat, which is generally defined as requiring malicious intent.
I would think a good lawyer could get this charge dropped. I would like to see what judge approved the warrant and how they felt the elements were met.
Should the public have an expectation of total privacy for their chats? It seems responsible for a chat provider to report things like this.
If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime? Basically everyone agrees that crime-committing advice is inappropriate…but if it is not ok to get advice, that means there must be a portal for law enforcement to step in when that may have happened. Then the question becomes what is the line for when to report? In other words, the issue needs to be adjudicated.
But we don’t want OpenAI/Claude to have some $20/hour reviewer making decisions that are this high stakes…we need the courts to do the judicial work because they (1) have a public charter, (2) have meaningful expertise and specialization at interpreting the law and (3) we can hold them accountable.
It doesn't really matter if they "should or shouldn't" have an expectation of privacy IMO, they already do have that expectation.
> If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime?
Yes it should be, but it should be illegal for a company providing chat service to respond with anything other than a refusal when doing so.
That detection and refusal should be a private closed loop though, anonymizing any data that will be passed into a training pipeline, or ads targeting. This requirement for closed loop private chats should be mandated by law sooner than later. Otherwise we're getting into very tricky territory where the temptation of alerting on things like pre-crime grows too close.
> making a written threat of violence under Florida law.
A diary constitutes making a threat?
Oh boy the roleplaying part of LLM world is in for a bad time
> The communication must be made in a manner in which another person may view it
IANAL but it will be interesting to see how the legal system decides if this counts as "another person may view it" or not. What has happened in similar ish cases where someone writes a threat like that that they thing is private to them but actually ends up in the hands of Someone at some tech company that reports it?
Not weighing in on the privacy issue but using Ollama you can run a smaller agent like Qwen 3.6 35b a3b on a sufficiently potent laptop. Pair it with something like Hermes for a nice interface and you have more than you might need for diary like usage.
Could this be considered free speech? Most of the limitations I know of require you to say it to or in front of other people as to create a panic or make someone feel threatened. Is it a crime to say to yourself "Im going to blow up New York." or about threats of self-harm? How about "Im going to steal the Mona Lisa!"?
To me this is just straight up thought crime, they just don't have a way to directly read your thoughts yet. But they will spy on you and try and catch you out for it.
This is a classic case of "Damned if they did, Damned if they didn't". Given this has come out, and Anthropic is considered to be the beacon of transparency, it would be appropriate for them to share what their thresholds are. The article also reports Open AI not considering the threat credible enough to alert, so there's clearly some thresholding that people need to be aware of.
In any case this is proof that law and negative PR can influence tech companies, including frontier AI providers.
Then again, I wish this worked in a way that would give users more privacy and agency, instead of less.
This kind of thing will get worse with humanoid robots because they have cameras and microphones. Will they be programmed to tell on you if you break any kind of rule in front of them because their owners are terrified of being sued?
https://www.npr.org/2026/07/10/nx-s1-5886113/waymo-police-pr...
Two teens riding in a Waymo were arrested because the AI detected them talking about having a gun.
Yes
this feels very thought-crimey to me, but I think I'd have to actually see that chats to really decide. Like is she making plans/asking for advice? is she just talking about her feelings exactly as if it's a diary?
This news article [1] had a quote supposedly from the chat session:
> The arrest report states that on Sept. 26 at approximately 5:10 a.m., Heller reportedly wrote, "I'm going to shoot up the sheriff's right the [expletive] now." The following day, at approximately 1:07 a.m., she was accused of posting, "This is 100% last chance I'm done. I got a new [expletive] gun today. [Expletive] you."
[1] https://www.gulfcoastnewsnow.com/article/florida-woman-arres...
She is not being charged with planning, just making threats in a place a person could read - the person being employees of the company.
Basically, under this interpretation, any personal note you store in the servers of a company could qualify, even if you didn't ever imagine someone would read and as such you couldn't have thought about it as a threat
> The communication must be made in a manner in which another person may view it.
How does a LLM prompt satisfy this? I guess it'll be an easy win for her.
LLM is not a person but T&C probably states that a human moderator may view any of it. Her lawyer could probably argue a moderator filtering usage isn't the intent of the law, it was more about publishing / sending message for other humans and it was never clear to her that a human was reviewing her private diary. Shouldn't LLM disclose that at some point when people are feeding really personal stuff?
Guess asking claude to write a 5 line example steganography tool wasn't such a good idea then.
I expect that anything I type / dictate / post / purchase on the web be it a chat conversation with an AI, a post on social media, a dm, a blog post, a blockchain reference, or an assett in a bucket, will be manually reviewed and forwarded to the authorities. It is their duty to do so, and I am glad that they do. If this woman did harm someone we would be hearing "why didn't Anthropic do something about this."
You only accept this because you are part of a low trust society. I wonder if tech has done anything to improve trust among people or just lowered it.
Aren't high trust societies staying that way by kicking out the incorrigible violators? Pondering violence might not be treated lightly even in a high-trust society.
Duh... If you don't want want to get reported to the police, don't upload illegal things to the internet.
If only the woman hosted open-weight model in her house.
Florida statute 836.10 puts the bar at the "sending, posting, or transmission of, a [...] record, in any manner in which it may be viewed by another person"
I think the defendant could successfully defend themselves by claiming they did not know (or intend!) the message could be viewed by another person, as they were plainly using it as a private diary.
I can’t believe people are using these proprietary models/subscriptions as personal assistants, sending their most personal context, thoughts, documents and information to the providers. Some even give them full disk access. After using their services for long enough, Anthropic/OpenAI essentially have your entire life mapped out to an insane level of detail, likely including detailed contents of your computer/phone.
People used to say that Facebook knew more about you than your closest friends/relatives. And compared to that, this is just on a completely different level. Absolutely insane.
That's a good reason to only talk about scary stuff with deepseek.
This is practically entrapment. All the AI labs sure don’t shy from plastering annoying disclaimers everywhere saying their tool can make mistakes. Shame on them for not also reminding everyone continually that anything you submit can and will be used against you. Of course they can’t afford to have a Flock-style user revolt.
Where are the people now who claimed that LLM chats are really private and not monitored?
Every single lie of yours is exposed in the past few months.
Personally, I’ve never seen anyone claim this. At least anyone who one would think is informed on these matters.
duck.ai et al? my chatbot claims its architecture provides some level of privacy yet it is honest enough to point out none of these systems are actually audited and "not stored" and "anonymous" does not mean "not monitored".
I haven't seen anybody claiming this for consumer/free accounts anywhere.
You get privacy if you're a big corporation that needs to make sure OpenAI/Google/Anthropic can't read your trade secrets etc.
But those contractual privacy protections have been in place for a long time. It doesn't have anything to do with AI, it's been the same with Office365, Google Docs, etc.
Who claimed this? I've never seen that claim.
Every single person who has told people to use LLMs with their proprietary information and code.
No sane person claimed this.
Uhm, those people never existed? Or maybe you just have very interesting friends?
LocalLLM enthusiasts exist for a good reason.
That reminds me, I need to rewatch Minority Report.
I hope this highlights that many (most?) non teach people don't consider or know that their use of AI services is not private.
If it's going to be this binary. Then there has to be a shift in the way this is handled. It shouldn't be "You wrote these terrible things to a computer, you're now accused of doing them" to maybe "We where notified you wrote these terrible things, and the a psychiatric evaluation is being mandated by the state in which has a law about this sort of thing"
Obviously, as others have pointed out if they don't act and she does the crime it raises the damned if you do damned if you don't. But I've also had the AI's go haywire saying I'm doing all sorts of nefarious things when literally doing math proofs.
So no one size fits all. But going the extreme first is probably not ideal.
In Florida. Where DeSantis said "You loot, we shoot":
https://www.politico.com/news/2023/08/30/desantis-warns-hurr...
This is why I use ZDR and API access.
They still read every single message you send and train on them.
I’ve had them email me before because I was testing it as a filter for abusive messages and they detected some no-no and wrongthink in those test messages.
I looked into ZDR, it's basically a vapid claim with little to no due diligence or auditing. I expect most providers to cave with only a minimal amount of legal pressure.
How much spend do you need to negotiate ZDR?
OpenRouter provides ZDR for many endpoints (if you trust OR and the provider). Naturally, the routed prompts process itself adds additional anonymity.
The downside is that you don't get cached prompt discounts, so you pay a heavy price for ZDR that way.
After people getting pilloried for social media posts from twenty years ago, I really hope (sensible) people will have the wherewithal to think twice about what they hand over to their chatbots.
Could they please report my stalker ex!
AI snitches don't get stitches.
> The communication must be made in a manner in which another person may view it.
Seems to fail this test at face value.
I mean, somebody you live with may find and read your diary. Is that the same thing?
Intent matters here. Did you intend somebody else to view it? Does a reasonable person have expectation of privacy with a chatbot?
Claude did this, its also important to remember that a human counselor or psychologist will also do the same thing.
For example, if you said to your counselor or psychologist that John Smith is going to rob the bank next Thursday, they will report that to prevent the imminent risk of serious harm to others.
And remember that anything but local AI you personally control is not anonymous. Not even AI you pool with your friends.
The future is incredibly dark if they manage to ban open source models.
Tech companies have always done this. Don’t misread that as blanket support for it, but consider Anthropic’s position here. Do they want a headline to come out that says “woman used Claude to plan murder” or “Anthropic reported potentially dangerous person to the police”? AI being dangerous is already a hot topic, and this seems like a sensible move to me. If you want a guarantee of privacy, you’ll need to run your own models locally.
If you want technology that won't report you to the police for credible threats at they can easily detect perhaps create your own. I hear pen and paper work great for this.
We are officially in the era of Thought Crimes.
Can the public also immediately get alerted when a cop or politician does some bad shit, though?
Pre-crime prevention and detention will be the end goal. The economic devastation caused by AI will make large swaths of people get angry. Better use that massive amounts of data being hovered up and the mass of compute being built up to stop those angry people lomg before they even get a chance to properly organize.
For that to happen, someone would need to be spying on all cops.
Knowing anthropic, im sure these keywords are hardcoded in a long list and regex get them for further reviews.
Does anyone else remember when the tech-savvy crowd was wary of software "phoning home" or spying on users, or listening, or whether Gmail scanned and read your emails, and whatnot?
And here we all are, happily typing our stuff into these spy chatbots. "AI" happily made our fears go away. Hopefully we're not planning anything criminal like this woman, but still.
I guess it's time to come up with a more plausible explanation about why I asked how to make nuclear weapons almost every month to test LLM refusals.
"Assume people can see everything you post on the internet" still applies when using AI. It's not as newsworthy when you put it that way.
Do you think they woulda been caught if they used duck.ai?
Writing in a diary is protected by the first amendment. Or at least should be.
Me: Claude, who wrote the song with the lyrics 'I shot the sheriff, but I did not shoot the deputy'?
Claude: Law enforcement has been notified, you are now under arrest.
What a ... nice company.
Snitching on the people - good mass surveillance company.
On the other hand, people need to learn to not trust these companies. It reminds me of others being surprised when a self-driving car reported a gun in the car. I mean, do people not think? Besides, of course, it's already messed up to want to have a gun. And it is constantly one country that has such issues, more so than many other countries.
Can we already have mind reading devices please. I can't wait for them to see what people really think!
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Thought crimes are real when you're sharing your thoughts with Claude
Auditing logs is a crazy interpretation of "another person may view it"
I wonder if "criminal intent" may be missing here given that most people probably operate under the assumption that Anthropic are not reading their messages.
I think people have a binary understanding on this. Someone is either reading their messages or not. While the reality is that all the messages are read by a machine (not unlike GMail and Outlook) and anything suspicious gets flagged up so a human can read it. This obscure the concept of "reading" as most laypeople understand it.
Summary: don't type in Claude anything you wouldn't like a human to read.
It tells you exactly what it does with your information if you ask it, including this exact scenario, and has for at least four months now (when I asked).
Yes but you are someone (i assume because you are on HN) that at least understands this point. There are A LOT of people that use it as their confidant, expecting it to be private. There is a massive education issue going on as its not in the interest of these Corpos to make you fear sharing all your details with them. Because, then you wont get Dot or whatever Anthropic comes up with and share all your personal info with it.
You're not wrong, and it may come down to this in court, but there's a difference between what people think happens, or the reasonable expectations, and what actually happens, and I think it's important to recognise that difference and why it comes about.
I don't think someone is an idiot for thinking that the information they type into their private Claude account is private. I also don't think people are idiots for thinking their phone is listening to them and giving them targeted advertising based on that. Both are reasonable deductions from their lived experiences. Both are wrong.
Should CEOs of Antrophic and OpenAI be reported too then? They cannot stop saying that AI will destroy humans.
laws for thee and not for me
this has nothing to do with claude. “thought” is fine. “written and shared” is illegal all over
https://www.nbcmiami.com/news/local/everyone-deserves-to-die...
https://www.wdsu.com/article/maryland-high-school-student-ch...
https://www.pinellassheriff.gov/21-023-deputies-arrest-pinel...
I think it's a reasonable point to make. Writing things down is a part of "thought" for many, including those who keep diaries/journals. If you write in a private journal you do so with the expectation that is not shared, and that wouldn't seem to break this law (with my naive reading).
I mean, at this point it's pretty well known that all tech services will be hacked by fable, anthropic themselves promised it, so writing your journal in google docs or claude or a txt document on your laptop or such is the same as releasing it publicly yeah?
If it were written on paper, and only in a room with no phones or cameras so fable couldn't hack it, then I think you wouldn't be sharing it.
> “written and shared” is illegal all over
I think it’s valid to ask if tapping something into Claude is legitimately sharing a threat.
I don’t think it is. I also think the sheriff could have found more-substantial evidence if she was actually planning domestic terrorism.
That said, if the shooting happened and we were looking at this from before? It’s a tough balance without an easy answer.
I think the zero-risk approach common nowadays is insanely corrosive to democracy and freedom. If a million people fantasise about shooting the sheriff, and one ever goes on to do it, I don't believe avoiding it warrants creating an apparatus of mass surveillance. After all, if people were really serious about zero murder, the only practical solution would be to lock up everyone. Some (most?) tradeoffs have exponential costs at the limit and we/lawmakers should recognise that.
> If a million people fantasise about shooting the sheriff
I think it’s fair to pre-identify folks who fantasise about shooting anyone. It’s a small fraction of the population that looks into logistics versus making offhand comments.
So now when everyone has their watch recording conversations, or perv glasses, or Alexas, or Jony Ives new personal AI gadget … recording everything we say and scanning it at scale … is it ok to be a doomer yet?
Why not report it? Seems reasonable. At the end of the day none of these services can guard against misuse. I personally optimized bomb creations and other stuff with AI just to see if it is possible. Maybe the way to secure the world is to not have corrupt old farts running every country. As the old saying goes. Information wants to be free. A kinder and more connected world is the only way to keep everyone safe.
isnt this the plot of the Movie Kimi? https://www.youtube.com/watch?v=_Gr2zXuEBL0
This reminds me, for his short story "Dolan's Cadillac", Stephen King claims -- perhaps exaggerating a bit -- he did the real research on how to murder a person as described in the story. How to dig the hole, what measures, what the falling rate of a car that size would be, etc, etc. He claims he fuzzied the details a bit because he didn't want to teach readers how to actually do it (again, maybe he's exaggerating a bit to promote his story).
Imagine if, instead of a friend, he had asked an AI chatbot. Would he have been reported to the police?
Aren't our private thoughts just that, private?
I get that a cloud-hosted AI chatbot is, to tech-savvy people, immediately "not private", but a lot of people don't understand this. What if Stephen King was talking to his friend about planning the murder, and his phone was listening? What if he was typing it up in MS Word and the program decided to phone home and report him to the police?
> Another incident has taken place that illustrates the need to be careful what you tell AI
Nope! Another incident has taken place that illustrates how committed Anthropic and Anthropic AI is to the law and public safety, unlike all that open model riff-raff
They were waiting with bated breaths for something like this to jump on, and make an example of.
They are good at keeping notes on your criminal conspiracy