My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ...
So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means.
Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
its lovely training data. no detection? add to training set -_-.
its also kind of laughable that somehow people are trying to prevent the outputs not to be altered. Asif you cannot manually paraphrase anything you can read. So the only solution would be, to make it utterly unreadable (which is not possible, it obviously defeats the purpose of the thing).
Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over the text and destroy the watermark. I predict an interesting cat and mouse game to develop.
I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated.
I wonder if this would be a good use for homeomorphic encryption. There might
be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!
> Any stenographic system that you have the code for can be trivially defeated.
They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it.
Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're submitting to does the same check it says the same thing.
Won't we just be able to fine tune OSS models to detect these patterns across providers? It will be cat-and-mouse but my bet is it converges to a central detector that isn't affiliated with any model provider.
> blindly trusting they won't train on any of that
being allowed to train on any data that you can legally obtain ought to be a right for anyone.
After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.
> being allowed to train on any data that you can legally obtain ought to be a right for anyone.
I have the opposit viewpoint to the extreme. They shouldn't be allowed to even read that data until they are very clear about what they will or not do with it.
Can they publish it? Can they store it? Can they use the information in it on prediction markets? Etc.
Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.
> Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.
It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.
It's exactly the same problem as with humans, though.
It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).
It's not exactly the same problem, in that you can parallelize usage of an LLM and copy it over to another computer, but cannot do the same things with a brain. Put it another way, humans do not have the processing power needed to answer hundreds of millions of queries per day, while LLMs do.
At the risk of stating the obvious, there are a lot of legal rights that are human-specific (voting, holding office, filling lawsuits, etc.). It's not at all obvious why you think that you as a human being legally allowed to learn from something implies that it should be legal to train an LLM on.
> The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.
But LLMs are replacing the original, just in different words.
And what does 'legally obtain' mean in this context? Copyrighted content is usually licensed for specific purposes. So if a license is given from training your LLM, then by all means do! But what if the license is 'for personal use'... ?
> You are one person. The corporation is not. Scale matters
Correct, if you violate it too often to count, you have to pay around less than ~2.5ct per violation.
So the lesson here is: Create a company to do torrenting professionally, and resell its values for higher prices. Then get sued and pay a dime on the dollar you made.
Oh so if I use mickey mouse in a completely original production that doesn't replace the existing work by Walt Disney, you reckon they'll be fine with that?
Copyright (or any other such restriction on free use of information) creates power for owners by the simple fact that it turns information into something that can be owned.
Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short.
Having proof that content (especially images and video evidence) is unmodified (whether via Photoshop, Paint or a model) is far more valuable then having evidence that an image was manipulated or generated fully by a model (which still leaves other forms of manipulation), I feel the same goes for human authored vs generated text. Free to admit that using models to generate any kind of media whole-cloth is still unappealing to me and I still pay for commissioned artwork or make it with my limited abilities for what that's worth. Do like to (poorly) write my musings too and see UX as something were thoughtful contributors (like the opinionated, sometimes controversial, but certainly talented GNOME Gitlab contributors) can make a major impact.
Code can be beautiful, interesting and serve purpose beyond execution, of course, but for most people, in most cases, it does not in the same way as audiovisual content (not limited to art). Having code just to execute and resolve a problem can have value all in itself, the code being a means to an end whose quality, let us be honest, was barely a concern in most corporations long before LLMs.
Also have rarely (honestly never) before LLMs fully owned all parts of any code base, always relied in part on someone's prior effort in (Flutter/Dart mostly) packages, whereas when writing, drawing, etc. I have far more situations where I make something from scratch and everything there is only there because of my conscious decision. Even simple marketing mockups that, quality wise, any modern model would beat feel different when I was fully in control, where to place what, etc. Objectively worse (at my skill level), probably, but still never the same.
Knowing something was made from scratch by a human has value to me, beyond misinformation prevention. Knowing for a fact that LLMs were used instead of importing a library, using a template, or something similar that leads to expending similar amounts of effort, I don't see that being nearly as valuable. Heck, with all the importing and my experience back then vs now, I am spending more effort actually fully reading any LLM output in my code then I spent back then auditing Flutter/Dart packages. Then again, LLM output fails far more unpredictable then those messy packages that simply got Gradle to take down my system...
Happy to admit, I have been skeptical of watermarking LLM output being feasible for quite some time and having looked into SynthID Text and proposals being researched, I am convinced that it is challenging to impossible beyond the lowest common denominator and less important then proofing human authorship.
I mean this is the thing that really comes off hard.
If you want precision and clarity of your writing, then you need to hand write it. Just like when you are optimising, its common to drop to a lower level language because the compiler doesn't express what you want. Sure its hard, but you know, thats kinda the point.
Even if you don't want that, the LLM is an average of the style it was trained to give out. Which is a homogenisation of the language to create a vague padding medium between a few generalised facts. (because a. it makes it less jarring when stuff is wrong, because its smeared over a higher amount of text and b. it looks more 'professional' because American business English is all guff and no meat)
Also yes, two isolated phrases may have subtly different meaning, frankly, the nuance is missed on most people. If you look at the interactions on here, at least 25% of the arguments are caused by people angrily reacting to the things _they_ thought the other person was saying, rather than what the actual person was saying.
So no its not a perversion, the LLM is, if you're gonna be picky about things.
The bigger problem is its effect on the way we use language. The amount of LLM generated or edited media is going to keep increasing and the media that people consume affects their own word choice.
What an absurd take on a very reasonable concern. As much as I hate “obviously AI” writing, I don’t understand why we have to handicap the tech and prevent it from improving. This manipulation of word choices virtually guarantees AI will always sound like a robot.
Have you compared pre- and post-watermarked text to make that assertion? And I'm not sure how writing your own text if you care deeply about the word choice is handicapping the technology?
I don't think there's been a single version of the major LLM providers that haven't handicapped the tech since the beginning by changing temperature/top-P/frequency penalty/etc. All of those stray from "highest probability" token selection. What's funny is it was done specifically to make it sound less like a robot/deterministic.
LLMs are no more than pen and paper at this point. Especially for those who aren't trying to create slop. We all would want our pens to accurately reflect the strokes (well in this case thoughts) rather than adding tiny watermarks to identify that it is generated by a particular pen or a user.
Watermarking per model is just the start. The method is cheap enough to distinguish individual users.
That is an insane statement, LLMs generate swaths of text from almost nothing.
If they are adding so little value as to be as transparent as a pen and paper then why use one at all? Transcription doesn't need an LLM so that's not what you're taking about I assume.
LLMs do a whole lot more than writing your thoughts down. They write extra text. If you just want a pen and paper, use Notepad. Or better, a pen and paper
The problem is, that LLMs worked very well for me to improve my writing. Especially as I'm not a native speaker, it was a great way to improve the legibility of my work.
I want a tool that helps me improve my writing. A tool I can learn from. Not a tool that switches out "bananas" to "airplanes."
I'm was using Claude Opus and now Fabel extensively for editing my texts and I find the recent updates abysmal. Not sure if it's due to the Text Watermark.
Before Claude was great in sharpening the meaning in my writing, it's now close to unusable.
I'll be interested to see if people can actually pick out which text is watermarked and which isn't once they introduce it. It won't switch "bananas" to "airplanes". It'll switch "I really enjoy eating bananas" to "I love eating bananas" or similar
To that point, given a corpus of writing from person A and another from person B, I have no doubt it’s easy to train a classifier to determine who wrote what. In fact I believe law enforcement agencies already have these classifiers.
The only difference here is that Anthropic is actively trying to make the watermark undetectable.
They're not trying to make the watermark undetectable, that would defeat the point of a watermark. They're making it detectable, but not make the text obviously watermarked
This is an excellent use case. It made learning German much easier. I write what I think is correct, then get a fixed version.
When writing in English though, I use it more like a dictionary. If you want to write past a certain level, an LLM works better as a metaphor and idiom search engine.
I also like to ask it to generate 20 ways to say the same thing. It’s a great way to simplify or smoothen sentences without losing your voice.
Interesting, are you being literal about requesting 20 ways of saying the same thing? It seems pretty excessive and I'm actually impressed that asking an LLM to rewrite a statement 20 times yields output that isn't excessively redundant. Does the LLM do a pretty good job reading your mind, or do you still find yourself manually piecing together pieces from the 20 suggestions into a satisfactory sentence?
Yep, I literally ask it to say it in 20 different ways. Sometimes a sentence structure or a combination of words will just work better. The goal is usually to simplify a sentence without losing meaning.
I don't expect the LLM to read my mind. The unit of work is too small for intent to matter, and I'll just steer the next recommendations in a direction as needed.
Most of the suggestions are crap, but they can contain the seeds of a good sentence.
As a writer, Claude’s metaphors are trite and obvious 90% of the time. It also has a terrible penchant for an immediately recognizable emphatic voice that makes even the best outputs super cringe.
But people do not notice and do not care. In our bubble and John Gruber’s bubble we really overestimate how much people must care. Truth is, we’re in this predicament because statistically speaking the people who hive a fuck are a rounding error.
They really are. This is why I prefer the volume approach. I might not accept any of the ideas it spits out, but it often guides me in a direction I was not considering.
I know that most people don't care, but my online presence is a search query for interesting people, so I care about what I put into it.
Nitpicking here in a way that I would usually avoid, but it is relevant to the conversation being had and it seems like you might appreciate the information... "Readability" would be the more correct word to use here instead of "legibility".
Legibility is close enough for me to know what you mean based on the context, but it really applies to the visual presentation and how easy something is to read at a symbolic level (whether someone's handwriting or font choice is good or bad impacts legibility, whether someone uses good grammar or not impacts readability).
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.
> To illustrate, in the special case that GPT had a bunch of possible tokens that it judged equally probable, you could simply choose whichever token maximized g [a cryptographic function]. The choice would look uniformly random to someone who didn’t know the key, but someone who did know the key could later sum g over all n-grams and see that it was anomalously large. The general case, where the token probabilities can all be different, is a little more technical, but the basic idea is similar.
(1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and
(2) The actual behavior that is approximately what you describe already usually involves use of PRNG (with a user or harness supplied seed), not a true RNG; the change to do watermarking isn't going from RNG to PRNG, it involves adding an additional set of constraints on token generation on top of the existing ones, which inherently compromises quality.
Same reason that you don't just replace your rand() implementation with "return 4; // chosen by fair dice roll". If you need randomness for whatever reason, biasing the generator is compromising quality.
In case of LLMs, you can look at it from high and low level.
At low level - if you could do with less randomness, you can always lower temperature. You usually keep it (or for SOTA providers' chat UI, they keep it) at a level where it's about right level - high enough to allow for more creative leaps and interpretations, low enough that it doesn't go off into crazy land after the third paragraph.
At high level - creativity is driven by randomness. If you had an author (fiction or nonfiction) you like for their both broad and deep range of insightful thoughts, would you be happy if they suddenly developed an acute porn obsession and uncontrollably added lewd subtext to every other sentence? Still creative, still deep, but now with that one strong attractor that biases their every thought in a single direction? Would you trust/enjoy their output as much as you did before?
That, slightly exaggerating to make it more obvious, is what "loss of quality" means here.
(1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.
This is a consequence of how they are trained.
> LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.
LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.
You know you can just try it and see on any inference system thst has this knob, right?
Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.
Not all values of "equally good" are equally good.
Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.
Never mind metaphors and similes, which are even more tightly constrained.
LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class journalism.
LLMs are still somewhere between the first two, and nowhere close to approaching the third.
> Not all values of "equally good" are equally good.
Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.
We already know that a non-zero temperature improves quality though with current models (particularly with creative writing). The assumption that always picking the 'best' token results in the 'best' output is not the current reality.
And if you are already intentionally putting in randomness, I can imagine that it would be possible to seed the randomness in a way that is detectable but results in the same quality.
This is obviously not true for queries where temp = 0, but at temp = 0 then it becomes easier to identify anyway. I assume this technique implies some level of temperature.
It's not just swapping synonyms. The way llms work is by predicting the likelihood of the next token. It's inherently probabilistic. Choices are made based on weighted random number generation, based on those probabilities. Changing how you generate the random numbers doesn't degrade the output.
wow even on HN people have no clue how any of that works? all LLM generation has some inherent randomness to it, if you replace part of that randomness to be deterministically random the result of the generation with the fingerprint and without it, is INDISTINGUISHABLE. This has absolutely nothing to do with "synonym swapping". Its also again people not understanding how anything works missing the real concern, which is that nobody can ever tell if it isn't also secretly a fingerprint with the user and session id.
I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want.
On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense.
Even good human writers would probably struggle with this constraint. It would be like someone interrupting your writing to tell you the next word MUST be such-and-such, and then you have to try and make it work as best you can first try, without going back to edit. The result would probably be a little clunky. (Maybe it’s impressive LLMs write as well as they do.)
This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.
>There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.
You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of tokens at a time" is poetic sounding jibberish though. What's actually happening is one distribution output for the next token computed from a hidden state that implicitly encodes where the text headed.
Your claim that if an LLM does happen to pick a token "th" instead of "tw", then the LLM isn't stuck with that decision is entirely false for autoregressive LLMs which is what all of the frontier models are. Whatever an LLM picks as its output token is final, it has no ability to undo that token selection and it must continue on the basis of that choice. It can't go back on that decision and revise the output.
If you're interested in this, Anthropic has a summary of a very technical paper on this topic that mostly deals with this issue with respect to poetry:
So we train a second copy of Claude to work backwards—reconstruct the original activation from the text explanation. We consider an explanation to be good if it leads to an accurate reconstruction. We then train Claude to produce better explanations according to this definition using standard AI training techniques.
Incentives to train a pathological liar. There's no baseline so can only catch out the worst of the lies/errors. Anything (including fabrications) that passes our filters is reinforced?
What are you even talking about? If there was a "baseline" for how to decode activations into text we wouldn't be having this conversation.
The choice is between "this reconstruction sucks" and "no reconstruction", and we're only now beginning to learn how to make those reconstructions suck less.
No, they really do one at a time. You're incorrect on that.
Mathematically, a long chain of conditional probabilities is equivalent to a single probability over the whole range. But computationally, for that to work out, the computation for the first probability needs to somehow consider all the downstream probabilities depending on it, which obviously isn't how autoregressive language models work. They can pack in as much downstream computation as their neural architecture allows for, which is quite a lot.
Suppose in some context you have three equally plausible conpletions after "Be": "tween a rock and a hard place", "twixed he stood there" and "lieve he can fly". To model this probability distribution of the whole sentence, the next token "tw" needs to appear at 2/3 probability and "lie" at 1/3. After "tw" would be a 1/2 chance of "ix" and a 1/2 chance of "een"; after "lie" would be a 100% chance of "ve " and in any case the rest of the sentence after that would be 100%.
The model needs to somehow "think ahead" to know those are the possible completions. For example if "lieve he can swim like a dolphin" was another equally plausible completion, that first token would need to be 50/50 instead of 67/33. So the computation of the first token somehow needs to encode the fact that the guy thinks he can fly but not swim, even though it doesn't become relevant in the output until several tokens later.
In practice this probably happens to some degree but definitely doesn't happen perfectly. To perfectly model the first token's probability distribution, it would have to include knowledge of the entire distribution of all possible outputs, which is just not happening. So it approximates. Surprisingly, the approximation is good enough to produce language.
You can see this breaking down in the seahorse emoji incident from last year. When you ask the model if there's a seahorse emoji, it first completes "Yes," as if a few tokens later it's about to produce a seahorse emoji. But when it actually gets to the token that would produce a seahorse emoji, it can't because there isn't one. But it's already outputted "Yes, the seahorse emoji is" and can't just go back and change that to "No, there's no seahorse emoji." Some models would try a few times and then say there isn't one or a system error seems to be making them unable to produce one, other models (including then-current ChatGPT) would loop forever with ensuing hilarity.
But on some level there is uncertainty, right? Even if it’s not token-specific but at the word- or phrase-level? Otherwise what does the temperature setting do? Or has architecture changed significantly in the background?
This was true in the ChatGPT era. Now we're in a world with reasoning tokens, where a model can thoroughly plan out the response it wants to make. If anything, it makes the style worse.
Yes, models can reason and plan, which helps them write more coherently. But when they write the final output, it’s still a single generation. It would be like letting a human make notes and write an outline, but not let them use the backspace once they start typing their response.
Presumably you could use the same reasoning trace, run multiple generations, and get different outputs (if the temperature is >0).
But now I’m interested in playing more with Cowork or Claude Code/Codex for prose writing to see if the set of tools there affects outputs at all. I guess you might need a more custom “writing” harness.
I also really liked the quote "Your existence is not impossible, but it's also not very likely" from the Night Vale podcast.
I feel like the existence of good writing is also not impossible but not very likely, and so of course LLM can only write mediocrity, even when taught only on great writing.
autoregressive generation doesn’t mean the model is myopic. the next-token distribution can already reflect a longer horizon plan for the output sequence.
Sure, but mightn’t there be several plausible long horizon plans?
Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was:
—
*Long journey tracks*
Clinic — “The Return of Evil Bill”
Guided by Voices — not really, wrong band
Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic.
—
So at some point there, the next token produced was “Guided” or “Guide” or whatever, and then because it can’t go back, it had to correct itself after the fact.
Reasoning/CoT have helped a lot, but I feel like small versions of this still happen all the time.
Would be fun to run an LLM on fake output from itself. Like just force the first N tokens to say the beginning of something really stupid, and then see how it finishes the sentence. "You're absolutely right! Human feces is actually the most effective engine coolant because $<completion>"
> Even good human writers would probably struggle with this constraint.
But that would be a fun writing exercise, I think. Thoroughly in the oulipo wheelhouse.
Maybe generate a Markov chain table over all of Project Gutenberg and then say every 10th word is whatever the Markov Chain thinks it should be at that point?
Or every Nth word has a P% possibility to be constrained by the chain? Optionally with the possibility building for each skipped word to guarantee it happens at some point. Bonus with this approach is that the human can't game the words leading up to the constraint because you don't know when it will happen.
It's a bit like trying to finish a sentence when you're really stoned... you vaguely remember the preceding couple of words you've said but don't really know how you got there and now you're wandering in the forest trying to stumble on coherency.
Well, I suppose it's nearly the opposite of that experience, upon further review. But for some reason, that's where my head jumped.
Humans already do struggle with this constraint. Good examples are JRR Martin, Tolkien, and Rothfuss. You cant describe the struggle of picking the next word and then act like humans don't sit at the table struggling to pick the next word.
Human writers do better because they can think, and adjust, based on context.
They are also usually worse (which is often better!) because they are usually lazy and don’t want to spend effort they do not have too, to accomplish their goals.
Today I learned a new word, "Oulipo". Interesting.
But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something?
I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.
Yeah this is my main issue with the argument.
He acknowledges in the article that LLMs are already non-deterministic, but he doesn’t seem to actually understand that.
I think the article is wrong on this but it's more subtle than that. Probability distributions have a peak; there is still a token with a peak probability. What's interesting about these techniques is that token by token it can actually make the peak token even more probable. A distribution doesn't have to be "flattened" to leave a watermark - it can be "amplified" and made "more peaky".
The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output.
Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.
No one uses a pure random function over the whole probability distribution described by the LLM's output. For example, there is exactly 0 probability that the chosen next token by any common API or even local LLM runner would be a token whose final value is "0.0001" if there exist at least K tokens whose value exceeds "0.7".
Also, as long as the same sampling strategy is used during training as the one used during inference, then the LLM will actually do much better with the biased sampling strategy than it would with a fair one - because that is what it was trained to optimize.
>No one uses a pure random function over the whole probability distribution described by the LLM's output.
So what? By definition with this system the LLM will chose tokens it otherwise would not, purely for watermarking reasons. Yes this token may have had a decent likelihood of being chosen anyway, but it wouldn't have been chosen and now it was for reasons nothing to do with output quality.
I'm not sure what your last paragraph is trying to say. The blue/green list system changes what output the LLM would otherwise produce. You can't train it to produce watermarked output with this system. If you tried to, there would be no delta between trained output and watermarked output for you to be able to detect.
It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.
We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.
Fundamentally the article is railing against --temp > 0.0. He doesn't know what he's talking about.
>It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.
You can't as a user tell by how much the quality of the output was degraded. True.
>We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.
I'm not saying it wasn't random and now it is. I know how these things work. I said that the quality of the system is in the quality of the probabilities. That quality is being degraded.
This is true and the author seems to not understand the problems with greedy (top 1) decoding or the fact that watermarking affects only high entropy tokens.
But the published watermarking methods still have a slight negative effect on perplexity, so there is something more to it.
That's missing the point. It's the distribution that's the "best", not the tokens. Then Anthropic comes in and makes the distribution something other than the best. The only saving grace is that Anthropic says it's not that bad.
Even so, I don't think it will stop here. Once this is in place, the next step is to put more and more identification into the AI generated content; might as well pack it in, it's not that bad, and if it is they won't admit it. There's no way for anyone to check. And your argument will still be technically correct but missing the point.
How do you know you picked the singular “best” set of tokens in your comment here?
Could it have been equal or better with slight variations in wording?
The slipper slop argument is too lazy to address directly. Argue A is bad because A, not because A might become B and you’ve got good arguments against B.
There's no difference between those two things. The distribution that matters is the distribution of tokens that are picked not the distribution of tokens the LLM model passed to the selector.
The distribution of token "ple" being the same on average, but lower after "crum" and higher after "cou", is not no difference. It's irrelevant that the single-token distribution is unchanged if the joint distribution is different.
That comment merely says quality must be compromised. It doesn’t make it clear why that must be true. Empirical study seems to say that quality is not compromised, and looking at various proposed schemes, it seems intuitively true.
Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
It seems to me like he started out mad and looked to justify it.
I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists.
If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experiment harness and rated a few hundred blinded examples and found a measurable difference.
But getting this upset in advance of any demonstrated problem? It really seems to me like the point isn't the point
> It seems to me like he started out mad and looked to justify it.
Yes, but that's neither surprising nor a reason to dismiss the anger. People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible. They're angry -- and Gruber acknowledges that factor too -- because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.
It might be another instance of consequentialism vs. honor ethics. Many consequentialists don't seem to understand that something that doesn't have demonstrable consequences can still have moral implications.
Yes but thats a thing that degrades something in a catastrophic way, as in I can use the thing one day, and not the next.
A different randomisation system on something that is a text generator which is designed to be unperceptable sounds like the people who are annoyed at FLAC vs MP3[1]
Done right you won't know the difference, done badly and you will.
Even if honor were real, LLMs do not have honor, and people using LLMs to write without disclosing that fact (or indeed at all, to some purists) do not have honor either.
> because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.
It's LLM output! It's not your domain, it's the LLM owner's!
It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.
The question here is not to what product management advice to the Gemini team. The discussion here is whether the watermarking is noticeable.
The easy thing to do here would be to have 1000 questions, randomly assigning one half to an LLM with a watermark, and the other half without. Then show people pairs and say, "Which one seems watermarked?" (Or, "Which text seems more natural" or "Which is a better answer" or something like that.) If they come out equal, the watermark really is indiscernible, at least to most people.
I assume they are just passing off AI prose as their own and don't want anyone to be able to tell. Which is surprising for someone who's been blogging for a thousand years. But I don't really see any other reason for this amount of heat and FUD.
I think in this case it doesn't help that there are multiple watermarking schemes, and the easiest for people to understand is the red/green scheme by Kirchenbauer et al. (https://arxiv.org/pdf/2301.10226), which does technically distort the logits (but I'd argue only in cases where you wouldn't notice it anyway).
I wasn't aware of this gumbel softmax scheme, it seems you're referring to https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-o... ? That's really clever as it doesn't even distort the logits, basically cryptographically indistinguishable from a "real" random sample unless you have the key.
The actual scheme Claude uses seems to be neither of those two though, they say it is SynthId-text which seems to be tournament sampling based.
If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone else, it must cross over the 50% threshold. This is the problem.
> the probability of the detection tools flagging completely human generated text as AI generated is non-trivial
How does that follow? AI-generated text is already not a perfect emulation of human writing. There's lots of room to affect it laterally without changing the level of quality.
As I understand it, LLMs with temperature >0 can select from many possible outputs. All they're doing is limiting the possible outputs to ones that contain this pattern. I don't see any reason why the quality of that subset should be lower than average. The very best outputs will likely be eliminated, but so will the very worst.
To sample from the probability distribution you already need random numbers.
If you get your random numbers from a cryptographic PRNG, then to notice the difference between that and 'real' random numbers even in theory, means you need to break the cryptography. In practice, your gut feeling about how good some text is won't break modern cryptography.
Yeah. And the detector can’t even tell you “X% chance this is watermarked” because it doesn’t know the input distribution. It can only tell you “Y% chance that an unwatermakred text would score this high” and how many history professors understand Bayes rule well enough to understand the distinction?
Worse, what will academic institutions decide is the threshold for detecting AI generated work. If you have a false positive how do you prove it was a false positive or we all just trust the watermark detector over the student saying "I swear I did it all by my self"
I don't think that's true. I think it's a binary 0% or near 100% probability of a watermark having been detected; the more changes to the text having been made after the text was output by the LLM and the less leeway the LLM had for probable word choices, the longer the passage necessary to see it.
The "problem" is that seeing the watermark doesn't mean that the person claiming to be the author didn't make extensive changes to the output of the LLM, or that the LLM wasn't simply the final editor of something that the author had put a lot of work into.
> Cognitive surrender.
I don't know what this means. It's just drama. Don't let the LLM write for you and this is not a worry. I'm not worried about the poetry of LLM output being subtly adulterated.
No, watermark detection is not binary, you get a real number. You decide on a threshold when looking for the watermark. This is the problem - by random chance, some human text will be detected as watermarked. You can turn the detection threshold up until it guarantees <0.001 false positive rate at the expense of higher false negatives, but seems inevitable that someone gets wrongly flagged.
I think he’s still generally good on business, UX, and hardware design. That’s all subjective and taste I suppose, but his taste works for me.
On deeper tech stuff, like this utterly nonsensical misunderstanding of watermarks… yeah, classic case of a guy who is smart, and has lost the ability to realize when they’re not knowledgeable in a domain.
> The very fact that there is generally no "best next token" with 100% certainty
Indeed.
It's frankly bizarre to see the assumption to the contrary being made by someone who's been passionately blogging by hand for years, who also happens to be responsible for the notoriously vague, humanistic, DWIMmy Markdown standard.
Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles?
Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News the next day calling for an invasion. As a former reader of Daring Fireball, it’s just sad to see.
Yes, I decided to stop reading his blog relatively recently after some extremely hot takes on EU policy. I don't feel his thoughts on the matter are particularly well-thought-out, and I feel like he's just stanning for Apple from his priors rather than from any grounding in reality.
I dunno, I guess that's what you should expect from Gruber but these EU-bashing articles lowered the enjoyment I got from his blog underneath the bar for me.
Gruber went from the naively wrong claim that it would insert secret hidden characters (which would be trivial to remove, obviously), to quickly writing a giant essay as if he's an expert on LLMs. Like you said, he is strangely fixated on the EU, and is certain any EU rule is the worst thing in the universe, and this whole piece seems motivated by that guiding force.
Further he later compares Gemini to Anthropic models, saying the latter "writes better", emptily ascribing this to the synthid stuff. I think he heard that Anthropic currently has superior models, but it certainly isn't because they "write better", and if anything Opus 5 now is virtually unintelligible, before the fingerprinting.
The fingerprinting stuff sounds weird. If the EU wants it, it should be limited to the EU, and Anthropic is fully capable of doing that but clearly saw value in recognizing their own output. Is it going to destroy the quality of the output? We'll have to see, and this anti-EU piece, predicated on utter ignorance of the field, is not convincing.
What a bunch of entitled whining. How is the system to know that it's just a private conversation that won't be used in some fraudulent way? Abuse is currently rampant, yes please let's find a way to mark LLM output. The thing I'm worried about is giving the providers the power to claim provenance. Even ignoring the privacy issues, the operational hassle of having to check N providers makes these approaches at best limited. I want to see research into providing a shared public or ideally self-hostable oracle that uses some standardized method for watermark detection. Similar to asymmetric crypto where users can't reasonably find out the secret part but can do something useful with it nonetheless.
> Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?
No and no. I am not sure I agree with his point but I know he is not ill-informed on either of these points, because I mentioned them to him a couple of days ago.
It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point.
The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text.
The point he is making is consistent with this, isn’t it? Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure. These are ethically distinct approaches, and since he disagrees with the EU objective he comes down on one side I guess.
Me, I don’t care about the hypothetical enough.
Not least because I think Claude writes depressingly badly and I doubt any steganographic change will enrage me less.
The point he is making is not consistent with understanding how temperature influences LLM text generation, no.
He repeatedly states that choosing "the best word" is the most important thing to him. I don't know how you reconcile that with creativity itself, let alone probabilistic sampling.
Yes, naively understood in his sense 'best' word means that you pick the word with the maximum score, instead of sampling from the distribution.
That doesn't actually give you the 'best' text in any human sense of the word. Just like playing the 'best' move in Poker without sampling leads you to lose a lot of money.
I mean creativity there in the LLM sense (temperature driving more creative solutions), not the human sense, and in the context of its consistency, configurability and being amenable to analysis. The watermarking approach makes that non-reproducible, yes? Because Anthropic can and will change it as they see fit.
>Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure
This framing does not make sense to me. What do you mean by "influenced and analysed"? How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing? What makes the unadulterated randomness "driving creativity" but a different random choice uncreative?
No, you are jumping to conclusions about how watermarking works. This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever. Intuitively this may be true or false depending on your personal prior but you’d need to show it mathematically. The overall token distribution shouldn’t change and the frequency at which you see the word “load-bearing” will remain the same.
Gruber has long been a talented, excellent writer. I hugely doubt he uses AI at all, nor does he plan to.
His first take on this situation was cutely naive, thinking they were going to inject secret hidden unicode characters. But ultimately he has a massive hate on for the EU -- they were mean to Apple once -- and it comes out in any topic that overlaps.
This is not it, no. He is not using AI and it is not I think remotely in his nature to surrender that control. He is engaging with this on principle. Again I am not sure I agree with him, but then it’s a hypothetical because I am not going to get an LLM to write for me either.
That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.
Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal?
It seems fully logical to me that someone who writes for a living (who, as it happens, developed the very markup language LLMs use for everything) should be invested in understanding the automatic plagiarism and word calculating machine from an intellectually honest position.
I personally am pretty severely big-two-AI-firms, increasingly anti-big-tech, but I am learning and researching uses of LLMs because for myself I really need to understand how to use them in an intellectually and (as far as is possible) ethically sound way. Learning because as a boring old freelance programmer I have to; foolish to pretend otherwise.
So I completely understand his position — that the AI industry is hot air and crooked and scammy and weird, and some of the people involved genuinely rather dark-sided, but the technology exists and if it hints at threatening your livelihood, you need to understand it.
From reading his work for the best part of twenty years or so (and emailing him intermittently over that time) it would seem to me that he's a lot less bearish on the tech industry than me, and a lot less fond of the EU than I am; he's more optimistic than I am. But he writes because he has to write. I should think that would make him highly invested in understanding what LLMs do.
Well, it cant be that he is super worried on behalf of people who publish AI slop. That’s not a credible motivation. In fact, he complained a lot about the new ChatGPT app so I can’t believe your claim that he is not using AI.
Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.
His complaint about the ChatGPT app is that it’s a shitty non-Mac-ish Mac app. Complaining about shitty non-Mac-ish Mac apps to people who hate shitty non-Mac-ish Mac apps is more or less how he became a full time writer.
What a truly bizarre article. Arguments about pre-existing randomness, temperature and whatnot aside, I simply cannot comprehend what the author here really thinks the "best word" is. There's no such thing. We humans fall on familiar patterns of writing ourselves, so we may forego something with a flourish in favor of a more commonly-used word unless we put in effort to be "special", which should be used sparingly. That is to say, human writers are likely to choose a "worse" word in far more than the supposed 51% of cases, and that has no effect on the actual quality of writing in the end.
But even if there were such a thing as a truly "best word", for some context, what are the examples here? Mango vs pineapple? Gray vs overcast? In what case is one of these better, that AI would normally infer but would suddenly be "perverted" by SynthID? Do you think your emotional state and preferences are being evaluated if they aren't explicitly in memory? And if they are there, do you think that the generator will bypass those instructions in favor of the watermark instead of placing it somewhere you won't care? I just. Genuinely don't get it. There may be words that matter in specific contexts or to you as a reader, so you should bloody well put them there.
Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude.
You know, back in the era when proofreaders were human, I never one met a proofreader who rewrote my text afresh, rather than annotating the text with a pen.
It's still possible to use Claude to proofread - highlight grammatical, flow, structure, logic errors and make simple suggestions for you to pick and choose or adapt as you wish. No watermarking will flag your text. No flaw accusations of LLM authorship will haunt you. All will be fine.
But if you want an LLM to rewrite your text, that's (a) not proofreading, and (b) should be flagged as LLM generated ... because it is.
This is a fact and this is generally not a problem. Customer support guy Joe did not write that email to you with a refund: someone else did it and Joe did pick the template. Alice did not write that post card to Bob, someone else did and she just googled some nice text. We deal with a lot of content that wasn’t written by the person who signed it. That content, when written by LLM, may indeed contain watermarks and nobody will care about the choice of words, because only the meaning matters in such communications.
People pay too much attention to authenticity here, which is no more than a demonstration of an effort. LLM can and should write scientific articles because the real effort is in directing research, not summarizing it. LLMs can and should write news, because it is cheap and efficient, and real reporting is in discovery. LLM can and should write fiction and make movies, because there is no reason why creators of various junk should earn their money easily. LLMs do not replace real talent. They just emphasize for an average person how easily replaceable they are. And that‘s ok. Creative industry is a blue collar job now.
If it's not a problem, then there should be no issue with not concealing the fact, no? Lying about things one considers inconsequential is a useful signal about one's willingness to lie with little benefit.
I'm not sure the quoted statement is true. Proofreading like "point to problems in the text", if you fix the problems yourself and don't copy-paste the solutions given to you, should still be safe, shouldn't it? So, human-written text should not be falsely flagged if you use LLM for proofreading.
And if you copy-paste the answers from LLM, I think it's only fair the end result gets flagged. You're not writing it yourself.
> And if you copy-paste the answers from LLF, I think it's only fair the end result gets flagged. You're not writing it yourself.
I wonder if it would even get flagged in that case, because wouldn't the probability distribution of a token when the LLM is suggesting an edit to your writing be different than the distribution of that token once it is in the context of the text it's editing?
I don't even ask LLMs to go that far. Tell me if I've made a spelling, punctuation, or grammatical error, period. Don't rewrite a thing, because LLMs suck at that.
The question is, when is the "pro writer" version coming that lets you control this behaviour but costs more? Like night follows day, this will happen.
They will need to dodge around the EU requirements but it will probably just come down to an alternative method to watermark or a contractual assurance you won't mis-represent the source of the text.
It’s a strawman argument because if the LLM is really just “proofreading” for you, there will be little or no watermarked text in your writing. Not enough to trip the watermark detector.
It is quite just, if you think about it. Human works are copyrighted and protected at the moment of creation. All rights reserved. Yet, LLM outputs are uncopyrightable. Therefore, if Claude or any AI has processed my copyrighted work, the end result is uncopyrightable and in the Public Domain. The public has a right to know: is this a human copyrighted work, an LLM PD work, or is the human falsely claiming authorship in order to retain copyright?
A point of confusion for me, however: is every watermark unique? Is every algorithm for watermarking going to vary amongst models and amongst model versions? Will each model publisher keep this watermarking as a trade secret, that they alone can detect? If so, this can't scale! How do you detect "JoeBob 4.3 LLM" output? By querying every single model's watermark-detector? And if they all work by re-running the model and using tokens anew? That is extraordinarily wasteful.
If a watermark is not self-evident, or universally detectable, then it is no good. Take, for example, US currency. The security measures are published and well known. Any count-out room in retail has a big poster indicating how you can detect authentic US bills. Nobody has to accept non-US currency in the US, and so the only authenticity you need to worry about is your US bills alone. LLM watermarking has none of this in common. Currently sounding like a shitshow, if you ask me.
That's an intriguing twist, isn't it? It could lead to a tug-of-war.
Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now?
If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works?
This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.
Some camera pointing at you while you work. Guessing the camera needs to be watermarked itself, which can be done by adding some watermark on the chip level (each camera will have different watermark), manufacturer can confirm the watermark.
The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.
As for it being "uncopyrightable" if it were the output of an LLM, I think computer people are making a very aspie interpretation of a single decision. I think it's more that the LLM (and thereby its owners) cannot itself hold a copyright on its output, that output has to be touched by a person before it is copyrightable. A particular view from the top of a mountain can't be copyrighted, for example, but a photograph of that view can be.
I'm not sure it at all precludes a "robosigning"* sort of situation, where machines generate output, hired temps sign and claim that output, and immediately sign it over to the people who hired them (as a work-for-hire.) Copyright is stupid, artificial law, not logical.
> The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.
For the watermark to be detectable, the text needs to be like 75% AI generated.
If you have an LLM “touch” one section of the article, it’s not gonna be detectable.
As I understand it, the current watermarking methods rely on a secret key, making the detection schemes a black box to anyone not in possession of the key. This means organizations like Anthropic are free to make any claim about authorship they want, true or not, and no one can call them on it.
> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.
Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best.
This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.
This seems like a non-issue, or maybe I have the wrong expectations about writing. You write a text, ask Claude to proof-read it, but then you wholesale just copy Claudes output and use that as the final text? Wouldn't you review the changes it suggests and only take those you agree with, there by completely bypassing the watermarking?
Alternatively, you ask Claude to write the whole thing and proof read it yourself. In that case I'd like to know how much you'd need to change to break the watermarking, i.e. how much of a text would you need to change for it to be considered your work and not that of Claude?
Gruber was a good voice in the industry but this article misses the mark in a lot of ways.
A company the size of Anthropic would not voluntarily jeopardize their massive valuation if they didn’t feel the resulting output would maintain a similar level of quality as before. Is there a similar worry that their system prompt, which is injected at the start of every conversation also influences token generation in an artificial way?
If regulation will ruin Claude as a product, market forces will fill the void. There are also a ton of open weight models to choose from. It’s going to be okay.
Gruber is not much of a details man - he helped invent Markdown (to be lauded) but ghosted its standardisation. I would be fascinated to hear Prod John MacFarlane of UC Berkeley's opinion on it all given he was heavily involved in the push to get Markdown standardised.
LLM inference already isn't deterministic, the watermarking technique only limits the space of possible random seeds. There is no reason to believe that this subset of seeds somehow produces lower quality output.
This article feels slightly incoherent. You want high quality precise writing and to use an LLM to generate it? Feels like those are diametrically opposed
Exactly. The watermark is proportional to how much text is AI generated. Either the AI really just “fixed some typos” (not enough AI content to hide a watermark) or the AI did most of the writing (enough AI content to hide a watermark).
This feels like the inevitable outcome of a STEM-only education system. Now people think there's a mathematical formula for picking the "best" words, instead of having to be thoughtful and creative.
I was initially surprised that Gruber was so invested in the "quality" of AI-generated text, which in my mind is an oxymoron. But really, Gruber's interest here is with the EU. This forms part of his ongoing attacks on the EU, all because they have been forcing Apple to align with regulations.
Can we install random unapproved apps on our iPhones yet, or is Apple aiming to just be fined a trillion dollars because they make more than that from the 30% cut?
I commented it last time the post about Claude watermarking went viral and I'm going to say the same thing again:
"I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver.
There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike.
As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026."
Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous).
And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs.
Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.
>in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code.
I wonder if this is why Opus 5 keeps writing excessively long comments, even though I keep instructing it not to (both in chat, CLAUDE.md, and in its memories)
LLMs are already perversions of writing, so what else is new. Oh no, the over-long circumlocution generated by three autocorrects in a trenchcoat might be slightly longer because of this and maybe people will start noticing the subtle rhythms of vaguely peculiar word choices as yet another cue that you are wasting their time with machine-generated wordslop, what a terrible fate. Your long rambling walls of machine-waffling might be 37.05% longer than they need to be instead of the mere 36.58% longer they are now.
I moved to Sol for my writing and it is so so much better. But it makes more mistakes. I think they have different ideas of product but it seems OpenAI is going to follow Anthropic’s lead over the next year. I think I am going to put more effort into my writing skills to remove myself from this awful situation
> But only Anthropic will be able to determine if text was seemingly generated by Claude, and Anthropic will only be able to detect the watermarks that are applied by Claude. Claude can’t detect the hidden watermark signals generated by, say, Gemini, and Gemini can’t detect the hidden watermark signals created by Claude, because each implementation is predicated on secret keys held only by the LLM provider
Well, akshwally...
> Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027
The issue here isn't (just) adulteration, it's that watermarking in general is unworkable.
If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text.
Which is clearly ridiculous.
And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial.
That's not even getting into the legal complexities of businesses running open source models without watermarking locally.
> Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027
If we wish to use “the very best, most precise words at every single decision point” then I will note that “writing” is not the best word for what LLMs are doing. A more precise synonym might be “generating”, in which case this essay becomes “Watermarking is a perversion of text generation” - which is true, albeit somewhat trivial.
I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value.
Everyone who invests in AI companies wants to see the value of their investment increase.
I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that path.
I think you're right that it's actually to do with attribution (i.e. They're not just watermarking your output as claude generated, but watermarking it as claude generated _by claude user id 73684_). I think it's more to do with the growing militarization of the internet. Just another brick in the wall of enter your phone number to create an account, send your id to prove your age, smile for the flock cameras, etc. Allowing the plebs to have privacy and anonymity is not allowed anymore.
I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms.
Seems unlikely unless Anthropic invented a time machine, given that the phenomenon predated Claude 1 by two years, and their stated introduction of watermarking (August 2) by 5.
I see how that was confusingly written. I'm not suggesting Anthropic are somehow retroactively causing it, just wondering if it could create a similar effect.
Less unlikely, but I would still suggest that it is somewhat unlikely for any recent LLM to place any significant probably on, e.g. "disappointment" instead of "failure" as the appropriate token (or series thereof) after "kidney" or any similar example.
It seems fine. I use an LLM to argue with me prior to posting blog posts so that I don't post obvious incorrectness, but the UX element to it is that it constructs notes about various sections of the text and we talk about those. There's no way for the generated text to enter the blog unless I copy-paste it and I'm not going to do that because the entire point is for me to write it.
At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.
Are there some narrow cases (like in outputting the input almost verbatim) where the probability for every best token is always much higher than second best, and in those cases would there be no watermark?
The fact that Gruber points to a "must-read" article about how the watermarking works, which, in turn, was very obviously entirely LLM-generated, says a lot about his lack of experience reading LLM output.
If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much.
NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as I believe it's quite relevant to the topic at hand.
I think writing is the killer use case for local LLMs.
We've had so many advancements in LLM samplers for improved text generation (off the top of my head: min-P, adaptive-P, XTC, DRY, p-less, Top-H, Top-n-Sigma, and so many more) but hosted LLM APIs only provide three basic knobs: temperature, top-k and top-p which are old as the mountains in LLM years at this point.
One thing that doesn't help the local LLM case is that all the popular VC backed local LLM wrappers also only support the same three ancient knobs because I suppose they're more preoccupied with their next fundraise than with keeping up with the advances in tech.
"possibly build tools to take a chunk of writing and try to remove your fingerprint from it, i.e, protect anonymity,"
The standard procedure to do this, is to chain translations to other languages and back. The message remains, but the wording will pick up some noise. --Dec 30, 2008.
Gruber has a ridiculous knee-jerk response to anything the EU does, so hardly a surprise he didn't come to the table with a sober facts-based response.
LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just generating pseudo-random results differently. Quality shouldn't be affected.
That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output.
If this didn't have a detectable effect on the quality of the token probability calculation, the watermark wouldn't be detectable. It may be a small degradation in the quality of the output relative to the neds of many users in many situations, but it's not zero. It's literally sometimes choosing different words that it otherwise would specifically for watermarking purposes.
An encrypted hard drive is EXACTLY uniformly distributed random bytes if you do not know the encryption key. No one would be able to tell the difference between a drive that is just purely random numbers or is actually filled with content. (Of course excluding the usually intentionally added readable header)
If this was not the case, the encryption would be broken, and most everyone agrees that good encryption does exist.
The "quality of the probability calculations" as you put it is 100% in this case and any less would be a huge deal (as in - breaks all of the internet).
So, now you just take those same random bytes and use them as the seed for your LLM token choices. The output has the _cryptographically_ proven exact same quality as if you were using a true RNG (which you likely weren't using anyways).
You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.
> choosing different words that it otherwise would
The "that is otherwise would" is carrying all the weight here. "Otherwise" is sampling from a distribution. You just sample from the same distribution but with a cryptographically secure, seeded RNG. https://en.wikipedia.org/wiki/Cryptographically_secure_pseud...
"Knowing the LLM distribution" seems to me like the only hard part because you don't know the context of any random snippet.
>You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.
I struggle to understand the relevance of that comment.
The blue/green token list biasing process literally does cause different tokens to be occasionally chosen. Not only that, but because a different token was chosen at one point, this changes the probabilities of all subsequent tokens, and the resulting later token stream every time it happens. If you had access to the token stream as it would have been, and the watermarked one by the end of the text they will be very noticeably different.
> Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot.
How is this supposed to work in an actual lawsuit? Will Anthropic offer some sort of tool / (paid?) webservice to check for watermarks using that "secret key", and a judge is supposed to just believe whatever that tool's verdict is? And then it takes the EU another 20 years to understand what a silly idea this was?
Yes, exactly, that is my understanding as well. Since the watermarking is based on a symmetric key by design (as to not be easily able to test and remove the watermarks), the providers will need to offer a watermark detection API.
I see even more problems with this. To check any text for those watermarks, it needs to be sent to dozens of AI companies to check, potentially paying them all for just determining whether it matches their watermark, and more concerningly sending all that mostly human-written, often high quality text like unpublished research or books, to AI companies that almost all proved to obtain training data through all kinds of dubious ways.
> The provider must mandate in their terms-of-service that users not remove the watermarking.
So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?
Security theater, and vague enough to be used as a weapon against who the government wishes.
Claude's writing was already easy to recognize. The fact that Anthropic complied without complaint makes me wonder if they already watermark their outputs and used the opportunity to create goodwill. Presumably they want to avoid training their new model on text generated by the previous model, so they have reasons to be able to recognize AI-generated text.
I often agree with John Gruber, but I think he’s lost the plot with this one.
The thing I don’t understand is why he seems to care so damned much about this subject--enough to write over 4,500 words on it!
John writes for a living. That’s his profession. He’s been writing for over 25 years now. When you’re that good at writing, and you care this much about your writing, you don’t allow an LLM to take over your job. I just can’t imagine that he’s in the market for LLMs and that literary excellence is his number one selection criterion.
So why is he so livid about it? It’s like being angry that wine is going to start coming in smaller bottles even though you don’t drink wine.
Even if he’s angry on behalf of other people, I don’t get it either. In my view, having LLMs write publishable content on your behalf is not a socially-acceptable use case, nor a professionally-acceptable one in most professions, even though people are abusing it for this purpose anyway. And besides, the models aren’t even all that good at it today. If you agree with that, then you certainly should not care if it’s using different phrasing than you otherwise might prefer if the meaning is similar enough.
I can't help but wonder if perhaps his hatred of EU technology regulation (which, admittedly, is mostly pretty dumb and is mainly just making life worse for users) is getting the better of him.
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.
Excessively precise word choice does not result in something that looks like content written by, or palatable to humans. It looks like you gave a high school 12 grade student a science paper and told them to apply a thesaurus to at least one word in every sentence and replace it with something else.
The main reason we don't see much quality degradation in LLM writing output is because they're already poor writers. This is the load bearing reason.
I was bulding a small interpreter and writing an article in ~markdown yesterday with Fable. And while it codes like a pro, it writes like a sixth grader.
Let's see how these watermarking stats hold up if/when llms start writing well.
There are many reasons to hate this watermarking but affecting the output quality isn’t one of them. The central argument he’s making is wrong. Switching out one RNG for another doesn’t make the results worse.
I dont see how there would be remotely enough entropy in most model outputs for this to be close to feasible with any kind of accuracy.
Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless.
Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.
I needed another reason to cancel my Claude sub. Thanks Anthropic!
This is akin to adding a giant watermark on things one would made with a free product "Made with XXX". Except you're paying $200/month for it, and there's no way to disable that watermark.
I don't disagree with EU regulations, but I strongly believe the onus should be on the content publisher, not the toolmaker. If the toolmaker watermarks whatever his tool produces, it opens a giant can of worms that cannot be closed. That means anything and everything you make with this tool is no longer fully yours, it contaminates everything and makes your work traceable. Who wants that?
I was already annoyed by the fact that Claude marked everything it did on my repos under its own account (I didn't ask for any of this), but now everything is invisibly marked, even the code. Not that I care that my writing would be watermarked, since I'd rather write my stuff myself, but code? No thanks.
Meanwhile I'm running a DeepSeek V4 Flash or Pro, or a Qwen3.8, and it writes my code without a peep. Resulting repos are clean, just the way I want them. No 'Claude' account, no watermarking, nothing. I won't be looking back after having tried these new models. Whoever makes good models that don't broadcast their maker will get my business.
This watermarking will simply push people more towards Chinese models. Keep pushing in the wrong direction Anthropic. Doing this right before an IPO is a great idea.
I guess I understand the complaint, but LLMs are already crap at writing, IMNSHO. And, yes, maybe this will make them marginally more crap, but in my mind we're talking the difference between a 30% grade and a 29%.
Some here are arguing that mechanisms used by LLM providers already derail the goal of "the very best, most precise words at every single decision point", therefore the author is misguided.
The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.
I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.
Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.
I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.
If the author has preferences on their "own writing" that conflict with Anthropic's, then they should actually write it themselves rather than paying Anthropic to do it. Private companies don't owe you anything, even less so when they're beholden to laws in foreign jurisdictions.
>Assume that there is a sequence of tokens, such that it is considered the absolute best by the author
You can't assume that because if that was the case he'd already know what sentence to write, because that's what that means.
The notion of a best sentence requires a final cause, an end to write to. By their very nature that's not how LLMs work, so you can't 'degrade' them on that front. They can't lose a property they didn't have.
People are very upset, especially in the arts, that Anthropic is changing the text to watermark it, but isn't that missing the point a little bit? They're not changing _your_ text whose every word you've carefully chosen for the exact effect, they're changing text that they're generating, i.e. text you've already chosen to give up control over. LLMs can't understand emotional nuance anyway.
The phrasing of the announcement implying that phrasing and diction don't change the meaning of text is insultingly dismissive of the whole field of literature, and I can see why people might take it as an afront, but the actual technology shouldn't have a negative impact as far as I can see. It seems to me that this one is more of a PR problem than something with real-world impact.
And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.
This is the yet another embarrassing idea from the EU they came up with, if they could focus less on dumb things and more on providing real support to the development of technology maybe we could have more healthy competition that will eventually lead to better overall technology in the hands of everyone instead of acting like dumb chickens and not just being passively useless but also stopping the real labs from doing the real work into doing this bs
Watermarks are garbage because they may embed account id, IP address and deanonimize you. That's why we should be using open-weights LLM whenever possible.
I don’t disagree about open weights (though the enabling aspect there is actually open source inference, right?)
But it feels to me like you would need a hell of a lot of text to bury even a simple account ID. The nudges they are talking about are of the order of a handful of bits over several hundred words, I think?
I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this.
> The difference between watermarked and un-watermarked text will not be distinguishable to readers
Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here.
If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.
I think there are two separate requirements? One that if you post something like an AI video on the internet or anywhere else, you must label it as AI. And another one that AI providers must watermark their outputs.
If you get caught uploading watermarked media without the clear label, you're in big trouble, mister.
I don’t understand how this works for anything but prose. Is that the point? In any code or structured output, there just isn’t the flexibility, and depending on how the user requests the output be constrained there is even less (“answer only True or False”). So is it just chat responses? If I ask the API to tell me a story about Alice and Bob then it watermarks it, but when I ask it some implausibly constrained thing like write a story about Alice and Bob with each word starting in rotation with the letters alicebob, does it try to do so and hope there are roughly équiprobable tokens regularly?
I should have read that, it’s actually quite reasonable and I don’t really understand the objections in TFA having read it.
> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.
Doesn’t make sense at all in light of the actual approach, they’re just choosing a different RNG. It’s not like they’re corrupting it by flipping words.
Should add I don’t support the watermarking and requiring it is idiotic.
I don't understand Gruber's points either, I wonder if there is some fundamental technical misunderstanding. Does he think that the logits should be sampled from in a "pure" manner without introducing any other bias? Does he know that there's already a sampling temperature, and that most providers have probably moved on to sampling strategies other than top-k? Does he know that the word choices have already been altered irreversibly during RLHF which is how you get the obvious Claudism like "load bearing" and "seams"?
Perhaps it would be useful to publish examples of samples with/without watermark. I'd suspect that the variability from simply sampling repeated times would dwarf any semantic differences you'd detect with the watermark.
The author makes a lot of great points. I find it surprising that somebody who has such a nuanced appreciation for the subtleties of language would be in the crowd complaining about the watermarking policy. I expected this kind of complaints from mostly students interested in academic dishonesty, who generally don't have enough command over written language to notice the slight decrease in output quality.
Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:
"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"
A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:
"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."
Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:
"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."
EU is a capitalist union first and foremost. So their regulations try tobstraddle the thin line between "regulate to death and tell companies exactly how to do things" and "the companies can do whatever the hell they want".
Since most Americans are in the latter camp, anything that even hints at making companies responsivle for anything is viewed as being squarely the former.
Whereas most EU regs are "play nice, be reaponsible, behave like adults. If not, this can always turn ugly". Same here.
The same absolute morons who gave us cookie consent strike again.
I swear, one of those days I will get into politics just to fight those two things, and the cottage industry of batshit crazy lawyers that gave birth to those things.
gruber is really out of his element with ai commentary, I fully support the general skepticism but he's seemingly arguing against something he doesn't quite grasp
I'll be honest, who fucking cares? Why would you use AI to write for you and then complain that people know AI wrote the code?? If you know people wouldn't like it, why even try!?
Now for the human generated watermark. Timez to addd the speelling mistakes, decreaze the usegage of big words and proper gramicaly usuage. Wish I was joking.
> the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text.
It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.
What is even the point of watermarking AI slop supposed to be? All it does is provide people with the false implication that anything which is not watermarked must not be AI-generated.
I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.
I keep seeing an irritating misconception in this space, which is that the alternatives chosen by these algorithms are supposed to mean the same things as what they're displacing. That's not true, and not how LLM generation works. Complaints that two different choices don't mean the same thing miss the entire point.
"Anthropic's . . . Claude is a Perversion of Writing."
FITFY.
I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.
> because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.
... and the opposite is also true, sometimes it will increase the probability of choosing the "best" word choice. So watermarking makes the LLM quality better then? /s
Anthropic should just pull out of the EU at this point. Europeans who really want to use it can VPN, and if they did they wouldn’t be able to hide behind their various comical tech laws.
Nanny state nonsense indeed.
As a non American/non European resident all I see from the Europeans are 0 contributions to software progress at any large scale while they surely do a lot of crying and huffing and puffing and demanding. Lots of complaining and rule making but not a lot of creating is a bad look.
Clearly Anthropic thinks its more profitable to comply and have access to the European market, but I'm sure you know better than the people who brought it to a $2 trillion valuation.
I didn't say anything about profitability. Since Anthropic is all about the good of mankind etc, ostensibly profitability isn't their number one priority.
This (Anthropic's "watermark" stance, I mean) is so fundamentally ludicrous that I have assumed it is a (wholly insincere, but arguably pragmatic, at least from their perspective) attempt to deal with the EU and their latest misguided, ham-fisted attempt to solve a real-world problem by drenching the entire world with more regulatory slop[1].
The "watermark" can be trivially defeated, but may be enough to satisfy the letter of the law, and like many people here, I would argue that if you are letting Claude write for you, you've already accepted getting the literary equivalent of turd soup, so the harm is — or at least could be — fairly minuscule.
(FWIW I have a more favorable view than most people seem to of the EU's efforts to at least try tackle problems like this — but predictably, the bureaucratic "solutions" they come up with don't work, but do make things objectively worse)
Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly.
As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over.
It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.
Not telling someone you used AI is a perversion of writing. Also agree that an AI proofreader should not claim authorship, but in most other cases, the AI is not reading your mind, it's only watermarking its own usage, and we kind of need more of that.
I recently found there was no website covering a historical subject, with sites begging to be visited. The information was scattered and not presented in the form people would find most useful (a map).
A couple of days later it was up with the most comprehensive review of the available info summarised and referenced ready for a human researcher to explore.
This was only possible due to AI. It would have taken me weeks to chase it down and summarise it, so it would never have happened. Let's get off our high horse about AI writing.
My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ...
So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means.
Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
its lovely training data. no detection? add to training set -_-.
its also kind of laughable that somehow people are trying to prevent the outputs not to be altered. Asif you cannot manually paraphrase anything you can read. So the only solution would be, to make it utterly unreadable (which is not possible, it obviously defeats the purpose of the thing).
Not to mention local models ofcourse :-)
Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over the text and destroy the watermark. I predict an interesting cat and mouse game to develop.
I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated.
I wonder if this would be a good use for homeomorphic encryption. There might be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!
> Any stenographic system that you have the code for can be trivially defeated.
They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it.
Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're submitting to does the same check it says the same thing.
> it says the same thing
Reference needed? I think it remains to be proven whether those detectors can be considered deterministic.
Won't we just be able to fine tune OSS models to detect these patterns across providers? It will be cat-and-mouse but my bet is it converges to a central detector that isn't affiliated with any model provider.
> blindly trusting they won't train on any of that
being allowed to train on any data that you can legally obtain ought to be a right for anyone.
After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.
> being allowed to train on any data that you can legally obtain ought to be a right for anyone.
I have the opposit viewpoint to the extreme. They shouldn't be allowed to even read that data until they are very clear about what they will or not do with it.
Can they publish it? Can they store it? Can they use the information in it on prediction markets? Etc.
Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.
> Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.
Citation needed. This is sounding tautological.
It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.
It's exactly the same problem as with humans, though.
It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).
It's not exactly the same problem, in that you can parallelize usage of an LLM and copy it over to another computer, but cannot do the same things with a brain. Put it another way, humans do not have the processing power needed to answer hundreds of millions of queries per day, while LLMs do.
It’s not because humans are rate limited which is the entire point.
At the risk of stating the obvious, there are a lot of legal rights that are human-specific (voting, holding office, filling lawsuits, etc.). It's not at all obvious why you think that you as a human being legally allowed to learn from something implies that it should be legal to train an LLM on.
> The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.
But LLMs are replacing the original, just in different words.
And what does 'legally obtain' mean in this context? Copyrighted content is usually licensed for specific purposes. So if a license is given from training your LLM, then by all means do! But what if the license is 'for personal use'... ?
You are one person. The corporation is not. Scale matters
> You are one person. The corporation is not. Scale matters
Correct, if you violate it too often to count, you have to pay around less than ~2.5ct per violation.
So the lesson here is: Create a company to do torrenting professionally, and resell its values for higher prices. Then get sued and pay a dime on the dollar you made.
edit: Actually it's 2.5ct per violation.
Oh so if I use mickey mouse in a completely original production that doesn't replace the existing work by Walt Disney, you reckon they'll be fine with that?
Why would that be a legal right?
Why should we hand over even MORE power to the owner class?
In a fantasy world this could be possible yes.
Copyright (or any other such restriction on free use of information) creates power for owners by the simple fact that it turns information into something that can be owned.
Make it a right, then companies/universities will think twice before using said APIs. Instead of this grey area where we will never know.
We don't hand over more power to the owner class by making fewer things ownable.
Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short.
Having proof that content (especially images and video evidence) is unmodified (whether via Photoshop, Paint or a model) is far more valuable then having evidence that an image was manipulated or generated fully by a model (which still leaves other forms of manipulation), I feel the same goes for human authored vs generated text. Free to admit that using models to generate any kind of media whole-cloth is still unappealing to me and I still pay for commissioned artwork or make it with my limited abilities for what that's worth. Do like to (poorly) write my musings too and see UX as something were thoughtful contributors (like the opinionated, sometimes controversial, but certainly talented GNOME Gitlab contributors) can make a major impact.
Code can be beautiful, interesting and serve purpose beyond execution, of course, but for most people, in most cases, it does not in the same way as audiovisual content (not limited to art). Having code just to execute and resolve a problem can have value all in itself, the code being a means to an end whose quality, let us be honest, was barely a concern in most corporations long before LLMs.
Also have rarely (honestly never) before LLMs fully owned all parts of any code base, always relied in part on someone's prior effort in (Flutter/Dart mostly) packages, whereas when writing, drawing, etc. I have far more situations where I make something from scratch and everything there is only there because of my conscious decision. Even simple marketing mockups that, quality wise, any modern model would beat feel different when I was fully in control, where to place what, etc. Objectively worse (at my skill level), probably, but still never the same.
Knowing something was made from scratch by a human has value to me, beyond misinformation prevention. Knowing for a fact that LLMs were used instead of importing a library, using a template, or something similar that leads to expending similar amounts of effort, I don't see that being nearly as valuable. Heck, with all the importing and my experience back then vs now, I am spending more effort actually fully reading any LLM output in my code then I spent back then auditing Flutter/Dart packages. Then again, LLM output fails far more unpredictable then those messy packages that simply got Gradle to take down my system...
Happy to admit, I have been skeptical of watermarking LLM output being feasible for quite some time and having looked into SynthID Text and proposals being researched, I am convinced that it is challenging to impossible beyond the lowest common denominator and less important then proofing human authorship.
> "The exact words we choose when writing matter."
Then write your own damn text if you care about the exact wording so much
If the exact words we choose when writing matter so much, then why use a non-deterministic LLM that produces slightly different output on every run?
I mean this is the thing that really comes off hard.
If you want precision and clarity of your writing, then you need to hand write it. Just like when you are optimising, its common to drop to a lower level language because the compiler doesn't express what you want. Sure its hard, but you know, thats kinda the point.
Even if you don't want that, the LLM is an average of the style it was trained to give out. Which is a homogenisation of the language to create a vague padding medium between a few generalised facts. (because a. it makes it less jarring when stuff is wrong, because its smeared over a higher amount of text and b. it looks more 'professional' because American business English is all guff and no meat)
Also yes, two isolated phrases may have subtly different meaning, frankly, the nuance is missed on most people. If you look at the interactions on here, at least 25% of the arguments are caused by people angrily reacting to the things _they_ thought the other person was saying, rather than what the actual person was saying.
So no its not a perversion, the LLM is, if you're gonna be picky about things.
The bigger problem is its effect on the way we use language. The amount of LLM generated or edited media is going to keep increasing and the media that people consume affects their own word choice.
What an absurd take on a very reasonable concern. As much as I hate “obviously AI” writing, I don’t understand why we have to handicap the tech and prevent it from improving. This manipulation of word choices virtually guarantees AI will always sound like a robot.
Have you compared pre- and post-watermarked text to make that assertion? And I'm not sure how writing your own text if you care deeply about the word choice is handicapping the technology?
I don't think there's been a single version of the major LLM providers that haven't handicapped the tech since the beginning by changing temperature/top-P/frequency penalty/etc. All of those stray from "highest probability" token selection. What's funny is it was done specifically to make it sound less like a robot/deterministic.
With all things going on among AI bros and the AI industry as a whole, are you really that surprised there is a widespread aversion against the tech?
LLMs are no more than pen and paper at this point. Especially for those who aren't trying to create slop. We all would want our pens to accurately reflect the strokes (well in this case thoughts) rather than adding tiny watermarks to identify that it is generated by a particular pen or a user.
Watermarking per model is just the start. The method is cheap enough to distinguish individual users.
That is an insane statement, LLMs generate swaths of text from almost nothing.
If they are adding so little value as to be as transparent as a pen and paper then why use one at all? Transcription doesn't need an LLM so that's not what you're taking about I assume.
LLMs do a whole lot more than writing your thoughts down. They write extra text. If you just want a pen and paper, use Notepad. Or better, a pen and paper
>LLMs are no more than pen and paper at this point.
Then use pen and paper. It is the same, you say, right?
One could also use butterflies to write ;) https://xkcd.com/378/
The problem is, that LLMs worked very well for me to improve my writing. Especially as I'm not a native speaker, it was a great way to improve the legibility of my work.
I want a tool that helps me improve my writing. A tool I can learn from. Not a tool that switches out "bananas" to "airplanes."
I'm was using Claude Opus and now Fabel extensively for editing my texts and I find the recent updates abysmal. Not sure if it's due to the Text Watermark.
Before Claude was great in sharpening the meaning in my writing, it's now close to unusable.
I'll be interested to see if people can actually pick out which text is watermarked and which isn't once they introduce it. It won't switch "bananas" to "airplanes". It'll switch "I really enjoy eating bananas" to "I love eating bananas" or similar
To that point, given a corpus of writing from person A and another from person B, I have no doubt it’s easy to train a classifier to determine who wrote what. In fact I believe law enforcement agencies already have these classifiers.
The only difference here is that Anthropic is actively trying to make the watermark undetectable.
They're not trying to make the watermark undetectable, that would defeat the point of a watermark. They're making it detectable, but not make the text obviously watermarked
Undetectable by a human reader. Come on, give the post a charitable reading.
This is an excellent use case. It made learning German much easier. I write what I think is correct, then get a fixed version.
When writing in English though, I use it more like a dictionary. If you want to write past a certain level, an LLM works better as a metaphor and idiom search engine.
I also like to ask it to generate 20 ways to say the same thing. It’s a great way to simplify or smoothen sentences without losing your voice.
Interesting, are you being literal about requesting 20 ways of saying the same thing? It seems pretty excessive and I'm actually impressed that asking an LLM to rewrite a statement 20 times yields output that isn't excessively redundant. Does the LLM do a pretty good job reading your mind, or do you still find yourself manually piecing together pieces from the 20 suggestions into a satisfactory sentence?
Yep, I literally ask it to say it in 20 different ways. Sometimes a sentence structure or a combination of words will just work better. The goal is usually to simplify a sentence without losing meaning.
I don't expect the LLM to read my mind. The unit of work is too small for intent to matter, and I'll just steer the next recommendations in a direction as needed.
Most of the suggestions are crap, but they can contain the seeds of a good sentence.
As a writer, Claude’s metaphors are trite and obvious 90% of the time. It also has a terrible penchant for an immediately recognizable emphatic voice that makes even the best outputs super cringe. But people do not notice and do not care. In our bubble and John Gruber’s bubble we really overestimate how much people must care. Truth is, we’re in this predicament because statistically speaking the people who hive a fuck are a rounding error.
They really are. This is why I prefer the volume approach. I might not accept any of the ideas it spits out, but it often guides me in a direction I was not considering.
I know that most people don't care, but my online presence is a search query for interesting people, so I care about what I put into it.
> improve the legibility of my work.
Nitpicking here in a way that I would usually avoid, but it is relevant to the conversation being had and it seems like you might appreciate the information... "Readability" would be the more correct word to use here instead of "legibility".
Legibility is close enough for me to know what you mean based on the context, but it really applies to the visual presentation and how easy something is to read at a symbolic level (whether someone's handwriting or font choice is good or bad impacts legibility, whether someone uses good grammar or not impacts readability).
> One could also use butterflies to write ;) https://xkcd.com/378/
This comparison is frankly absurd.
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.
The best discussion I've seen so far is from Scott Aaronson: https://scottaaronson.blog/?p=6823
> To illustrate, in the special case that GPT had a bunch of possible tokens that it judged equally probable, you could simply choose whichever token maximized g [a cryptographic function]. The choice would look uniformly random to someone who didn’t know the key, but someone who did know the key could later sum g over all n-grams and see that it was anomalously large. The general case, where the token probabilities can all be different, is a little more technical, but the basic idea is similar.
That's inaccurate in two ways:
(1) The behavior that is approximately what you describe is not "fundamental" (though it may not be something you can disable on some hosted providers), it is an option that is not fundamental (and with runtimes where you have full control can be either disabled or tuned in a large number of manners), and
(2) The actual behavior that is approximately what you describe already usually involves use of PRNG (with a user or harness supplied seed), not a true RNG; the change to do watermarking isn't going from RNG to PRNG, it involves adding an additional set of constraints on token generation on top of the existing ones, which inherently compromises quality.
You're generating a pseudo random number one way instead of another way. How would that inherently compromise quality?
Same reason that you don't just replace your rand() implementation with "return 4; // chosen by fair dice roll". If you need randomness for whatever reason, biasing the generator is compromising quality.
( https://xkcd.com/221/ )
In case of LLMs, you can look at it from high and low level.
At low level - if you could do with less randomness, you can always lower temperature. You usually keep it (or for SOTA providers' chat UI, they keep it) at a level where it's about right level - high enough to allow for more creative leaps and interpretations, low enough that it doesn't go off into crazy land after the third paragraph.
At high level - creativity is driven by randomness. If you had an author (fiction or nonfiction) you like for their both broad and deep range of insightful thoughts, would you be happy if they suddenly developed an acute porn obsession and uncontrollably added lewd subtext to every other sentence? Still creative, still deep, but now with that one strong attractor that biases their every thought in a single direction? Would you trust/enjoy their output as much as you did before?
That, slightly exaggerating to make it more obvious, is what "loss of quality" means here.
1) we’re not discussing those systems. We’re discussing a chat AI product called Claude, which does not offer those knobs.
2) Claude’s PRNG having a P is immaterial
(1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time. This is a consequence of how they are trained.
> LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.
LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.
Citation needed
You know you can just try it and see on any inference system thst has this knob, right?
Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
> which inherently compromises quality.
I don’t see how this follows? Tokens are chosen randomly. If you choose tokens with a different RNG in the same distribution, you’re still getting equally good or bad tokens.
Not all values of "equally good" are equally good.
Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.
Never mind metaphors and similes, which are even more tightly constrained.
LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class journalism.
LLMs are still somewhere between the first two, and nowhere close to approaching the third.
> Not all values of "equally good" are equally good. Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.
We already know that a non-zero temperature improves quality though with current models (particularly with creative writing). The assumption that always picking the 'best' token results in the 'best' output is not the current reality.
And if you are already intentionally putting in randomness, I can imagine that it would be possible to seed the randomness in a way that is detectable but results in the same quality.
This is obviously not true for queries where temp = 0, but at temp = 0 then it becomes easier to identify anyway. I assume this technique implies some level of temperature.
It's not just swapping synonyms. The way llms work is by predicting the likelihood of the next token. It's inherently probabilistic. Choices are made based on weighted random number generation, based on those probabilities. Changing how you generate the random numbers doesn't degrade the output.
wow even on HN people have no clue how any of that works? all LLM generation has some inherent randomness to it, if you replace part of that randomness to be deterministically random the result of the generation with the fingerprint and without it, is INDISTINGUISHABLE. This has absolutely nothing to do with "synonym swapping". Its also again people not understanding how anything works missing the real concern, which is that nobody can ever tell if it isn't also secretly a fingerprint with the user and session id.
I think this is a key reason why humans write better prose than LLMs - we can try to choose the best word every time, and go back and restructure sentences and paragraphs if we want.
On the other hand, LLMs are forced into picking some likely-ish word, and then have to build the rest of their response to retcon that choice into making sense.
Even good human writers would probably struggle with this constraint. It would be like someone interrupting your writing to tell you the next word MUST be such-and-such, and then you have to try and make it work as best you can first try, without going back to edit. The result would probably be a little clunky. (Maybe it’s impressive LLMs write as well as they do.)
This is the classic misunderstanding that LLMs only pick the next token at a time. Really, they are coalescing the probabilities of a range of tokens at a time. There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.
>There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.
You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of tokens at a time" is poetic sounding jibberish though. What's actually happening is one distribution output for the next token computed from a hidden state that implicitly encodes where the text headed.
Your claim that if an LLM does happen to pick a token "th" instead of "tw", then the LLM isn't stuck with that decision is entirely false for autoregressive LLMs which is what all of the frontier models are. Whatever an LLM picks as its output token is final, it has no ability to undo that token selection and it must continue on the basis of that choice. It can't go back on that decision and revise the output.
If you're interested in this, Anthropic has a summary of a very technical paper on this topic that mostly deals with this issue with respect to poetry:
https://www.anthropic.com/research/natural-language-autoenco...
Reasoning tokens are a way to escape autoregressive woes. The model can generate a draft, then ponder on it, and use this to generate a final version
They’re a way to mitigate it. It still writes like an LLM and everyone can see it.
What a crazy link:
Incentives to train a pathological liar. There's no baseline so can only catch out the worst of the lies/errors. Anything (including fabrications) that passes our filters is reinforced?What are you even talking about? If there was a "baseline" for how to decode activations into text we wouldn't be having this conversation.
The choice is between "this reconstruction sucks" and "no reconstruction", and we're only now beginning to learn how to make those reconstructions suck less.
No, they really do one at a time. You're incorrect on that.
Mathematically, a long chain of conditional probabilities is equivalent to a single probability over the whole range. But computationally, for that to work out, the computation for the first probability needs to somehow consider all the downstream probabilities depending on it, which obviously isn't how autoregressive language models work. They can pack in as much downstream computation as their neural architecture allows for, which is quite a lot.
Suppose in some context you have three equally plausible conpletions after "Be": "tween a rock and a hard place", "twixed he stood there" and "lieve he can fly". To model this probability distribution of the whole sentence, the next token "tw" needs to appear at 2/3 probability and "lie" at 1/3. After "tw" would be a 1/2 chance of "ix" and a 1/2 chance of "een"; after "lie" would be a 100% chance of "ve " and in any case the rest of the sentence after that would be 100%.
The model needs to somehow "think ahead" to know those are the possible completions. For example if "lieve he can swim like a dolphin" was another equally plausible completion, that first token would need to be 50/50 instead of 67/33. So the computation of the first token somehow needs to encode the fact that the guy thinks he can fly but not swim, even though it doesn't become relevant in the output until several tokens later.
In practice this probably happens to some degree but definitely doesn't happen perfectly. To perfectly model the first token's probability distribution, it would have to include knowledge of the entire distribution of all possible outputs, which is just not happening. So it approximates. Surprisingly, the approximation is good enough to produce language.
You can see this breaking down in the seahorse emoji incident from last year. When you ask the model if there's a seahorse emoji, it first completes "Yes," as if a few tokens later it's about to produce a seahorse emoji. But when it actually gets to the token that would produce a seahorse emoji, it can't because there isn't one. But it's already outputted "Yes, the seahorse emoji is" and can't just go back and change that to "No, there's no seahorse emoji." Some models would try a few times and then say there isn't one or a system error seems to be making them unable to produce one, other models (including then-current ChatGPT) would loop forever with ensuing hilarity.
But on some level there is uncertainty, right? Even if it’s not token-specific but at the word- or phrase-level? Otherwise what does the temperature setting do? Or has architecture changed significantly in the background?
There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.
Are there any diffusion-based or otherwise non-transformer-based models in mainstream use?
If by mainstream you mean "ChatGPT" or "Claude", then no.
But also, no one really knows as they're closed.
everyone in this thread is saying something kind of imprecise and reductive and varies between models and even modifications among the models
I tried brainstorming what an agent harness for writers would look like.
https://chainofbranches.com/conversations/2/branches/20/
I’m not convinced it’s possible. A good nights sleep and a notepad in a quiet room still feels like the state of the art toolchain for writers.
This was true in the ChatGPT era. Now we're in a world with reasoning tokens, where a model can thoroughly plan out the response it wants to make. If anything, it makes the style worse.
Yes, models can reason and plan, which helps them write more coherently. But when they write the final output, it’s still a single generation. It would be like letting a human make notes and write an outline, but not let them use the backspace once they start typing their response.
Presumably you could use the same reasoning trace, run multiple generations, and get different outputs (if the temperature is >0).
But now I’m interested in playing more with Cowork or Claude Code/Codex for prose writing to see if the set of tools there affects outputs at all. I guess you might need a more custom “writing” harness.
Isn't this just chain-of-thought though, doing the same thing multiple times without necessarily defining one path?
Chain of thought attempts to guide a single forward pass.
Reasoning tokens with tool calling tell the model to loop on a one phase of a question and call a tool to indications completion when done.
Related, but not the same thing.
> [...] and go back and restructure sentences and paragraphs if we want.
That's what LLMs in reasoning mode do, too, to the text they present to you.
I also really liked the quote "Your existence is not impossible, but it's also not very likely" from the Night Vale podcast.
I feel like the existence of good writing is also not impossible but not very likely, and so of course LLM can only write mediocrity, even when taught only on great writing.
autoregressive generation doesn’t mean the model is myopic. the next-token distribution can already reflect a longer horizon plan for the output sequence.
Sure, but mightn’t there be several plausible long horizon plans?
Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was:
—
*Long journey tracks*
Clinic — “The Return of Evil Bill”
Guided by Voices — not really, wrong band
Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic.
—
So at some point there, the next token produced was “Guided” or “Guide” or whatever, and then because it can’t go back, it had to correct itself after the fact.
Reasoning/CoT have helped a lot, but I feel like small versions of this still happen all the time.
Human writing is like 90% editing.
Would be fun to run an LLM on fake output from itself. Like just force the first N tokens to say the beginning of something really stupid, and then see how it finishes the sentence. "You're absolutely right! Human feces is actually the most effective engine coolant because $<completion>"
It can but it is limited because it's only got a single pass through the network to fit the entire "longer horizon plan".
> Even good human writers would probably struggle with this constraint.
But that would be a fun writing exercise, I think. Thoroughly in the oulipo wheelhouse.
Maybe generate a Markov chain table over all of Project Gutenberg and then say every 10th word is whatever the Markov Chain thinks it should be at that point?
Or every Nth word has a P% possibility to be constrained by the chain? Optionally with the possibility building for each skipped word to guarantee it happens at some point. Bonus with this approach is that the human can't game the words leading up to the constraint because you don't know when it will happen.
It's a bit like trying to finish a sentence when you're really stoned... you vaguely remember the preceding couple of words you've said but don't really know how you got there and now you're wandering in the forest trying to stumble on coherency.
Well, I suppose it's nearly the opposite of that experience, upon further review. But for some reason, that's where my head jumped.
After "stoned" you triggered into a different state.
You be a human who's brain shifted into LLM mode (chainneling Markov?).
Or perhaps you're an LLM impersonating humanity.
I often wonder how much LLMs are just mirroring our own brain's patterns.
You're thinking of Markov chains.
Humans already do struggle with this constraint. Good examples are JRR Martin, Tolkien, and Rothfuss. You cant describe the struggle of picking the next word and then act like humans don't sit at the table struggling to pick the next word.
Models can easily do multiple passes
Human writers do better because they can think, and adjust, based on context.
They are also usually worse (which is often better!) because they are usually lazy and don’t want to spend effort they do not have too, to accomplish their goals.
Their goals are often complex and nuanced.
None of this is true of LLMs.
auto-oulipo
Today I learned a new word, "Oulipo". Interesting.
But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something?
I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.
Yeah this is my main issue with the argument. He acknowledges in the article that LLMs are already non-deterministic, but he doesn’t seem to actually understand that.
I think the article is wrong on this but it's more subtle than that. Probability distributions have a peak; there is still a token with a peak probability. What's interesting about these techniques is that token by token it can actually make the peak token even more probable. A distribution doesn't have to be "flattened" to leave a watermark - it can be "amplified" and made "more peaky".
The quality of the LLM just _is_ the quality of the token probabilities it generates. Better quality token probabilities, better quality output. Worse quality token probabilities, worse quality output.
Watermarking changes the probability calculations for reasons other than quality. It can't not compromise quality. It literally leads the LLM to occasionally chose different tokens just for watermarking purposes.
No one uses a pure random function over the whole probability distribution described by the LLM's output. For example, there is exactly 0 probability that the chosen next token by any common API or even local LLM runner would be a token whose final value is "0.0001" if there exist at least K tokens whose value exceeds "0.7".
Also, as long as the same sampling strategy is used during training as the one used during inference, then the LLM will actually do much better with the biased sampling strategy than it would with a fair one - because that is what it was trained to optimize.
>No one uses a pure random function over the whole probability distribution described by the LLM's output.
So what? By definition with this system the LLM will chose tokens it otherwise would not, purely for watermarking reasons. Yes this token may have had a decent likelihood of being chosen anyway, but it wouldn't have been chosen and now it was for reasons nothing to do with output quality.
I'm not sure what your last paragraph is trying to say. The blue/green list system changes what output the LLM would otherwise produce. You can't train it to produce watermarked output with this system. If you tried to, there would be no delta between trained output and watermarked output for you to be able to detect.
It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.
We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.
Fundamentally the article is railing against --temp > 0.0. He doesn't know what he's talking about.
>It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.
You can't as a user tell by how much the quality of the output was degraded. True.
>We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.
I'm not saying it wasn't random and now it is. I know how these things work. I said that the quality of the system is in the quality of the probabilities. That quality is being degraded.
This is true and the author seems to not understand the problems with greedy (top 1) decoding or the fact that watermarking affects only high entropy tokens. But the published watermarking methods still have a slight negative effect on perplexity, so there is something more to it.
That's missing the point. It's the distribution that's the "best", not the tokens. Then Anthropic comes in and makes the distribution something other than the best. The only saving grace is that Anthropic says it's not that bad.
Even so, I don't think it will stop here. Once this is in place, the next step is to put more and more identification into the AI generated content; might as well pack it in, it's not that bad, and if it is they won't admit it. There's no way for anyone to check. And your argument will still be technically correct but missing the point.
there's absolutely no reason to think Claude produces absolute best token distributions or that slight adjustments would be noticeable.
In fact we know it's not that good because we can often tell Claude's writing apart from human writing.
How do you know you picked the singular “best” set of tokens in your comment here?
Could it have been equal or better with slight variations in wording?
The slipper slop argument is too lazy to address directly. Argue A is bad because A, not because A might become B and you’ve got good arguments against B.
The watermark doesn’t change the distribution, only per-token selection. I think not understanding that is the source of most people’s FUD.
There's no difference between those two things. The distribution that matters is the distribution of tokens that are picked not the distribution of tokens the LLM model passed to the selector.
On average the distribution of selected watermarked tokens is the same as the original distribution. You can test this experimentally.
I think there is a possible weakness in the context of the watermarker but that is not your claim iiuc.
The distribution of token "ple" being the same on average, but lower after "crum" and higher after "cou", is not no difference. It's irrelevant that the single-token distribution is unchanged if the joint distribution is different.
This comment disagrees with you: https://news.ycombinator.com/item?id=49324387
That comment merely says quality must be compromised. It doesn’t make it clear why that must be true. Empirical study seems to say that quality is not compromised, and looking at various proposed schemes, it seems intuitively true.
Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
It seems to me like he started out mad and looked to justify it.
I'm skeptical that anybody generating LLM text is really all that concerned about optimal word choice. Or even particularly good prose. But let's pretend that person exists.
If that person tried, say, an open model and that same model with watermarking applied, I'd be eager to hear their thoughts on the prose quality. Especially if they built an experiment harness and rated a few hundred blinded examples and found a measurable difference.
But getting this upset in advance of any demonstrated problem? It really seems to me like the point isn't the point
> It seems to me like he started out mad and looked to justify it.
Yes, but that's neither surprising nor a reason to dismiss the anger. People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible. They're angry -- and Gruber acknowledges that factor too -- because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.
It might be another instance of consequentialism vs. honor ethics. Many consequentialists don't seem to understand that something that doesn't have demonstrable consequences can still have moral implications.
> People get angry about DRM schemes
Yes but thats a thing that degrades something in a catastrophic way, as in I can use the thing one day, and not the next.
A different randomisation system on something that is a text generator which is designed to be unperceptable sounds like the people who are annoyed at FLAC vs MP3[1]
Done right you won't know the difference, done badly and you will.
[1] ex audio engineer, try me.
> People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible.
Any potential "slowdown" doesn't even come close to making the list of top reasons people get upset about DRM.
Even if honor were real, LLMs do not have honor, and people using LLMs to write without disclosing that fact (or indeed at all, to some purists) do not have honor either.
> because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.
It's LLM output! It's not your domain, it's the LLM owner's!
> It seems to me like he started out mad and looked to justify it.
that's been his thing since it was just a blog about apple product speculation and update. It's always been tedious.
Google has A/B tested watermarking on millions of responses. They say they observed no difference in user behavior.
It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.
How exactly do you propose they should keep track of quality, then, if not by A/B testing?
The question here is not to what product management advice to the Gemini team. The discussion here is whether the watermarking is noticeable.
The easy thing to do here would be to have 1000 questions, randomly assigning one half to an LLM with a watermark, and the other half without. Then show people pairs and say, "Which one seems watermarked?" (Or, "Which text seems more natural" or "Which is a better answer" or something like that.) If they come out equal, the watermark really is indiscernible, at least to most people.
Considering Gruber's always comically butthurt about regulation, especially EU regulation, your theory seems accurate.
I assume they are just passing off AI prose as their own and don't want anyone to be able to tell. Which is surprising for someone who's been blogging for a thousand years. But I don't really see any other reason for this amount of heat and FUD.
>fails to understand the gumbel softmax technique
I think in this case it doesn't help that there are multiple watermarking schemes, and the easiest for people to understand is the red/green scheme by Kirchenbauer et al. (https://arxiv.org/pdf/2301.10226), which does technically distort the logits (but I'd argue only in cases where you wouldn't notice it anyway).
I wasn't aware of this gumbel softmax scheme, it seems you're referring to https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-o... ? That's really clever as it doesn't even distort the logits, basically cryptographically indistinguishable from a "real" random sample unless you have the key.
The actual scheme Claude uses seems to be neither of those two though, they say it is SynthId-text which seems to be tournament sampling based.
>It does not affect writing quality at all, provably
Prove it, then? It's not a claim that GumbelSoft paper makes: "Regarding generation quality (perplexity), GumbelSoft shows relatively low perplexity"
https://arxiv.org/html/2402.12948v3
If this is true then the probability of the detection tools flagging completely human generated text as AI generated is non-trivial. Let's say I write a completely original piece and the detection tool says there is a 36% probability it was generated with Claude. What then? Now it's up to the person looking at the score to cast a subjective judgement. Maybe to me, anything over 25% is unacceptable. Maybe to someone else, it must cross over the 50% threshold. This is the problem.
Cognitive surrender.
> the probability of the detection tools flagging completely human generated text as AI generated is non-trivial
How does that follow? AI-generated text is already not a perfect emulation of human writing. There's lots of room to affect it laterally without changing the level of quality.
As I understand it, LLMs with temperature >0 can select from many possible outputs. All they're doing is limiting the possible outputs to ones that contain this pattern. I don't see any reason why the quality of that subset should be lower than average. The very best outputs will likely be eliminated, but so will the very worst.
To sample from the probability distribution you already need random numbers.
If you get your random numbers from a cryptographic PRNG, then to notice the difference between that and 'real' random numbers even in theory, means you need to break the cryptography. In practice, your gut feeling about how good some text is won't break modern cryptography.
That's an interesting problem to discuss, but unfortunately TFA spends no time discussing that.
Yeah. And the detector can’t even tell you “X% chance this is watermarked” because it doesn’t know the input distribution. It can only tell you “Y% chance that an unwatermakred text would score this high” and how many history professors understand Bayes rule well enough to understand the distinction?
Worse, what will academic institutions decide is the threshold for detecting AI generated work. If you have a false positive how do you prove it was a false positive or we all just trust the watermark detector over the student saying "I swear I did it all by my self"
I don't think that's true. I think it's a binary 0% or near 100% probability of a watermark having been detected; the more changes to the text having been made after the text was output by the LLM and the less leeway the LLM had for probable word choices, the longer the passage necessary to see it.
The "problem" is that seeing the watermark doesn't mean that the person claiming to be the author didn't make extensive changes to the output of the LLM, or that the LLM wasn't simply the final editor of something that the author had put a lot of work into.
> Cognitive surrender.
I don't know what this means. It's just drama. Don't let the LLM write for you and this is not a worry. I'm not worried about the poetry of LLM output being subtly adulterated.
No, watermark detection is not binary, you get a real number. You decide on a threshold when looking for the watermark. This is the problem - by random chance, some human text will be detected as watermarked. You can turn the detection threshold up until it guarantees <0.001 false positive rate at the expense of higher false negatives, but seems inevitable that someone gets wrongly flagged.
Making blog posts about AI that make it apparent that the tech is going whoosh is a choice.
"Smart"? Have you read his writings in the last decade? It's all nonsense, which is why I stopped reading circa 2018
I think he’s still generally good on business, UX, and hardware design. That’s all subjective and taste I suppose, but his taste works for me.
On deeper tech stuff, like this utterly nonsensical misunderstanding of watermarks… yeah, classic case of a guy who is smart, and has lost the ability to realize when they’re not knowledgeable in a domain.
> The very fact that there is generally no "best next token" with 100% certainty
Indeed.
It's frankly bizarre to see the assumption to the contrary being made by someone who's been passionately blogging by hand for years, who also happens to be responsible for the notoriously vague, humanistic, DWIMmy Markdown standard.
I couldn't be happier that people are mad about it. To quote Calvin, "nothing helps a bad mood like spreading it around"
Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles?
Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News the next day calling for an invasion. As a former reader of Daring Fireball, it’s just sad to see.
Yes, I decided to stop reading his blog relatively recently after some extremely hot takes on EU policy. I don't feel his thoughts on the matter are particularly well-thought-out, and I feel like he's just stanning for Apple from his priors rather than from any grounding in reality.
I dunno, I guess that's what you should expect from Gruber but these EU-bashing articles lowered the enjoyment I got from his blog underneath the bar for me.
Gruber went from the naively wrong claim that it would insert secret hidden characters (which would be trivial to remove, obviously), to quickly writing a giant essay as if he's an expert on LLMs. Like you said, he is strangely fixated on the EU, and is certain any EU rule is the worst thing in the universe, and this whole piece seems motivated by that guiding force.
Further he later compares Gemini to Anthropic models, saying the latter "writes better", emptily ascribing this to the synthid stuff. I think he heard that Anthropic currently has superior models, but it certainly isn't because they "write better", and if anything Opus 5 now is virtually unintelligible, before the fingerprinting.
The fingerprinting stuff sounds weird. If the EU wants it, it should be limited to the EU, and Anthropic is fully capable of doing that but clearly saw value in recognizing their own output. Is it going to destroy the quality of the output? We'll have to see, and this anti-EU piece, predicated on utter ignorance of the field, is not convincing.
What a bunch of entitled whining. How is the system to know that it's just a private conversation that won't be used in some fraudulent way? Abuse is currently rampant, yes please let's find a way to mark LLM output. The thing I'm worried about is giving the providers the power to claim provenance. Even ignoring the privacy issues, the operational hassle of having to check N providers makes these approaches at best limited. I want to see research into providing a shared public or ideally self-hostable oracle that uses some standardized method for watermark detection. Similar to asymmetric crypto where users can't reasonably find out the secret part but can do something useful with it nonetheless.
>I want any LLM I use to choose the very best, most precise words at every single decision point.
Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?
This entire article just seems so detached from the basics of how LLMs work.
> Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?
No and no. I am not sure I agree with his point but I know he is not ill-informed on either of these points, because I mentioned them to him a couple of days ago.
It didn’t take, apparently.
It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point.
The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text.
The point he is making is consistent with this, isn’t it? Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure. These are ethically distinct approaches, and since he disagrees with the EU objective he comes down on one side I guess.
Me, I don’t care about the hypothetical enough.
Not least because I think Claude writes depressingly badly and I doubt any steganographic change will enrage me less.
The point he is making is not consistent with understanding how temperature influences LLM text generation, no.
He repeatedly states that choosing "the best word" is the most important thing to him. I don't know how you reconcile that with creativity itself, let alone probabilistic sampling.
Yes, naively understood in his sense 'best' word means that you pick the word with the maximum score, instead of sampling from the distribution.
That doesn't actually give you the 'best' text in any human sense of the word. Just like playing the 'best' move in Poker without sampling leads you to lose a lot of money.
I mean creativity there in the LLM sense (temperature driving more creative solutions), not the human sense, and in the context of its consistency, configurability and being amenable to analysis. The watermarking approach makes that non-reproducible, yes? Because Anthropic can and will change it as they see fit.
>Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure
This framing does not make sense to me. What do you mean by "influenced and analysed"? How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing? What makes the unadulterated randomness "driving creativity" but a different random choice uncreative?
No, you are jumping to conclusions about how watermarking works. This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever. Intuitively this may be true or false depending on your personal prior but you’d need to show it mathematically. The overall token distribution shouldn’t change and the frequency at which you see the word “load-bearing” will remain the same.
> This is some audiophile thinking that because your RNG is “pure”, you get text with an expansive soundstage or whatever.
You are projecting that onto me, and I cannot tell you how comically poorly aimed it is.
What does he think of all the other adulterations of LLMs that already happen?
I think the author is just mad people will be able to detect and filter out their AI slop writing in the future.
Gruber just hates any kind of EU regulation of US tech companies ever since they started making what he calls "product decisions" for Apple.
Gruber has long been a talented, excellent writer. I hugely doubt he uses AI at all, nor does he plan to.
His first take on this situation was cutely naive, thinking they were going to inject secret hidden unicode characters. But ultimately he has a massive hate on for the EU -- they were mean to Apple once -- and it comes out in any topic that overlaps.
This is not it, no. He is not using AI and it is not I think remotely in his nature to surrender that control. He is engaging with this on principle. Again I am not sure I agree with him, but then it’s a hypothetical because I am not going to get an LLM to write for me either.
> He is not using AI
That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.
Why? I really don't understand this. Why can't a tech writer take a deep but neutral intellectual interest in something? Isn't it important that some do? Do people have to be stakeholders or clearly on one given team, pro- or anti-, for their opinion to matter? Is it that tribal?
It seems fully logical to me that someone who writes for a living (who, as it happens, developed the very markup language LLMs use for everything) should be invested in understanding the automatic plagiarism and word calculating machine from an intellectually honest position.
I personally am pretty severely big-two-AI-firms, increasingly anti-big-tech, but I am learning and researching uses of LLMs because for myself I really need to understand how to use them in an intellectually and (as far as is possible) ethically sound way. Learning because as a boring old freelance programmer I have to; foolish to pretend otherwise.
So I completely understand his position — that the AI industry is hot air and crooked and scammy and weird, and some of the people involved genuinely rather dark-sided, but the technology exists and if it hints at threatening your livelihood, you need to understand it.
From reading his work for the best part of twenty years or so (and emailing him intermittently over that time) it would seem to me that he's a lot less bearish on the tech industry than me, and a lot less fond of the EU than I am; he's more optimistic than I am. But he writes because he has to write. I should think that would make him highly invested in understanding what LLMs do.
Well, it cant be that he is super worried on behalf of people who publish AI slop. That’s not a credible motivation. In fact, he complained a lot about the new ChatGPT app so I can’t believe your claim that he is not using AI.
Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.
His complaint about the ChatGPT app is that it’s a shitty non-Mac-ish Mac app. Complaining about shitty non-Mac-ish Mac apps to people who hate shitty non-Mac-ish Mac apps is more or less how he became a full time writer.
People already are, and do.
What a truly bizarre article. Arguments about pre-existing randomness, temperature and whatnot aside, I simply cannot comprehend what the author here really thinks the "best word" is. There's no such thing. We humans fall on familiar patterns of writing ourselves, so we may forego something with a flourish in favor of a more commonly-used word unless we put in effort to be "special", which should be used sparingly. That is to say, human writers are likely to choose a "worse" word in far more than the supposed 51% of cases, and that has no effect on the actual quality of writing in the end.
But even if there were such a thing as a truly "best word", for some context, what are the examples here? Mango vs pineapple? Gray vs overcast? In what case is one of these better, that AI would normally infer but would suddenly be "perverted" by SynthID? Do you think your emotional state and preferences are being evaluated if they aren't explicitly in memory? And if they are there, do you think that the generator will bypass those instructions in favor of the watermark instead of placing it somewhere you won't care? I just. Genuinely don't get it. There may be words that matter in specific contexts or to you as a reader, so you should bloody well put them there.
Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude.
I think that was intended, yes.
You know, back in the era when proofreaders were human, I never one met a proofreader who rewrote my text afresh, rather than annotating the text with a pen.
It's still possible to use Claude to proofread - highlight grammatical, flow, structure, logic errors and make simple suggestions for you to pick and choose or adapt as you wish. No watermarking will flag your text. No flaw accusations of LLM authorship will haunt you. All will be fine.
But if you want an LLM to rewrite your text, that's (a) not proofreading, and (b) should be flagged as LLM generated ... because it is.
Where is the problem with using LLM generated text?
You could use your own hypothetical house elf to do it for you, or pay someone to do it. LLMs are just cheaper for a certain set of problems.
People will find ways to circumvent this, so this limitation will only hit the technically less adept people.
> Where is the problem with using LLM generated text?
In the fact that you didn't write it.
> You could use your own hypothetical house elf to do it for you, or pay someone to do it.
Yes, and those would be similarly problematic (and more expensive).
> In the fact that you didn't write it.
This is a fact and this is generally not a problem. Customer support guy Joe did not write that email to you with a refund: someone else did it and Joe did pick the template. Alice did not write that post card to Bob, someone else did and she just googled some nice text. We deal with a lot of content that wasn’t written by the person who signed it. That content, when written by LLM, may indeed contain watermarks and nobody will care about the choice of words, because only the meaning matters in such communications.
People pay too much attention to authenticity here, which is no more than a demonstration of an effort. LLM can and should write scientific articles because the real effort is in directing research, not summarizing it. LLMs can and should write news, because it is cheap and efficient, and real reporting is in discovery. LLM can and should write fiction and make movies, because there is no reason why creators of various junk should earn their money easily. LLMs do not replace real talent. They just emphasize for an average person how easily replaceable they are. And that‘s ok. Creative industry is a blue collar job now.
If it's not a problem, then there should be no issue with not concealing the fact, no? Lying about things one considers inconsequential is a useful signal about one's willingness to lie with little benefit.
Nobody says it's a problem. We'd just like to know.
Factory farming also makes meat cheaper than organic practices. I'd just like to know which one I'm getting.
Nonsense. Forget proofreaders. Think editors. In publishing some editors practically wrote the books. And then theres ghostwriting ! Think of that!
Part of me wishes we had the same regulation for ghostwriting etc. Nobody should be claiming to have written a book they didn't.
Okay, then it should be acknowledged if a work was AI-edited-written, or AI-ghostwritten.
I'm not sure the quoted statement is true. Proofreading like "point to problems in the text", if you fix the problems yourself and don't copy-paste the solutions given to you, should still be safe, shouldn't it? So, human-written text should not be falsely flagged if you use LLM for proofreading.
And if you copy-paste the answers from LLM, I think it's only fair the end result gets flagged. You're not writing it yourself.
> And if you copy-paste the answers from LLF, I think it's only fair the end result gets flagged. You're not writing it yourself.
I wonder if it would even get flagged in that case, because wouldn't the probability distribution of a token when the LLM is suggesting an edit to your writing be different than the distribution of that token once it is in the context of the text it's editing?
I don't even ask LLMs to go that far. Tell me if I've made a spelling, punctuation, or grammatical error, period. Don't rewrite a thing, because LLMs suck at that.
The question is, when is the "pro writer" version coming that lets you control this behaviour but costs more? Like night follows day, this will happen.
They will need to dodge around the EU requirements but it will probably just come down to an alternative method to watermark or a contractual assurance you won't mis-represent the source of the text.
I had the same thought.
I hope the other providers will add a geographical limitation on this EU rule.
(On a side note, I wish they would replace those EU beauracts with LLms).
Can't the LLM just generate e.g diffs? Or some other intermediate language. Then the watermark is lost when the translation step is applied.
wait what?
but proof reading is a linter, not a writer. the proof reader will say "I think this is clumsy can you try x,y & z"
Rands made this point a few days ago as I recall. Worries about having his tool corrupt his writing during editing, etc.
The thing that could change is interpreting "the whole thing as generated by Claude"
Well yeah, if it's output from Claude it's likely to get detected as being output from Claude.
It’s a strawman argument because if the LLM is really just “proofreading” for you, there will be little or no watermarked text in your writing. Not enough to trip the watermark detector.
It is quite just, if you think about it. Human works are copyrighted and protected at the moment of creation. All rights reserved. Yet, LLM outputs are uncopyrightable. Therefore, if Claude or any AI has processed my copyrighted work, the end result is uncopyrightable and in the Public Domain. The public has a right to know: is this a human copyrighted work, an LLM PD work, or is the human falsely claiming authorship in order to retain copyright?
A point of confusion for me, however: is every watermark unique? Is every algorithm for watermarking going to vary amongst models and amongst model versions? Will each model publisher keep this watermarking as a trade secret, that they alone can detect? If so, this can't scale! How do you detect "JoeBob 4.3 LLM" output? By querying every single model's watermark-detector? And if they all work by re-running the model and using tokens anew? That is extraordinarily wasteful.
If a watermark is not self-evident, or universally detectable, then it is no good. Take, for example, US currency. The security measures are published and well known. Any count-out room in retail has a big poster indicating how you can detect authentic US bills. Nobody has to accept non-US currency in the US, and so the only authenticity you need to worry about is your US bills alone. LLM watermarking has none of this in common. Currently sounding like a shitshow, if you ask me.
Perhaps LLM outputs are uncopyrightable, but derivative works of copyrighted works are not automatically in the public domain.
That's an intriguing twist, isn't it? It could lead to a tug-of-war.
Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now?
If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works?
This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.
How do you assert it now? I post some text on the internet, you claim you have copyright, how do you prove that?
Some camera pointing at you while you work. Guessing the camera needs to be watermarked itself, which can be done by adding some watermark on the chip level (each camera will have different watermark), manufacturer can confirm the watermark.
Is that realistically how that is proven in court today?
The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.
As for it being "uncopyrightable" if it were the output of an LLM, I think computer people are making a very aspie interpretation of a single decision. I think it's more that the LLM (and thereby its owners) cannot itself hold a copyright on its output, that output has to be touched by a person before it is copyrightable. A particular view from the top of a mountain can't be copyrighted, for example, but a photograph of that view can be.
I'm not sure it at all precludes a "robosigning"* sort of situation, where machines generate output, hired temps sign and claim that output, and immediately sign it over to the people who hired them (as a work-for-hire.) Copyright is stupid, artificial law, not logical.
-----
* https://www.mortgageauditsonline.com/what-are-robo-signers/
> The watermark, if I'm understanding correctly, only proves that something has been touched at some point by a particular model, not that it was entirely generated by a particular model.
For the watermark to be detectable, the text needs to be like 75% AI generated.
If you have an LLM “touch” one section of the article, it’s not gonna be detectable.
As I understand it, the current watermarking methods rely on a secret key, making the detection schemes a black box to anyone not in possession of the key. This means organizations like Anthropic are free to make any claim about authorship they want, true or not, and no one can call them on it.
Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume).
Wouldn’t having that be enough to eventually reverse engineer the key?
Probably not to get the key, but you could certainly use it adversarially to remove the watermark.
Removal may come down to changing every third token to a different one.
Not if they designed the algorithm right.
> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.
Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best.
This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.
Yeah, I snorted at the sentence "The exact words we choose when writing matter." Well, then why the heck are you using an LLM to "write," man?
This seems like a non-issue, or maybe I have the wrong expectations about writing. You write a text, ask Claude to proof-read it, but then you wholesale just copy Claudes output and use that as the final text? Wouldn't you review the changes it suggests and only take those you agree with, there by completely bypassing the watermarking?
Alternatively, you ask Claude to write the whole thing and proof read it yourself. In that case I'd like to know how much you'd need to change to break the watermarking, i.e. how much of a text would you need to change for it to be considered your work and not that of Claude?
Gruber was a good voice in the industry but this article misses the mark in a lot of ways.
A company the size of Anthropic would not voluntarily jeopardize their massive valuation if they didn’t feel the resulting output would maintain a similar level of quality as before. Is there a similar worry that their system prompt, which is injected at the start of every conversation also influences token generation in an artificial way?
If regulation will ruin Claude as a product, market forces will fill the void. There are also a ton of open weight models to choose from. It’s going to be okay.
Gruber is not much of a details man - he helped invent Markdown (to be lauded) but ghosted its standardisation. I would be fascinated to hear Prod John MacFarlane of UC Berkeley's opinion on it all given he was heavily involved in the push to get Markdown standardised.
LLM inference already isn't deterministic, the watermarking technique only limits the space of possible random seeds. There is no reason to believe that this subset of seeds somehow produces lower quality output.
There is no very best token to choose at each decision point. It is context dependent and subjective.
This article feels slightly incoherent. You want high quality precise writing and to use an LLM to generate it? Feels like those are diametrically opposed
Exactly. The watermark is proportional to how much text is AI generated. Either the AI really just “fixed some typos” (not enough AI content to hide a watermark) or the AI did most of the writing (enough AI content to hide a watermark).
This feels like the inevitable outcome of a STEM-only education system. Now people think there's a mathematical formula for picking the "best" words, instead of having to be thoughtful and creative.
I was initially surprised that Gruber was so invested in the "quality" of AI-generated text, which in my mind is an oxymoron. But really, Gruber's interest here is with the EU. This forms part of his ongoing attacks on the EU, all because they have been forcing Apple to align with regulations.
Can we install random unapproved apps on our iPhones yet, or is Apple aiming to just be fined a trillion dollars because they make more than that from the 30% cut?
Yes: https://support.apple.com/en-mk/117767
This says we can only install Apple-approved apps.
I commented it last time the post about Claude watermarking went viral and I'm going to say the same thing again:
"I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver. There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike. As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026."
Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous).
And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs.
Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.
>in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code.
I wonder if this is why Opus 5 keeps writing excessively long comments, even though I keep instructing it not to (both in chat, CLAUDE.md, and in its memories)
LLMs are already perversions of writing, so what else is new. Oh no, the over-long circumlocution generated by three autocorrects in a trenchcoat might be slightly longer because of this and maybe people will start noticing the subtle rhythms of vaguely peculiar word choices as yet another cue that you are wasting their time with machine-generated wordslop, what a terrible fate. Your long rambling walls of machine-waffling might be 37.05% longer than they need to be instead of the mere 36.58% longer they are now.
This is not meant to be snarky, But almost any writing done by Claude is a perversion of writing.
I honestly can't stand the way Claude writes. This watermark change just makes it scarier.
I moved to Sol for my writing and it is so so much better. But it makes more mistakes. I think they have different ideas of product but it seems OpenAI is going to follow Anthropic’s lead over the next year. I think I am going to put more effort into my writing skills to remove myself from this awful situation
Sorry, what's scary about it exactly?
> But only Anthropic will be able to determine if text was seemingly generated by Claude, and Anthropic will only be able to detect the watermarks that are applied by Claude. Claude can’t detect the hidden watermark signals generated by, say, Gemini, and Gemini can’t detect the hidden watermark signals created by Claude, because each implementation is predicated on secret keys held only by the LLM provider
Well, akshwally...
> Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027
The issue here isn't (just) adulteration, it's that watermarking in general is unworkable.
If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text.
Which is clearly ridiculous.
And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial.
That's not even getting into the legal complexities of businesses running open source models without watermarking locally.
> Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027
No idea why this was flagged to death. I vouched for it because it is a direct quote from the AI transparency act: https://theaicounsel.net/wp-content/uploads/2026/07/07_26_co... It clarifies interoperability requirements.
If we wish to use “the very best, most precise words at every single decision point” then I will note that “writing” is not the best word for what LLMs are doing. A more precise synonym might be “generating”, in which case this essay becomes “Watermarking is a perversion of text generation” - which is true, albeit somewhat trivial.
I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value.
Everyone who invests in AI companies wants to see the value of their investment increase.
I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that path.
I think you're right that it's actually to do with attribution (i.e. They're not just watermarking your output as claude generated, but watermarking it as claude generated _by claude user id 73684_). I think it's more to do with the growing militarization of the internet. Just another brick in the wall of enter your phone number to create an account, send your id to prove your age, smile for the flock cameras, etc. Allowing the plebs to have privacy and anonymity is not allowed anymore.
As someone who doesn't utilize LLMs for writing production code I'm looking forward to starting a consultancy and reaping the rewards.
I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms.
https://arxiv.org/abs/2107.06751
Seems unlikely unless Anthropic invented a time machine, given that the phenomenon predated Claude 1 by two years, and their stated introduction of watermarking (August 2) by 5.
I see how that was confusingly written. I'm not suggesting Anthropic are somehow retroactively causing it, just wondering if it could create a similar effect.
Less unlikely, but I would still suggest that it is somewhat unlikely for any recent LLM to place any significant probably on, e.g. "disappointment" instead of "failure" as the appropriate token (or series thereof) after "kidney" or any similar example.
It seems fine. I use an LLM to argue with me prior to posting blog posts so that I don't post obvious incorrectness, but the UX element to it is that it constructs notes about various sections of the text and we talk about those. There's no way for the generated text to enter the blog unless I copy-paste it and I'm not going to do that because the entire point is for me to write it.
At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.
it serves to show just how little regard the people behind these generated-text fingerprinting schemes have for the actual craft of writing.
LLMs have never been the place I've thought to expect any commitment to the craft of writing, to be fair.
Are there some narrow cases (like in outputting the input almost verbatim) where the probability for every best token is always much higher than second best, and in those cases would there be no watermark?
The fact that Gruber points to a "must-read" article about how the watermarking works, which, in turn, was very obviously entirely LLM-generated, says a lot about his lack of experience reading LLM output.
If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much.
NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as I believe it's quite relevant to the topic at hand.
Gruber really doesn't like any EU regulation
I think writing is the killer use case for local LLMs.
We've had so many advancements in LLM samplers for improved text generation (off the top of my head: min-P, adaptive-P, XTC, DRY, p-less, Top-H, Top-n-Sigma, and so many more) but hosted LLM APIs only provide three basic knobs: temperature, top-k and top-p which are old as the mountains in LLM years at this point.
One thing that doesn't help the local LLM case is that all the popular VC backed local LLM wrappers also only support the same three ancient knobs because I suppose they're more preoccupied with their next fundraise than with keeping up with the advances in tech.
> I chose to depend on a private company to express my own thoughts and now I'm mad that I'm not in control of the output
Who could have seen this coming???
"possibly build tools to take a chunk of writing and try to remove your fingerprint from it, i.e, protect anonymity,"
The standard procedure to do this, is to chain translations to other languages and back. The message remains, but the wording will pick up some noise. --Dec 30, 2008.
Gruber has a ridiculous knee-jerk response to anything the EU does, so hardly a surprise he didn't come to the table with a sober facts-based response.
LLM output, as the author acknowledges here, is already non-deterministic. Next token probabilities are set, and tokens are chosen pseudo-randomly. As I understand it, this watermark is just going to be a matter of using a known seed and algorithm to make those pseudo-random choices, such that a signature can be detected. The important thing is, it's not replacing intentional choices with random ones, it's just generating pseudo-random results differently. Quality shouldn't be affected.
That cannot be true. The quality of an LLM's output is the quality of the probability calculations for the next token. Anything that degrades the relationship between the system's best assessment of the appropriate probability and the actual probability used is a degradation of the quality of that probability and therefore of the output.
If this didn't have a detectable effect on the quality of the token probability calculation, the watermark wouldn't be detectable. It may be a small degradation in the quality of the output relative to the neds of many users in many situations, but it's not zero. It's literally sometimes choosing different words that it otherwise would specifically for watermarking purposes.
An encrypted hard drive is EXACTLY uniformly distributed random bytes if you do not know the encryption key. No one would be able to tell the difference between a drive that is just purely random numbers or is actually filled with content. (Of course excluding the usually intentionally added readable header)
If this was not the case, the encryption would be broken, and most everyone agrees that good encryption does exist.
The "quality of the probability calculations" as you put it is 100% in this case and any less would be a huge deal (as in - breaks all of the internet).
So, now you just take those same random bytes and use them as the seed for your LLM token choices. The output has the _cryptographically_ proven exact same quality as if you were using a true RNG (which you likely weren't using anyways).
You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.
> choosing different words that it otherwise would
The "that is otherwise would" is carrying all the weight here. "Otherwise" is sampling from a distribution. You just sample from the same distribution but with a cryptographically secure, seeded RNG. https://en.wikipedia.org/wiki/Cryptographically_secure_pseud...
"Knowing the LLM distribution" seems to me like the only hard part because you don't know the context of any random snippet.
>You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.
I struggle to understand the relevance of that comment.
The blue/green token list biasing process literally does cause different tokens to be occasionally chosen. Not only that, but because a different token was chosen at one point, this changes the probabilities of all subsequent tokens, and the resulting later token stream every time it happens. If you had access to the token stream as it would have been, and the watermarked one by the end of the text they will be very noticeably different.
> Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot.
How is this supposed to work in an actual lawsuit? Will Anthropic offer some sort of tool / (paid?) webservice to check for watermarks using that "secret key", and a judge is supposed to just believe whatever that tool's verdict is? And then it takes the EU another 20 years to understand what a silly idea this was?
Yes, exactly, that is my understanding as well. Since the watermarking is based on a symmetric key by design (as to not be easily able to test and remove the watermarks), the providers will need to offer a watermark detection API.
I see even more problems with this. To check any text for those watermarks, it needs to be sent to dozens of AI companies to check, potentially paying them all for just determining whether it matches their watermark, and more concerningly sending all that mostly human-written, often high quality text like unpublished research or books, to AI companies that almost all proved to obtain training data through all kinds of dubious ways.
> The provider must mandate in their terms-of-service that users not remove the watermarking.
So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?
Security theater, and vague enough to be used as a weapon against who the government wishes.
I hope it's left off for non-EU customers.
You know you can just ignore EU laws outside of the EU
They are literally asking for Anthropic to do that rather than roll out 1 system for everyone.
Claude's writing was already easy to recognize. The fact that Anthropic complied without complaint makes me wonder if they already watermark their outputs and used the opportunity to create goodwill. Presumably they want to avoid training their new model on text generated by the previous model, so they have reasons to be able to recognize AI-generated text.
I often agree with John Gruber, but I think he’s lost the plot with this one.
The thing I don’t understand is why he seems to care so damned much about this subject--enough to write over 4,500 words on it!
John writes for a living. That’s his profession. He’s been writing for over 25 years now. When you’re that good at writing, and you care this much about your writing, you don’t allow an LLM to take over your job. I just can’t imagine that he’s in the market for LLMs and that literary excellence is his number one selection criterion.
So why is he so livid about it? It’s like being angry that wine is going to start coming in smaller bottles even though you don’t drink wine.
Even if he’s angry on behalf of other people, I don’t get it either. In my view, having LLMs write publishable content on your behalf is not a socially-acceptable use case, nor a professionally-acceptable one in most professions, even though people are abusing it for this purpose anyway. And besides, the models aren’t even all that good at it today. If you agree with that, then you certainly should not care if it’s using different phrasing than you otherwise might prefer if the meaning is similar enough.
I can't help but wonder if perhaps his hatred of EU technology regulation (which, admittedly, is mostly pretty dumb and is mainly just making life worse for users) is getting the better of him.
This guarantees he can never use it for the one purpose he might care about though
Heck. The entire process itself of LLM text generation is a perversion of writing.-
Further heck: It can be said it ain't even writing.-
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.
That is not what that means. Generally, precise word choice requires more than autocomplete. Larger models simulate this with hidden layers.
Excessively precise word choice does not result in something that looks like content written by, or palatable to humans. It looks like you gave a high school 12 grade student a science paper and told them to apply a thesaurus to at least one word in every sentence and replace it with something else.
There is a difference between precise word choice and concise word choice. You can be precisely accessible the same as you can be concisely terse.
The main reason we don't see much quality degradation in LLM writing output is because they're already poor writers. This is the load bearing reason.
I was bulding a small interpreter and writing an article in ~markdown yesterday with Fable. And while it codes like a pro, it writes like a sixth grader.
Let's see how these watermarking stats hold up if/when llms start writing well.
Just copy Claude’s output and shove it into Gemini and ask to rephrase.
Watermarking will be one more nail in the coffin of proprietary models if the world is so fortunate.
Reminds me of printer tracking dots.
https://en.wikipedia.org/wiki/Printer_tracking_dots
And yet we still use printers and 90% of our color documents have the tracking dots.
There are many reasons to hate this watermarking but affecting the output quality isn’t one of them. The central argument he’s making is wrong. Switching out one RNG for another doesn’t make the results worse.
There is no coherent position in which the watermarking is a perversion of writing but AI writing as a whole is not a worse one.
Lots of faux outrage, rambling and hyperbole here from Gruber.
“Absurdly and insultingly”? Come on…
I dont see how there would be remotely enough entropy in most model outputs for this to be close to feasible with any kind of accuracy.
Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless.
Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.
Then then don’t use Claude ? What’s wrong with all these people getting vendor locked in.
I needed another reason to cancel my Claude sub. Thanks Anthropic!
This is akin to adding a giant watermark on things one would made with a free product "Made with XXX". Except you're paying $200/month for it, and there's no way to disable that watermark.
I don't disagree with EU regulations, but I strongly believe the onus should be on the content publisher, not the toolmaker. If the toolmaker watermarks whatever his tool produces, it opens a giant can of worms that cannot be closed. That means anything and everything you make with this tool is no longer fully yours, it contaminates everything and makes your work traceable. Who wants that?
I was already annoyed by the fact that Claude marked everything it did on my repos under its own account (I didn't ask for any of this), but now everything is invisibly marked, even the code. Not that I care that my writing would be watermarked, since I'd rather write my stuff myself, but code? No thanks.
Meanwhile I'm running a DeepSeek V4 Flash or Pro, or a Qwen3.8, and it writes my code without a peep. Resulting repos are clean, just the way I want them. No 'Claude' account, no watermarking, nothing. I won't be looking back after having tried these new models. Whoever makes good models that don't broadcast their maker will get my business.
This watermarking will simply push people more towards Chinese models. Keep pushing in the wrong direction Anthropic. Doing this right before an IPO is a great idea.
I guess I understand the complaint, but LLMs are already crap at writing, IMNSHO. And, yes, maybe this will make them marginally more crap, but in my mind we're talking the difference between a 30% grade and a 29%.
Some here are arguing that mechanisms used by LLM providers already derail the goal of "the very best, most precise words at every single decision point", therefore the author is misguided.
The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.
I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.
Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.
I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.
If the author has preferences on their "own writing" that conflict with Anthropic's, then they should actually write it themselves rather than paying Anthropic to do it. Private companies don't owe you anything, even less so when they're beholden to laws in foreign jurisdictions.
>Assume that there is a sequence of tokens, such that it is considered the absolute best by the author
You can't assume that because if that was the case he'd already know what sentence to write, because that's what that means.
The notion of a best sentence requires a final cause, an end to write to. By their very nature that's not how LLMs work, so you can't 'degrade' them on that front. They can't lose a property they didn't have.
People are very upset, especially in the arts, that Anthropic is changing the text to watermark it, but isn't that missing the point a little bit? They're not changing _your_ text whose every word you've carefully chosen for the exact effect, they're changing text that they're generating, i.e. text you've already chosen to give up control over. LLMs can't understand emotional nuance anyway.
The phrasing of the announcement implying that phrasing and diction don't change the meaning of text is insultingly dismissive of the whole field of literature, and I can see why people might take it as an afront, but the actual technology shouldn't have a negative impact as far as I can see. It seems to me that this one is more of a PR problem than something with real-world impact.
This is moronic. This is like being mad that the slot machine you think is lucky is occupied.
It's good to be the King.
And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.
This is the yet another embarrassing idea from the EU they came up with, if they could focus less on dumb things and more on providing real support to the development of technology maybe we could have more healthy competition that will eventually lead to better overall technology in the hands of everyone instead of acting like dumb chickens and not just being passively useless but also stopping the real labs from doing the real work into doing this bs
Watermarks are garbage because they may embed account id, IP address and deanonimize you. That's why we should be using open-weights LLM whenever possible.
I don’t disagree about open weights (though the enabling aspect there is actually open source inference, right?)
But it feels to me like you would need a hell of a lot of text to bury even a simple account ID. The nudges they are talking about are of the order of a handful of bits over several hundred words, I think?
This is the first post I've seen mention it. How traceable are the embedded codes?
There was an earlier instance of this here: https://news.ycombinator.com/item?id=48734373
I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this.
> The difference between watermarked and un-watermarked text will not be distinguishable to readers
https://www.anthropic.com/news/claude-text-watermark
Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here.
If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.
> The EU regulation, from my understanding, requires AI content to be labeled for human viewers.
How would that work? Claude appending " written by AI" to each of its messages? That would both be impractical and useless.
I think there are two separate requirements? One that if you post something like an AI video on the internet or anywhere else, you must label it as AI. And another one that AI providers must watermark their outputs.
If you get caught uploading watermarked media without the clear label, you're in big trouble, mister.
And it mandates that Anthropic put it in their TOS rather than make it law.
We were talking about text.
Which must also be marked as AI, I hope. (But I doubt that is the law, because the EU is cucked to big businesses interests)
I don’t understand how this works for anything but prose. Is that the point? In any code or structured output, there just isn’t the flexibility, and depending on how the user requests the output be constrained there is even less (“answer only True or False”). So is it just chat responses? If I ask the API to tell me a story about Alice and Bob then it watermarks it, but when I ask it some implausibly constrained thing like write a story about Alice and Bob with each word starting in rotation with the letters alicebob, does it try to do so and hope there are roughly équiprobable tokens regularly?
This could be why claude code has recently started to write reams of inane comments alongside the code it generates.
Yes, and it says that in https://www.anthropic.com/news/claude-text-watermark
I should have read that, it’s actually quite reasonable and I don’t really understand the objections in TFA having read it.
> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.
Doesn’t make sense at all in light of the actual approach, they’re just choosing a different RNG. It’s not like they’re corrupting it by flipping words.
Should add I don’t support the watermarking and requiring it is idiotic.
I don't understand Gruber's points either, I wonder if there is some fundamental technical misunderstanding. Does he think that the logits should be sampled from in a "pure" manner without introducing any other bias? Does he know that there's already a sampling temperature, and that most providers have probably moved on to sampling strategies other than top-k? Does he know that the word choices have already been altered irreversibly during RLHF which is how you get the obvious Claudism like "load bearing" and "seams"?
Perhaps it would be useful to publish examples of samples with/without watermark. I'd suspect that the variability from simply sampling repeated times would dwarf any semantic differences you'd detect with the watermark.
> I don't understand Gruber's points either,
Gruber's point is to bash the EU. He couldn't care much about anything else.
I think Anthropic should have put all the info into one blog post. Splitting it up is really confusing people.
> In any code or structured output, there just isn’t the flexibility
Variable name perversion incoming...
The author makes a lot of great points. I find it surprising that somebody who has such a nuanced appreciation for the subtleties of language would be in the crowd complaining about the watermarking policy. I expected this kind of complaints from mostly students interested in academic dishonesty, who generally don't have enough command over written language to notice the slight decrease in output quality.
Great, now Claude will sound even more generic. It’s not only annoying - it’s also dumb. We are all correct to push back on this load-bearing issue
I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156
Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:
"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"
A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:
"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."
Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:
"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."
EU is a capitalist union first and foremost. So their regulations try tobstraddle the thin line between "regulate to death and tell companies exactly how to do things" and "the companies can do whatever the hell they want".
Since most Americans are in the latter camp, anything that even hints at making companies responsivle for anything is viewed as being squarely the former.
Whereas most EU regs are "play nice, be reaponsible, behave like adults. If not, this can always turn ugly". Same here.
I will be happy to move off Anthropic given the chance. They are burning all of their good will.
The same absolute morons who gave us cookie consent strike again.
I swear, one of those days I will get into politics just to fight those two things, and the cottage industry of batshit crazy lawyers that gave birth to those things.
The cookie consent banner is not the EU's fault.
It's either don't track or ask for consent. The fact that the industry chooses to track is not on the EU.
Reminder: your favorite distilled model does not treat you, the customer, as an adversary and mess with your output.... May the free market win.
Wow, never has a single article revealed the incompetency of a tech writer.
gruber is really out of his element with ai commentary, I fully support the general skepticism but he's seemingly arguing against something he doesn't quite grasp
The objection seems to be that Claude will always write worse prose than a human writer, even if the writing Claude generates is understandable.
Yeah, John. We're all OK with that.
... If watermarking is a perversion of writing, what does that make 'writing' with an LLM?
Related:
How Claude's text watermarking works
https://news.ycombinator.com/item?id=49303350
I'll be honest, who fucking cares? Why would you use AI to write for you and then complain that people know AI wrote the code?? If you know people wouldn't like it, why even try!?
Now for the human generated watermark. Timez to addd the speelling mistakes, decreaze the usegage of big words and proper gramicaly usuage. Wish I was joking.
> the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text.
It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.
What is even the point of watermarking AI slop supposed to be? All it does is provide people with the false implication that anything which is not watermarked must not be AI-generated.
I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.
I keep seeing an irritating misconception in this space, which is that the alternatives chosen by these algorithms are supposed to mean the same things as what they're displacing. That's not true, and not how LLM generation works. Complaints that two different choices don't mean the same thing miss the entire point.
"Anthropic's . . . Claude is a Perversion of Writing."
FITFY.
I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.
> because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.
... and the opposite is also true, sometimes it will increase the probability of choosing the "best" word choice. So watermarking makes the LLM quality better then? /s
> My initial speculation was that maybe they’d hide invisible non-printing Unicode characters in the text.
Or just em dashes? /s
Anthropic should just pull out of the EU at this point. Europeans who really want to use it can VPN, and if they did they wouldn’t be able to hide behind their various comical tech laws.
Nanny state nonsense indeed.
As a non American/non European resident all I see from the Europeans are 0 contributions to software progress at any large scale while they surely do a lot of crying and huffing and puffing and demanding. Lots of complaining and rule making but not a lot of creating is a bad look.
Clearly Anthropic thinks its more profitable to comply and have access to the European market, but I'm sure you know better than the people who brought it to a $2 trillion valuation.
I didn't say anything about profitability. Since Anthropic is all about the good of mankind etc, ostensibly profitability isn't their number one priority.
Hi Gruber.
This (Anthropic's "watermark" stance, I mean) is so fundamentally ludicrous that I have assumed it is a (wholly insincere, but arguably pragmatic, at least from their perspective) attempt to deal with the EU and their latest misguided, ham-fisted attempt to solve a real-world problem by drenching the entire world with more regulatory slop[1].
The "watermark" can be trivially defeated, but may be enough to satisfy the letter of the law, and like many people here, I would argue that if you are letting Claude write for you, you've already accepted getting the literary equivalent of turd soup, so the harm is — or at least could be — fairly minuscule.
[1]: https://digital-strategy.ec.europa.eu/en/policies/code-pract...
(FWIW I have a more favorable view than most people seem to of the EU's efforts to at least try tackle problems like this — but predictably, the bureaucratic "solutions" they come up with don't work, but do make things objectively worse)
Respectfully, you are all missing the point.
Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly.
As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over.
It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.
You should submit an article about this instead of having your point buried in a comment section of an article making an unrelated argument.
I suspect there’s a niche market for software that records your writing process to help you avoid being falsely accused of using AI to write.
tl;dr:
> It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of (watermarking)
And an example of the impact of watermarking on word choice [0]:
> The results of the study were quite [important || significant || substantial || notable]
The meaning of the sentence to changes slightly even in just this tiny example. Imagine the degradation when applied across an entire response!
[0] https://declaude.org/watermarking/
Not telling someone you used AI is a perversion of writing. Also agree that an AI proofreader should not claim authorship, but in most other cases, the AI is not reading your mind, it's only watermarking its own usage, and we kind of need more of that.
I recently found there was no website covering a historical subject, with sites begging to be visited. The information was scattered and not presented in the form people would find most useful (a map).
A couple of days later it was up with the most comprehensive review of the available info summarised and referenced ready for a human researcher to explore.
This was only possible due to AI. It would have taken me weeks to chase it down and summarise it, so it would never have happened. Let's get off our high horse about AI writing.