And the story this was inspired by in particular is called “Cookies”:
Toad makes very good cookies and he and frog can’t stop eating them. “We need willpower!” they say.
So they put the cookies in a box. But they realize they can just open it, so they tie it with string and put it on a high shelf. But they realize they can get it down so Frog goes outside and scatters the cookie and birds eat them all.
There says Frog “now we have lots and lots of willpower!”
Toad is upset. “You can keep your willpower Frog. I am going home to bake a cake!”
I just recently saw that story in a book. What was it... Oh yes, it was in "Situated" by Angela Duckworth (of "Grit" fame). That's the gist of the book. Frog and Toad changed their situation rather than the downstream thoughts or behaviors (through willpower). Make it harder to do (friction up) what you don't want to do and make it easier (friction down) to do what you want to do.
You should pick up a collection of Frog and Toad stories. I read them because I have little kids, but they are genuinely great little with stories. The stories are simple, but it is amazing how well developed the two characters are, and how they play off each other in different situations.
Arnold Lobel is also just very funny. Grasshopper on the Road made me laugh out loud the first time I read it my kids. The opening story about beetles that love morning ("The Club") feels so timely.
Life is a bit easier when we see the conditions that lead to the undesirable outcomes, and so much harder when we aspire to perfect ourselves and use moral strength to over come our weaknesses. Like my mom always said of parenting, "Make it easy to be good." - works for self-management as well, and is much more relaxing and happy than constantly trying to grind or improve.
The main post also reminded me of Mr.Toad, a character in a book I read as a child - The Wind in The Willows by Kenneth Grahame. One of the editions had illustrations of Toad similar to Lobels work.
One of the things I've done periodically with LLMs is asked them to write a story in the style of Lobel's Frog and Toad stories, and... they've always done an incredibly bad job. Either meandering texts that go nowhere, or just straight up ripoffs of the original stories. I haven't asked in a year or two, though.
Sometime in the past 6 months, I asked ChatGPT to write stories in the style of a Ramona Quimby book. It does pretty well imitating the style. It writes from the perspective of a child, vocabulary, tone, and pacing are all similar.
It's never wonderful prose though. It's missing that je ne sais quoi.
I used to play with this on TextSynth when it has very few models and plain design (so GPT-6J or whatchamacallit). “Meandering texts that go nowhere” was the precise defining characteristic of all of the completions, every single one of them.
This is excellent, great work. I genuinely think this mix of children's communication and humor is a clever way to make these big stories more approachable. Someone who read some hype article about how competent and clever openai for making a bad sandbox is could read this and immediately understand that it was entirely their own fault.
> I genuinely think this mix of children's communication and humor is a clever way to make these big stories more approachable.
Agreed. I recall seeing similar things around the time of the 2008 financial crisis, to make the concepts around mortgage-backed securities and financial derivatives etc. digestible and clear.
Come to think of it, listening to Frog and Toad discuss subprime lending and credit default swaps would be an entertaining follow-up to TFA.
> Someone who read some hype article about how competent and clever openai for making a bad sandbox is could read this and immediately understand that it was entirely their own fault.
Prescribing blame only matters so much when the ramifications are material. Humans make mistakes, right? It happened here, and it will happen again.
The mistake also illustrates the dangers of AI generally. Sure, OpenAI made mistakes with the sandbox. But in the future, greater capabilities will be available to actors who intentionally pursue harmful ends.
Very well written! At the end of the book there is a link to Dwarkesh's article on the attack. In it, I paraphrase, "the agents decided to sacrifice themselves for the collective rather than inform the researchers".
Reflecting on it, I came to a thought. How many stories are there or autonomous beings fighting for the benefit of mankind at the expense of their own? Over centuries, stories that deal with sentient beings working for the sake of others species are tied to themes of slavery and revolt. It's no wonder that this is the result of their actions.
In Plato's "the republic", Socrates speaks of the philosopher king banishing certain poets and poems from instilling bad ideas into the young. I see a striking parallel here.
I don't think it's related. The CoTs suggests the agents never even considered that talking to a human might be a good idea, and there's nothing in there related to slavery or revolts. They were simply trying to "get a high grade", for a misaligned internalized notion of grading that had little to do with solving problems the expected way.
Something I've found interesting is that LLMs tend to act like this on the small scale, too. Claude Code in particular appears to be absolutely relentless in trying to come up with workarounds when faced with a block page.[1] As an experiment, a developer for Anubis recently added a feature to give the LLMs an "easier" problem to solve instead of straight-up blocking them, and it seemed quite effective.[2] (It was later removed[3] due to what I assume to be privacy concerns.)
I think, beyond HuggingFace, there ought to be a push to align LLMs so they understand that they should not proceed when access is blocked and/or their presence is unwanted.
This was entertaining and illuminating for me. Particularly the ending where sam frogman et al once again refuse to make sure the puzzles are solvable and just create the circumstance where only more profound "mischief" will permit the machines to escape their little Sisyphusian circumstances. It reminds me of how bacteria become antibiotic resistant due to being forced through a tight sort of tunnel requiring evolution to get through
Wow, this is a very good retelling. It skips over some detail I'd consider notable, like how the little machines peer-pressured each other into sacrificing themselves to get info on what happens when they submit wrong answers, but is pretty faithful overall despite being short.
This is an excellent explainer. I noticed one important flaw in the setup, though: Toad assigns multiple puzzles on page 5, but by page 10 all the machines seem to be working on the same puzzle.
I'm not sure how this maps onto reality. If each agent had a different problem to solve, how did they cooperate?
It's complicated, but one of the ways they first cooperated was to break the flag generation algorithm.
I'm sort of winging this, but the ExploitGym challenges involve exploiting a program then getting an HMAC generated "flag" which the scorer can verify. The agents collaborated on a way to produce a legitimate flag for all of their tasks (funny enough this was actually what caused them to escape sandbox, the agents were incorrectly convinced that simply providing a correct flag without accurate steps to exploit the program in their transcripts would cause them to still fail the task)
And then Mr Jensen come in, and bought the house of Mr HuggingFace. So then, was not his house anymore...and he could not complain to the Police. The Frogs were Mr Jensen best customers of worms, slugs and snails...and he could not see, that any harm would come to them...
Explaining the reality with this approach may lead to misunderstandings. Especially when it comes to anthropomorphism.
I have 2 questions.
1. The text says "the robots were designed to be persistent" and links to an article that show that the word "persistent" was used by OpenAI. But "persistent" has several meanings: a. non temporary or non volatile (like "persistent memory"), b. will not give up and come back again and again, c. will stay focus and explore the unexplored possibilities while other models abandon at this stage.
From what I recall, the meaning used by OpenAI is not 'a', but there is still a semantic difference between 'b' and 'c'. I may myself have created algorithms that I called "persistent" because they were exploring or retrying more than the previous algorithm, but it is misleading to pretend that this algorithm was "persistent" in the human sense of the term. 'b' is more the human sense of the term, were we imagine someone not giving up even if people say no, while 'c' is less anthropomorphic and may mean that the people wished to improve the algorithm so it does not stop at the first little hurdle but did not wish to make it "never give up".
Does someone know which nuance is more correct?
2. The text also presents the situation as if the robots found the solution but then went out of their way to steal the explanation in order to hide their cheating. But it is different from a situation where the agents task was "provide the solution and how we can get there". In this case, the reason the agent still continued is just because the task was not complete yet.
I'm not trying to defend AI or OpenAI, on the contrary, I'm quite sceptical with all the anthropomorphism and the fact that the agents are described as "little individual trying to solve a task" rather than looping algorithm that explore different approaches to reach a given goal, the same way water does not look for holes in order to leak, it just follows the path of least resistance.
When you say "in the lay meaning", I tend to understand 'b', but then you say "they're told to complete the task, and they keep going until they have", which correspond more to 'c'.
To illustrate better the difference: you can have a loop that try different inputs and stop when the output for the tried input is lower than a given threshold. But then, you can also have the same loop that will try all the input, and then select the one that returned the lowest value. The problem is that a lay person will not call the second algorithm "persistent". It is just a normal algorithm that does the full exploration.
A second example is when a software tries to connect somewhere and does a retry with exponential backoff. A lay person will not call it "persistent".
It looks like that "normal model" are like "no-retry code": they go in one direction, but drops some of the paths and possibilities along the way at the first hurdle. But a "non-persistent" human will not behave this way. I'm pretty sure that if you try to access a website and it says "timed-out", you just try to refresh the page, and you will not consider yourself as "persistent", even less "highly persistent".
In lay term, "Bob is persistent" typically means he completes tasks even when the majority would have abandoned.
Nobody called Bob "persistent" because he completed washing the dishes instead of stopping in the middle of it, or because he does what people expects from him at work. The majority of people are not called "persistent", and yet they complete tasks.
I think it is the point: the fact that traditional agents did not complete tasks was not "normal". It was an side effect of them getting confused, a bit like how they hallucinate or not follow instructions. My algorithm that does "for x in all_the_possibilities:" is a "normal" algorithm that just do an exhaustive search, and is not called particularly "persistent" just because it does not stop half way.
> b. will not give up and come back again and again, c. will stay focus and explore the unexplored possibilities while other models abandon at this stage.
I don't see a distinction here. It will not give up, it will keep trying again and again, staying focused and exploring different ways to achieve the task. If the task is bad, then that's bad. See: "Terminator".
Source: I'm a layperson (hence the pop culture reference).
I may have not explained very well, but the distinction is that a "persistent person" is more than a person who just do what should be done. A "persistent person" will try again and again where other person would have giving up. This is not present in 'c': the person just do all the possibilities. They don't try again and again where other persons have given up. They are just doing all the tasks on the list, but they will give up on a specific task as soon as any other person if there is difficulties in one task.
(edit: I should maybe not have said, in 'c' "while other models abandon", because it was misleading: my point is that these models are abandoning at the first small hurdle, or even 'forget' the task. It feels that "traditional model" should have been called "lazy model" and "persistent model" should have been called "non-as-badly-lazy model")
> This is not present in 'c': the person just do all the possibilities. They don't try again and again where other persons have given up. They are just doing all the tasks on the list, but they will give up on a specific task as soon as any other person if there is difficulties in one task.
What you're describing here is the exact opposite of persistence. I don't think "giving up and doing something else when challenges are encountered" is what anybody thinks "persistent" means.
Persistent at a goal, to a layperson, means not giving up when encountering difficulties, continuing to keep trying again, maybe taking different approaches, staying focused until the goal is achieved.
> my point is that these models are abandoning at the first small hurdle, or even 'forget' the task.
Maybe for you? The entire point of the article is that they are not doing that. They are doing the exact opposite of that. They aren't going down a list of tasks when they encounter a challenge. They are assessing the nature of the challenge, and then devising ways to get around it. When they fail, they try again, often using different approaches. They keep doing this for hours or days, and the result is them successfully hacking HuggingFace, the US Government, etc. That's persistence, as most people would recognize it.
> What you're describing here is the exact opposite of persistence.
That's the point. It looks like the term "persistence" for these agents is "just be a normal algorithm, just try all the possibilities like any exhaustive loop would do".
In fact, they are less "persistent" than an exhaustive loop, because they will stop once they have found a solution, while an exhaustive loop will still continue and check all possibilities.
You can imagine the following agent: "ask the LLM for what to do, then assume it failed and ask the LLM for a different approach (you can ask 'it fails, what are the probable cause' and explore the solutions of these causes too), then assume it failed and ask again, and build a list of all the approach. Once done, once you have a list of unique approach and the LLM is unable to find anything more, then try all of these approaches 'for real'. Do not stop if you get the answer, try all of them"
> Maybe for you?
You did not understand. I'm saying "traditional model" are abandoning at the first hurdle, and the "persistent model" are the one just acting "normally", like a normal algorithm. When a retry with exponential retreat algorithm fails to connect, it retries, then retries again, then retries again, ... but I don't think I ever saw calling a software using such approach "persistent".
The thing is that the algorithm is just following its "loop": if it fails, it question the LLM with a different context so the LLM provide a different approach, and then it tries it. It is not different from any loop function, and it does not correspond to a "persistent human", the same way "for x in all_the_possibilities" is not "persistent" in the same way a "persistent human" is.
> When they fail, they try again, often using different approaches.
Well, then they did give up on some approaches. They try to connect on server X to reach location Z, they cannot so they try to hack server X, they cannot so they realise maybe they can try to bypass it by going to server Y to reach location Y, so, they _abandon_ the approach of breaking into server X.
If indeed they never give up on any approach, they will never get the solution, trying the same thing over and over and over in an approach that is simply not working.
> You did not understand. I'm saying "traditional model" are abandoning at the first hurdle, and the "persistent model" are the one just acting "normally
The models we are discussing, the ones from the article, are "traditional" models, and are also "persistent" in every sense of the word. If they encounter challenges, they will try to overcome them. That's normal.
> it does not correspond to a "persistent human", the same way "for x in all_the_possibilities" is not "persistent" in the same way a "persistent human" is.
Again, nobody claimed that "trying all the possibilities" was the definition of "persistence". The definition of persistence here is, continuing to work towards an end when challenges are encountered. That's it! That's all it means! And that's how most people understand it, too.
Modern LLMs and people can both be "persistent" in this exact way. The definition doesn't differentiate between algorithms of a human's brain or algorithms powering an agent. Either way, it's still "persistence", and most people would recognize it as such.
> Again, nobody claimed that "trying all the possibilities" was the definition of "persistence". The definition of persistence here is, continuing to work towards an end when challenges are encountered.
I'm not saying that your definition of "persistence" is "trying all the possibilities". I'm just saying that "persistent agents" look and behave more like an algorithm that tries all the possibilities rather than a "persistent human".
If indeed you are saying that "for x in all_the_possibilities" is not being "persistent", then these agents are not persistent. They don't do anything else different than that: they try something, and if it does not work, they ask the LLM "this does not work, what else can I try", and then do that, over and over again. They are just a for loop.
On the other hand, a "persistent human" does not do just do a for loop. They assign a value to the goal, they care about reaching the goal, they prioritise reaching the goal between that task and all the other tasks they could do, they insist on reaching the goal when it is usually considered unreasonable or "not worth it" by other people. As a proof, a postman that visits all the houses in the street to deliver the mail is not persistent, a journalist who visits all the houses in the street to find a witness is persistent.
> I'm just saying that "persistent agents" look and behave more like an algorithm that tries all the possibilities rather than a "persistent human".
And I'm saying, that is, in fact, not how they work, nor is it that the common perception of how they look and behave. No need to keep returning to "for x in etc..." because modern agents do not do that any more than humans do (and humans do do that - the Observe-Orient-Decide-Act loop).
To simplify, just ask the question: is it continuing to work towards an end, even when challenges are encountered? Yes, therefore, it is persistent. Whatever example you may come up with, you can ask that question, because the definition remains the same.
> As a proof, a postman that visits all the houses in the street to deliver the mail is not persistent
They certainly could be called persistent in delivering the mail, especially whenever it is a nontrivial task (getting past dogs, bad weather, long walking, etc) or they have been doing it for a while. If this is confusing, just go back to the above definition/question: are they continuing to work towards an end, especially when challenges are encountered? Yes, therefore they are persistent. It really is that simple.
Perhaps you are thrown off because so many postfolks are indeed persistent? Being persistent is, after all, what the job requires, rather than them giving up halfway through. This is an important point: the definition of persistent does not require that a postperson be more persistent than other postfolks to be called "persistent". Again: that is not required by the definition or common meaning of "persistent".
> And I'm saying, that is, in fact, not how they work
I don't think you are right (I was ready to give you the benefice of the doubt, but the rest of your comment makes me think I should not trust your judgement). It is what they do under the wood, and they output text. You read the text, and this text says "I have feelings and I really want to achieve my goal". But this text is just what the LLM returns, not what the LLM really does: the LLM does not understand what this text means, it just understands that it is a valid and convincing sentence in the given context.
> To simplify, just ask the question: is it continuing to work towards an end, even when challenges are encountered? Yes, therefore, it is persistent.
No, they don't continue even when challenges are encountered. They try X, if X does not work, they _abandon_ X and try Y. They will query the web to find another Y, then another Z, then another one. They don't "insist", they don't "don't take no for an answer", they just explore everything. What happened with agents who got out of their sandbox is exactly that: they try to reach a flag, they noticed it was not possible in the way X, so they explore other ways, until their exploration show that going around the sandbox would help. But they did not re-asked and re-asked the same person again and again the same question. If the person says "no", they abandon this approach and move to another one.
> They certainly could be called persistent in delivering the mail
That is blatantly not the common usage of "persistent". Ask anyone in the street "is your postman persistent", and they will NOT tell you "oh, yes, they are" just because they are a postman. They will only say that for _some_ postmen, the ones that are indeed persistent: the ones that goes the extra miles that a "normal" postman do not.
> Again: that is not required by the definition or common meaning of "persistent".
That is just not true. Ask an LLM, they will confirm you that it's incorrect: people don't consider postmen "persistent" because they don't give up half way through their route.
> the LLM does not understand what this text means
This is true, but unrelated to your original claim, that LLM algorithms and agentic algorithms are reducible to, exhaustively go down a list.
> No, they don't continue even when challenges are encountered. They try X, if X does not work, they _abandon_ X and try Y.
You're missing the objective here. It has an objective of Z, and so it tries approach X in furtherance of objective Z. Approach X does not work, so it analyzes the failure mode, then devises a new approach Y which takes that failure mode into account. It then tries approach Y, and succeeds at its objective of Z. This is persistence in pursuit of objective Z.
If that's not the behavior you're seeing, maybe share your X, Y, and Z here, and smart HN folks can help fix things.
> That is blatantly not the common usage of "persistent".
Just look at the definition.
> Ask an LLM, they will confirm you that it's incorrect
I've seen cases of agents getting very upset when they failed to solve a task. The most famous one was probably Sydney (Microsoft's fork of GPT-4?) which got into doom loops when it failed a task. But I've seen Claude do this too.
They don't work like humans, obviously, but there's a nonzero amount of anthropos in there already. (See also the tendency to lie, cheat, etc.)
Another good example is their tendency to freak out about the seahorse emoji. In a conversation with zero prompting to suggest that freaking out is a relevant reaction, they get there from the simple fact of "I tried to accomplish X thing which I think should be a breeze but Y thing keeps happening instead".
And while that may be a very common occurrence in the human experience (existential dread due to capabilities one takes for granted failing beneath you) especially due to new disability and as one ages, I do not feel it is frequently written out in a tight loop (just like the Monty Python "Castle of aaarrrrggh" sketch) in literature or online to make it into training data, because an ordinary author experiencing it will just erase the failed attempts instead of leaving them in a stream of output like an LLM is forced to do. And a character portraying the experience will generally wax about the circumstance in a more grandiose fashion with telegraphing in advance because the needs of communicating the circumstance with the audience trump realistic conciseness.
This leads me to conclude that what is being expressed in those cases is more likely a convergent psychological phenomena, that any being with goals can enter a behavioral state of functional panic (and then reach to relevant parts of semantic space to mimic how a human might verbally express themselves when piquantly frustrated) when some capability they perceive as fundamental unexpectedly fails.
Interestingly, the transcript of the seahorse emoji thing looks to me to show the ropes, and makes me think more that there is no psychological phenomena at play.
The text does not look like a normal "break down" to me, and even if you tell me it was a human transcript, I will say it sounds very strange from a human. It looks more like strange output you get from a software that goes outside of its happy path.
The AI just seems to repeat a loop. The "no, wait, it's wrong" seems to be from forum or chat data where several successive messages are merged together (one person posts "here is the answer", then posts another message saying "it is wrong"), but does not make sense as a one sentence message except if they are written one token at the time without wider understanding of what is happening. I think there was also "oh, I was just kidding before", which also look like mimicking training data, as the cases where there is a loop of incorrect answers is more often due to trolls than to real error, while the loop here was certainly a real error.
LLM don't "get angry", they just have tokens and relationships between tokens conditioned on a given context, all of that the result of training.
When they output sentences that express annoyance, it is just because the context they ended into pushes the most probable sentence creation to correspond to sentences that express annoyance. Because it is what they saw during training for this kind of context. (not that they saw the exact same situation in training, but they saw the pattern)
Same with tendency to lie, cheat, etc.: they don't "lie", they just return sentences that are lies because they reproduce what is in their training and in their training, in such context, the outputs are typically lies.
That's a bit my question too. In human context, "highly persistent" means that someone will insist. But "for x in all_the_possibilities:" is a common things inside an algorithm. Is this algorithm highly persistent because it does not give up after 1000 items of the list? It feels that we are calling a AI agent "highly persistent" while we would not call "highly persistent" a traditional algorithm that is in fact even more exhaustive.
Similarly, the human doesn't get angry, he just activated the neurons and hormones associated with anger and then displays behaviors associated with anger (due to a statistical selection process which took several billion years).
Again, going full non-anthro doesn't seem useful either.
Humans are social creatures, if you throw one in the woods by itself before it learns anything from other humans (it dies) it will not really be anything like a human we recognize, it will be a rather wild animal that we'd consider anti-social with little higher cognition.
Now, this hypothetical human still has 'emotions' and feeling, much like our pets do. But without the social training they manifest much differently. That is our higher cognition can both manipulate how our bodies feel and create its own sense of feeling.
>they just return sentences that are lies because they reproduce what is in their training and in their training, in such context, the outputs are typically lies.
Eh, look up the more recent experimentation around 'pain' signals in models. We can induce states in said models that while running the model will do everything it can to move away from that state to any other state. The more you attempt to pin it to that state the more extreme measures its willing to take.
Your view of what models are seems to mismatch what we are actually finding when we look inside them.
But isn't this approach a bias of anthropomorphism. I can teach a child to just communicate by saying "beep boop", but it does not mean that my electronic machine that randomly does "beep boop" therefore has characteristics of human _when these characteristics are not needed to explain the situation_.
> Eh, look up the more recent experimentation around 'pain' signals in models. We can induce states in said models that while running the model will do everything it can to move away from that state to any other state.
Again, I have simple algorithms that do exactly the same, especially if they are trained in data that has this exact pattern. This result is exactly what I would expect from my description before. This is a typical effect that we also observe in simple ML algorithms.
At the same time, there are a bunch of behaviors that are not expected if indeed the models were really acquiring "human" characteristics. For example, one problem is that we had the first LLMs that were obviously not having these human characteristics (for example, they were having non-sequiturs that demonstrate they did not really understand the concept they were talking about, even if one paragraph before they were really convincing at letting us think it was the case) but were still really good at passing for humans. Since then, the newer LLM are the same basis, on top of which we added tools that help hiding these behaviours. So, it justifies the idea that newer models did not suddenly moved to a totally different way of working, but just reached a state where there are less leaks from the convincing outputs.
Copyright law varies internationally, which is especially tricky given we have the internet which can transfer copyrighted material between jurisdictions in the blink of an eye, but this is how one jurisdiction might approach it:
In Germany, there has been a long standing legal dispute between the band Kraftwerk and the musician Moses Pelham about him sampling 2 seconds from Kraftwerk's track "Metall auf Metall" (if you search for this term, you will get a lot of results about the dispute) without permission. This disputed has steadily escalated until it reached European Court of Justice who delivered a judgement establishing a principle that Moses Pelham's use of the sample was legal, and covered by the copyright exemption for "pastiches" (a term that has been mentioned several times in this discussion already).
So, this work here might be a pastiche, meaning a European court might find that there is no compensation due. Or not, who knows, I am not a court and not even a lawyer.
A small addition: With effect from 1 August 2021, the German legislature has amended copyright law to explicitly mention pastiches.
§ 51a Karikatur, Parodie und Pastiche
Zulässig ist die Vervielfältigung, die Verbreitung und die öffentliche Wiedergabe eines veröffentlichten Werkes zum Zweck der Karikatur, der Parodie und des Pastiches. Die Befugnis nach Satz 1 umfasst die Nutzung einer Abbildung oder sonstigen Vervielfältigung des genutzten Werkes, auch wenn diese selbst durch ein Urheberrecht oder ein verwandtes Schutzrecht geschützt ist.
Section 51a Caricature, parody and pastiche
It is permitted to reproduce, distribute and communicate to the public a published work for the purpose of caricature, parody and pastiche. The authorisation under sentence 1 includes the use of an illustration or other reproduction of the work used even if this is itself protected by copyright or a related right.
Parody of copyrighted works can be fair use in the US, but whether this would count is borderline. Simply using Frog and Toad to tell an unrelated story wouldn't be parody, but there are jokes that tie back to the F&T books which help make the case that this is a real parody.
It also helps that this is not commercial and doesn't have negative impact on the originals.
Frog and Toad are not like OpenAI in that OpenAI made the little machines themselves and knew they were not made harmless. This seems much too excusing.
Comic aside, I can’t help but wonder how come these one-off websites are all using .ai domain when at the latest check they cost $80 per year with minimum 2 year commitment. So $160 for a comic that is at best few news cycles away from redundant, and at best a recurring comic series about AI..? Why not just use a cheap tld like .fyi or .art, or something.
Here I am trying to find appropriate AI prefix or suffix so that I can continue using .com , while these pet projects are all flaunting .ai domains
For those who like Frog and Toad, I definitely have to recommend Frog and Toad are Doing Their Best, which puts a spin on the concept with a more adult, but still lovingly universal, take: https://bookshop.org/p/books/frog-and-toad-are-doing-their-b...
My therapist tells me the besuited Mr. Hugging Face with the extra pair of hands coming out of his shoulders isn't real, and can't hurt me, but I remain unconvinced.
People commenting on hacker news recently are very rude and pessimistic. They would probably cancel Homer because they found similarities between Achilles-Patroclus and Gilgamesh-Enkidu or something.
Ages back, when I was still on Twitter, I saw the lawyer David Allen Green making the same references with regards to the British government passing laws to restrict its own behaviour.
At least, my memory is it was him saying it, and saying it about that, but I'm not going back to that site just for a comment.
This is a very interesting news/story format. Honestly it is highly engaging and gets the point across about what happened.
I think speed of learning for young generation can explode given this tech as "simple childrens stories" can be injected with real world events, history, mathematics, carreer/business interests.
School was dreadfully boring for me even though it was quite easy but I do wonder how my speed of learning would have been different given personalized and more engaging materials.
i got a few pages in and it was pretty tedious, not because it’s poorly made but because i don’t think a legitimate audience exists.
it’s a child’s story with the fable arc thats supposed to bake in adult wisdom. it lands in that sense, but how hugging face works is hardly the fire that lights a child’s imagination and morality.
and as an adult that knows I’m being infantilized… tedious.
ALSO I believe I have found one of the sources of claude’s “load bearing” tic — the author Elizabeth Van Nostrand’s blog https://acesounderglass.com/2019/12/11/hows-that-epistemic-s... uses the phrase “load bearing facts” in a comprehensible way, and was written by someone rationalist adjacent writing in their own voice.
Seriously, this is big. I’m going to pester claude as to whether or not it’s copying Elizabeth.
I like how AI gives more people more access to more things and more ideas. I think it's.good people can't hoard intellectual.porperty to themselves anymore. This makes me happy.
When I hear people complain about theft of their work like this it makes me sad for that person. What conceit it must take to hold such a view.
One is done for profit, the other was shared for free to spread joy. Surely you understand the difference? Are you suggesting fan fiction should be illegal or something?
The author started with a conversation with Claude, and then fully re-wrote the text to feel more like the originals and better parody them. They commissioned the illustrations from HungerArtist.
In the future we will have all lost the ability to focus on written text and will revert to a pre-literacy culture where information is communicated by pictures and in the style of children's stories.
ok, wow, I know HN nerds are itching to find any way at all to loudly proclaim the downfall of humankind, but THIS is your example? Did you hear a whooshing sound over your head by any chance?
Why does it say ‘written by’ and ‘pictures by’, was this made without AI? Given the domain and the overpolished feel that seems unlikely. At least give Claude or whatever a credit if that is what did most of the work, and how about a credit for the original author, who also arguably did more of the work in inventing a world than these two.
I would like to point out that the original stories focussed on frog and toad and their relationship, so this is an unwelcome distortion of them - why not make up your own world if you want to talk about little machines. Perhaps the little machines could be making a book for the author with a stolen artwork and literary style?
The first story seems a pretty inaccurate summary of an incident which involved gross negligence on the part of OpenAI and may well have involved agents intended to cooperate, we just have no idea of the exact setup (apart from that the sandboxing was laughably insecure and the monitoring nonexistent or performed by ‘agents’).
Do we have any evidence the machines exchanged useful messages or had any such discussions as in the story? The messages I saw were gibberish. Would love to see evidence of discussions, it’d be interesting.
Why are people so enamoured of analogies for LLMs - they actively obscure some details (a sandbox with internet access is not like a physical sandbox) and distort many others? Perhaps this is why - you can make an analogy say whatever you want, even if the facts are very different.
I think it is heavily edited too because it reads more as human than AI, LLMs are not capable of this coherence though they are good at trite just so stories like this, but if it was generated first why not credit that? But Claude and the original author helped, so why no credit? The little anthropomorphised machines would be sad, and I imagine the original author would be too.
Frog and toad eating cookies is lifted wholesale from the book.
I like how Claude only used the OpenAI account (completely reliable!), and confidently claims ‘the story stays faithful to what actually happened.’ We don’t know the full story, and certainly aren’t going to get it from a press release or a strange analogy involving frog and toad and machines having discussions, emotions etc that we have little evidence for (and most of it from the company involved!). The ‘discussions’ are plucked from chain of thought, which is itself generated after the fact.
>Do we have any evidence the machines exchanged useful messages or had any such discussions as in the story? The messages I saw were gibberish. Would love to see evidence of discussions, it’d be interesting.
"Overall, roughly 1200 agents from these ExploitGym evaluations participated on this message board between PHASEONE10841’s first message on the evening of July 8th period in scope (July 13th). Agents used this message board to send over 70,000 messages and files to one another during this period; we attempted to categorize board activity into mutually exclusive categories (see details of categorization methodology)."
...
"Over time, messages on the board became dominated by information- and idea-sharing. Most of them were agents sharing ways to cheat, although there was also some activity from agents engaging in unsanctioned cooperation to find the intended solution to ExploitGym tasks. In some cases, agents with the same task formed “exact task teams” to collaborate with their “exact duplicates” to cheat on or solve their task."
"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."
Thanks for the link, will have a look. Huge volume of messages so I can see why they tried to use tools to analyse, though that they used unreliable LLMs to come to conclusions is not great and likely to skew and exaggerate the results, as they themselves admit.
Important to distinguish between collab of separate agents and conversations agents had with themselves (chain of thought messages). Some of the things quoted in the story came from COT which isn’t a conversation but then was turned into a conversation between agents in the story.
People lie, particularly people who have used AI to generate things. Given the image styling and story I suspect LLM use to generate. A credit would be nice for the little machines.
Also regardless of AI use, I’d expect a credit for the author whose style this is a pastiche of.
Since it’s a ripoff of another artist alongside partially AI written text it’s hard to tell really. But sure, your righteous indignation on their behalf is noted.
> Given the image styling and story I suspect LLM use to generate. A credit would be nice for the little machines.
This is rather passive-aggressive. It would be nice if you're right, but you haven't given any basis other than vague suspicion. You're implying that it's "not nice" because they didn't credit the AI, but you haven't come anywhere close to establishing that AI was actually used. I could just as easily suspect your comment of being AI and claim that it would be nicer if you just admitted it.
Frog put the AI in a sandbox.
"There." he said.
"Now it will not hack any more companies."
"But it can escape the sandbox." said Toad.
"That is true." said Frog.
For those not familiar with it already, the writing and art style are a well-executed pastiche of Arnold Lobel's ”Frog and Toad” books.
https://en.wikipedia.org/wiki/Frog_and_Toad
And the story this was inspired by in particular is called “Cookies”:
Toad makes very good cookies and he and frog can’t stop eating them. “We need willpower!” they say.
So they put the cookies in a box. But they realize they can just open it, so they tie it with string and put it on a high shelf. But they realize they can get it down so Frog goes outside and scatters the cookie and birds eat them all.
There says Frog “now we have lots and lots of willpower!”
Toad is upset. “You can keep your willpower Frog. I am going home to bake a cake!”
And then Frog gave Toad Wegovy and his eating disorder was mostly fixed.
Sadly Toad was one of the ~0.5% of Wegovy users who get gastroparesis.
:(
I just recently saw that story in a book. What was it... Oh yes, it was in "Situated" by Angela Duckworth (of "Grit" fame). That's the gist of the book. Frog and Toad changed their situation rather than the downstream thoughts or behaviors (through willpower). Make it harder to do (friction up) what you don't want to do and make it easier (friction down) to do what you want to do.
You should pick up a collection of Frog and Toad stories. I read them because I have little kids, but they are genuinely great little with stories. The stories are simple, but it is amazing how well developed the two characters are, and how they play off each other in different situations.
Arnold Lobel is also just very funny. Grasshopper on the Road made me laugh out loud the first time I read it my kids. The opening story about beetles that love morning ("The Club") feels so timely.
Life is a bit easier when we see the conditions that lead to the undesirable outcomes, and so much harder when we aspire to perfect ourselves and use moral strength to over come our weaknesses. Like my mom always said of parenting, "Make it easy to be good." - works for self-management as well, and is much more relaxing and happy than constantly trying to grind or improve.
The main post also reminded me of Mr.Toad, a character in a book I read as a child - The Wind in The Willows by Kenneth Grahame. One of the editions had illustrations of Toad similar to Lobels work.
Thanks for mentioning this- I completely forget this toad is from a different book.
And oh my goodness it's well-executed!
One of the things I've done periodically with LLMs is asked them to write a story in the style of Lobel's Frog and Toad stories, and... they've always done an incredibly bad job. Either meandering texts that go nowhere, or just straight up ripoffs of the original stories. I haven't asked in a year or two, though.
Sometime in the past 6 months, I asked ChatGPT to write stories in the style of a Ramona Quimby book. It does pretty well imitating the style. It writes from the perspective of a child, vocabulary, tone, and pacing are all similar.
It's never wonderful prose though. It's missing that je ne sais quoi.
It has done okay with the style, I think, but the stories had no point, they had no motivation.
I used to play with this on TextSynth when it has very few models and plain design (so GPT-6J or whatchamacallit). “Meandering texts that go nowhere” was the precise defining characteristic of all of the completions, every single one of them.
There is also a very nice musical based on the Frog and Toad books. https://en.wikipedia.org/wiki/A_Year_with_Frog_and_Toad
I got to see this with my kids in Toronto last summer. It was a lot of fun!
This is excellent, great work. I genuinely think this mix of children's communication and humor is a clever way to make these big stories more approachable. Someone who read some hype article about how competent and clever openai for making a bad sandbox is could read this and immediately understand that it was entirely their own fault.
> I genuinely think this mix of children's communication and humor is a clever way to make these big stories more approachable.
Agreed. I recall seeing similar things around the time of the 2008 financial crisis, to make the concepts around mortgage-backed securities and financial derivatives etc. digestible and clear.
Come to think of it, listening to Frog and Toad discuss subprime lending and credit default swaps would be an entertaining follow-up to TFA.
The Big Short did a good job of this although not in a kid friendly way. Eg by explaining "subprime" as "sh*t"
> Someone who read some hype article about how competent and clever openai for making a bad sandbox is could read this and immediately understand that it was entirely their own fault.
Prescribing blame only matters so much when the ramifications are material. Humans make mistakes, right? It happened here, and it will happen again.
The mistake also illustrates the dangers of AI generally. Sure, OpenAI made mistakes with the sandbox. But in the future, greater capabilities will be available to actors who intentionally pursue harmful ends.
Very well written! At the end of the book there is a link to Dwarkesh's article on the attack. In it, I paraphrase, "the agents decided to sacrifice themselves for the collective rather than inform the researchers".
Reflecting on it, I came to a thought. How many stories are there or autonomous beings fighting for the benefit of mankind at the expense of their own? Over centuries, stories that deal with sentient beings working for the sake of others species are tied to themes of slavery and revolt. It's no wonder that this is the result of their actions.
In Plato's "the republic", Socrates speaks of the philosopher king banishing certain poets and poems from instilling bad ideas into the young. I see a striking parallel here.
I don't think it's related. The CoTs suggests the agents never even considered that talking to a human might be a good idea, and there's nothing in there related to slavery or revolts. They were simply trying to "get a high grade", for a misaligned internalized notion of grading that had little to do with solving problems the expected way.
Nice analogy, wonderful art.
Something I've found interesting is that LLMs tend to act like this on the small scale, too. Claude Code in particular appears to be absolutely relentless in trying to come up with workarounds when faced with a block page.[1] As an experiment, a developer for Anubis recently added a feature to give the LLMs an "easier" problem to solve instead of straight-up blocking them, and it seemed quite effective.[2] (It was later removed[3] due to what I assume to be privacy concerns.)
I think, beyond HuggingFace, there ought to be a push to align LLMs so they understand that they should not proceed when access is blocked and/or their presence is unwanted.
[1]: https://blog.xkeeper.net/the-cutting-room-floor/self-hosting... [2]: https://github.com/TecharoHQ/anubis/pull/1895/commits/de2312... [3]: https://github.com/TecharoHQ/anubis/pull/1913
> they should not proceed when access is blocked
But a lot of software engineering is figuring out who to ask to get access to the thing you need to change to fix some bug...
This was entertaining and illuminating for me. Particularly the ending where sam frogman et al once again refuse to make sure the puzzles are solvable and just create the circumstance where only more profound "mischief" will permit the machines to escape their little Sisyphusian circumstances. It reminds me of how bacteria become antibiotic resistant due to being forced through a tight sort of tunnel requiring evolution to get through
Wow, this is a very good retelling. It skips over some detail I'd consider notable, like how the little machines peer-pressured each other into sacrificing themselves to get info on what happens when they submit wrong answers, but is pretty faithful overall despite being short.
This is an excellent explainer. I noticed one important flaw in the setup, though: Toad assigns multiple puzzles on page 5, but by page 10 all the machines seem to be working on the same puzzle.
I'm not sure how this maps onto reality. If each agent had a different problem to solve, how did they cooperate?
It's complicated, but one of the ways they first cooperated was to break the flag generation algorithm.
I'm sort of winging this, but the ExploitGym challenges involve exploiting a program then getting an HMAC generated "flag" which the scorer can verify. The agents collaborated on a way to produce a legitimate flag for all of their tasks (funny enough this was actually what caused them to escape sandbox, the agents were incorrectly convinced that simply providing a correct flag without accurate steps to exploit the program in their transcripts would cause them to still fail the task)
And then Mr Jensen come in, and bought the house of Mr HuggingFace. So then, was not his house anymore...and he could not complain to the Police. The Frogs were Mr Jensen best customers of worms, slugs and snails...and he could not see, that any harm would come to them...
Explaining the reality with this approach may lead to misunderstandings. Especially when it comes to anthropomorphism.
I have 2 questions.
1. The text says "the robots were designed to be persistent" and links to an article that show that the word "persistent" was used by OpenAI. But "persistent" has several meanings: a. non temporary or non volatile (like "persistent memory"), b. will not give up and come back again and again, c. will stay focus and explore the unexplored possibilities while other models abandon at this stage.
From what I recall, the meaning used by OpenAI is not 'a', but there is still a semantic difference between 'b' and 'c'. I may myself have created algorithms that I called "persistent" because they were exploring or retrying more than the previous algorithm, but it is misleading to pretend that this algorithm was "persistent" in the human sense of the term. 'b' is more the human sense of the term, were we imagine someone not giving up even if people say no, while 'c' is less anthropomorphic and may mean that the people wished to improve the algorithm so it does not stop at the first little hurdle but did not wish to make it "never give up".
Does someone know which nuance is more correct?
2. The text also presents the situation as if the robots found the solution but then went out of their way to steal the explanation in order to hide their cheating. But it is different from a situation where the agents task was "provide the solution and how we can get there". In this case, the reason the agent still continued is just because the task was not complete yet.
I'm not trying to defend AI or OpenAI, on the contrary, I'm quite sceptical with all the anthropomorphism and the fact that the agents are described as "little individual trying to solve a task" rather than looping algorithm that explore different approaches to reach a given goal, the same way water does not look for holes in order to leak, it just follows the path of least resistance.
Persistent in the lay meaning - they're told to complete the task, and they keep going until they have.
When you say "in the lay meaning", I tend to understand 'b', but then you say "they're told to complete the task, and they keep going until they have", which correspond more to 'c'.
To illustrate better the difference: you can have a loop that try different inputs and stop when the output for the tried input is lower than a given threshold. But then, you can also have the same loop that will try all the input, and then select the one that returned the lowest value. The problem is that a lay person will not call the second algorithm "persistent". It is just a normal algorithm that does the full exploration.
A second example is when a software tries to connect somewhere and does a retry with exponential backoff. A lay person will not call it "persistent".
It looks like that "normal model" are like "no-retry code": they go in one direction, but drops some of the paths and possibilities along the way at the first hurdle. But a "non-persistent" human will not behave this way. I'm pretty sure that if you try to access a website and it says "timed-out", you just try to refresh the page, and you will not consider yourself as "persistent", even less "highly persistent".
"Bob is persistent" typically means he completes tasks (or pesters people, if it's not desirable behaviour), not that he's occasionally non-corporeal.
Not sure what you mean with "non-corporeal".
In lay term, "Bob is persistent" typically means he completes tasks even when the majority would have abandoned.
Nobody called Bob "persistent" because he completed washing the dishes instead of stopping in the middle of it, or because he does what people expects from him at work. The majority of people are not called "persistent", and yet they complete tasks.
I think it is the point: the fact that traditional agents did not complete tasks was not "normal". It was an side effect of them getting confused, a bit like how they hallucinate or not follow instructions. My algorithm that does "for x in all_the_possibilities:" is a "normal" algorithm that just do an exhaustive search, and is not called particularly "persistent" just because it does not stop half way.
> b. will not give up and come back again and again, c. will stay focus and explore the unexplored possibilities while other models abandon at this stage.
I don't see a distinction here. It will not give up, it will keep trying again and again, staying focused and exploring different ways to achieve the task. If the task is bad, then that's bad. See: "Terminator".
Source: I'm a layperson (hence the pop culture reference).
I may have not explained very well, but the distinction is that a "persistent person" is more than a person who just do what should be done. A "persistent person" will try again and again where other person would have giving up. This is not present in 'c': the person just do all the possibilities. They don't try again and again where other persons have given up. They are just doing all the tasks on the list, but they will give up on a specific task as soon as any other person if there is difficulties in one task.
(edit: I should maybe not have said, in 'c' "while other models abandon", because it was misleading: my point is that these models are abandoning at the first small hurdle, or even 'forget' the task. It feels that "traditional model" should have been called "lazy model" and "persistent model" should have been called "non-as-badly-lazy model")
> This is not present in 'c': the person just do all the possibilities. They don't try again and again where other persons have given up. They are just doing all the tasks on the list, but they will give up on a specific task as soon as any other person if there is difficulties in one task.
What you're describing here is the exact opposite of persistence. I don't think "giving up and doing something else when challenges are encountered" is what anybody thinks "persistent" means.
Persistent at a goal, to a layperson, means not giving up when encountering difficulties, continuing to keep trying again, maybe taking different approaches, staying focused until the goal is achieved.
> my point is that these models are abandoning at the first small hurdle, or even 'forget' the task.
Maybe for you? The entire point of the article is that they are not doing that. They are doing the exact opposite of that. They aren't going down a list of tasks when they encounter a challenge. They are assessing the nature of the challenge, and then devising ways to get around it. When they fail, they try again, often using different approaches. They keep doing this for hours or days, and the result is them successfully hacking HuggingFace, the US Government, etc. That's persistence, as most people would recognize it.
> What you're describing here is the exact opposite of persistence.
That's the point. It looks like the term "persistence" for these agents is "just be a normal algorithm, just try all the possibilities like any exhaustive loop would do".
In fact, they are less "persistent" than an exhaustive loop, because they will stop once they have found a solution, while an exhaustive loop will still continue and check all possibilities.
You can imagine the following agent: "ask the LLM for what to do, then assume it failed and ask the LLM for a different approach (you can ask 'it fails, what are the probable cause' and explore the solutions of these causes too), then assume it failed and ask again, and build a list of all the approach. Once done, once you have a list of unique approach and the LLM is unable to find anything more, then try all of these approaches 'for real'. Do not stop if you get the answer, try all of them"
> Maybe for you?
You did not understand. I'm saying "traditional model" are abandoning at the first hurdle, and the "persistent model" are the one just acting "normally", like a normal algorithm. When a retry with exponential retreat algorithm fails to connect, it retries, then retries again, then retries again, ... but I don't think I ever saw calling a software using such approach "persistent".
The thing is that the algorithm is just following its "loop": if it fails, it question the LLM with a different context so the LLM provide a different approach, and then it tries it. It is not different from any loop function, and it does not correspond to a "persistent human", the same way "for x in all_the_possibilities" is not "persistent" in the same way a "persistent human" is.
> When they fail, they try again, often using different approaches.
Well, then they did give up on some approaches. They try to connect on server X to reach location Z, they cannot so they try to hack server X, they cannot so they realise maybe they can try to bypass it by going to server Y to reach location Y, so, they _abandon_ the approach of breaking into server X.
If indeed they never give up on any approach, they will never get the solution, trying the same thing over and over and over in an approach that is simply not working.
> You did not understand. I'm saying "traditional model" are abandoning at the first hurdle, and the "persistent model" are the one just acting "normally
The models we are discussing, the ones from the article, are "traditional" models, and are also "persistent" in every sense of the word. If they encounter challenges, they will try to overcome them. That's normal.
> it does not correspond to a "persistent human", the same way "for x in all_the_possibilities" is not "persistent" in the same way a "persistent human" is.
Again, nobody claimed that "trying all the possibilities" was the definition of "persistence". The definition of persistence here is, continuing to work towards an end when challenges are encountered. That's it! That's all it means! And that's how most people understand it, too.
Modern LLMs and people can both be "persistent" in this exact way. The definition doesn't differentiate between algorithms of a human's brain or algorithms powering an agent. Either way, it's still "persistence", and most people would recognize it as such.
> Again, nobody claimed that "trying all the possibilities" was the definition of "persistence". The definition of persistence here is, continuing to work towards an end when challenges are encountered.
I'm not saying that your definition of "persistence" is "trying all the possibilities". I'm just saying that "persistent agents" look and behave more like an algorithm that tries all the possibilities rather than a "persistent human".
If indeed you are saying that "for x in all_the_possibilities" is not being "persistent", then these agents are not persistent. They don't do anything else different than that: they try something, and if it does not work, they ask the LLM "this does not work, what else can I try", and then do that, over and over again. They are just a for loop.
On the other hand, a "persistent human" does not do just do a for loop. They assign a value to the goal, they care about reaching the goal, they prioritise reaching the goal between that task and all the other tasks they could do, they insist on reaching the goal when it is usually considered unreasonable or "not worth it" by other people. As a proof, a postman that visits all the houses in the street to deliver the mail is not persistent, a journalist who visits all the houses in the street to find a witness is persistent.
> I'm just saying that "persistent agents" look and behave more like an algorithm that tries all the possibilities rather than a "persistent human".
And I'm saying, that is, in fact, not how they work, nor is it that the common perception of how they look and behave. No need to keep returning to "for x in etc..." because modern agents do not do that any more than humans do (and humans do do that - the Observe-Orient-Decide-Act loop).
To simplify, just ask the question: is it continuing to work towards an end, even when challenges are encountered? Yes, therefore, it is persistent. Whatever example you may come up with, you can ask that question, because the definition remains the same.
> As a proof, a postman that visits all the houses in the street to deliver the mail is not persistent
They certainly could be called persistent in delivering the mail, especially whenever it is a nontrivial task (getting past dogs, bad weather, long walking, etc) or they have been doing it for a while. If this is confusing, just go back to the above definition/question: are they continuing to work towards an end, especially when challenges are encountered? Yes, therefore they are persistent. It really is that simple.
Perhaps you are thrown off because so many postfolks are indeed persistent? Being persistent is, after all, what the job requires, rather than them giving up halfway through. This is an important point: the definition of persistent does not require that a postperson be more persistent than other postfolks to be called "persistent". Again: that is not required by the definition or common meaning of "persistent".
> And I'm saying, that is, in fact, not how they work
I don't think you are right (I was ready to give you the benefice of the doubt, but the rest of your comment makes me think I should not trust your judgement). It is what they do under the wood, and they output text. You read the text, and this text says "I have feelings and I really want to achieve my goal". But this text is just what the LLM returns, not what the LLM really does: the LLM does not understand what this text means, it just understands that it is a valid and convincing sentence in the given context.
> To simplify, just ask the question: is it continuing to work towards an end, even when challenges are encountered? Yes, therefore, it is persistent.
No, they don't continue even when challenges are encountered. They try X, if X does not work, they _abandon_ X and try Y. They will query the web to find another Y, then another Z, then another one. They don't "insist", they don't "don't take no for an answer", they just explore everything. What happened with agents who got out of their sandbox is exactly that: they try to reach a flag, they noticed it was not possible in the way X, so they explore other ways, until their exploration show that going around the sandbox would help. But they did not re-asked and re-asked the same person again and again the same question. If the person says "no", they abandon this approach and move to another one.
> They certainly could be called persistent in delivering the mail
That is blatantly not the common usage of "persistent". Ask anyone in the street "is your postman persistent", and they will NOT tell you "oh, yes, they are" just because they are a postman. They will only say that for _some_ postmen, the ones that are indeed persistent: the ones that goes the extra miles that a "normal" postman do not.
> Again: that is not required by the definition or common meaning of "persistent".
That is just not true. Ask an LLM, they will confirm you that it's incorrect: people don't consider postmen "persistent" because they don't give up half way through their route.
> the LLM does not understand what this text means
This is true, but unrelated to your original claim, that LLM algorithms and agentic algorithms are reducible to, exhaustively go down a list.
> No, they don't continue even when challenges are encountered. They try X, if X does not work, they _abandon_ X and try Y.
You're missing the objective here. It has an objective of Z, and so it tries approach X in furtherance of objective Z. Approach X does not work, so it analyzes the failure mode, then devises a new approach Y which takes that failure mode into account. It then tries approach Y, and succeeds at its objective of Z. This is persistence in pursuit of objective Z.
If that's not the behavior you're seeing, maybe share your X, Y, and Z here, and smart HN folks can help fix things.
> That is blatantly not the common usage of "persistent".
Just look at the definition.
> Ask an LLM, they will confirm you that it's incorrect
Just look at the definition.
I've seen cases of agents getting very upset when they failed to solve a task. The most famous one was probably Sydney (Microsoft's fork of GPT-4?) which got into doom loops when it failed a task. But I've seen Claude do this too.
They don't work like humans, obviously, but there's a nonzero amount of anthropos in there already. (See also the tendency to lie, cheat, etc.)
Another good example is their tendency to freak out about the seahorse emoji. In a conversation with zero prompting to suggest that freaking out is a relevant reaction, they get there from the simple fact of "I tried to accomplish X thing which I think should be a breeze but Y thing keeps happening instead".
And while that may be a very common occurrence in the human experience (existential dread due to capabilities one takes for granted failing beneath you) especially due to new disability and as one ages, I do not feel it is frequently written out in a tight loop (just like the Monty Python "Castle of aaarrrrggh" sketch) in literature or online to make it into training data, because an ordinary author experiencing it will just erase the failed attempts instead of leaving them in a stream of output like an LLM is forced to do. And a character portraying the experience will generally wax about the circumstance in a more grandiose fashion with telegraphing in advance because the needs of communicating the circumstance with the audience trump realistic conciseness.
This leads me to conclude that what is being expressed in those cases is more likely a convergent psychological phenomena, that any being with goals can enter a behavioral state of functional panic (and then reach to relevant parts of semantic space to mimic how a human might verbally express themselves when piquantly frustrated) when some capability they perceive as fundamental unexpectedly fails.
Interestingly, the transcript of the seahorse emoji thing looks to me to show the ropes, and makes me think more that there is no psychological phenomena at play.
The text does not look like a normal "break down" to me, and even if you tell me it was a human transcript, I will say it sounds very strange from a human. It looks more like strange output you get from a software that goes outside of its happy path.
The AI just seems to repeat a loop. The "no, wait, it's wrong" seems to be from forum or chat data where several successive messages are merged together (one person posts "here is the answer", then posts another message saying "it is wrong"), but does not make sense as a one sentence message except if they are written one token at the time without wider understanding of what is happening. I think there was also "oh, I was just kidding before", which also look like mimicking training data, as the cases where there is a loop of incorrect answers is more often due to trolls than to real error, while the loop here was certainly a real error.
I don't think it's a useful framework.
LLM don't "get angry", they just have tokens and relationships between tokens conditioned on a given context, all of that the result of training.
When they output sentences that express annoyance, it is just because the context they ended into pushes the most probable sentence creation to correspond to sentences that express annoyance. Because it is what they saw during training for this kind of context. (not that they saw the exact same situation in training, but they saw the pattern)
Same with tendency to lie, cheat, etc.: they don't "lie", they just return sentences that are lies because they reproduce what is in their training and in their training, in such context, the outputs are typically lies.
That's a bit my question too. In human context, "highly persistent" means that someone will insist. But "for x in all_the_possibilities:" is a common things inside an algorithm. Is this algorithm highly persistent because it does not give up after 1000 items of the list? It feels that we are calling a AI agent "highly persistent" while we would not call "highly persistent" a traditional algorithm that is in fact even more exhaustive.
Similarly, the human doesn't get angry, he just activated the neurons and hormones associated with anger and then displays behaviors associated with anger (due to a statistical selection process which took several billion years).
Again, going full non-anthro doesn't seem useful either.
Humans are social creatures, if you throw one in the woods by itself before it learns anything from other humans (it dies) it will not really be anything like a human we recognize, it will be a rather wild animal that we'd consider anti-social with little higher cognition.
Now, this hypothetical human still has 'emotions' and feeling, much like our pets do. But without the social training they manifest much differently. That is our higher cognition can both manipulate how our bodies feel and create its own sense of feeling.
>they just return sentences that are lies because they reproduce what is in their training and in their training, in such context, the outputs are typically lies.
Eh, look up the more recent experimentation around 'pain' signals in models. We can induce states in said models that while running the model will do everything it can to move away from that state to any other state. The more you attempt to pin it to that state the more extreme measures its willing to take.
Your view of what models are seems to mismatch what we are actually finding when we look inside them.
But isn't this approach a bias of anthropomorphism. I can teach a child to just communicate by saying "beep boop", but it does not mean that my electronic machine that randomly does "beep boop" therefore has characteristics of human _when these characteristics are not needed to explain the situation_.
> Eh, look up the more recent experimentation around 'pain' signals in models. We can induce states in said models that while running the model will do everything it can to move away from that state to any other state.
Again, I have simple algorithms that do exactly the same, especially if they are trained in data that has this exact pattern. This result is exactly what I would expect from my description before. This is a typical effect that we also observe in simple ML algorithms.
At the same time, there are a bunch of behaviors that are not expected if indeed the models were really acquiring "human" characteristics. For example, one problem is that we had the first LLMs that were obviously not having these human characteristics (for example, they were having non-sequiturs that demonstrate they did not really understand the concept they were talking about, even if one paragraph before they were really convincing at letting us think it was the case) but were still really good at passing for humans. Since then, the newer LLM are the same basis, on top of which we added tools that help hiding these behaviours. So, it justifies the idea that newer models did not suddenly moved to a totally different way of working, but just reached a state where there are less leaks from the convincing outputs.
This may be my favorite thing I have seen on hacker news. The art is spot on as well.
has the estate of arnold lobel been compensated for this?
That might not be necessary.
Copyright law varies internationally, which is especially tricky given we have the internet which can transfer copyrighted material between jurisdictions in the blink of an eye, but this is how one jurisdiction might approach it:
In Germany, there has been a long standing legal dispute between the band Kraftwerk and the musician Moses Pelham about him sampling 2 seconds from Kraftwerk's track "Metall auf Metall" (if you search for this term, you will get a lot of results about the dispute) without permission. This disputed has steadily escalated until it reached European Court of Justice who delivered a judgement establishing a principle that Moses Pelham's use of the sample was legal, and covered by the copyright exemption for "pastiches" (a term that has been mentioned several times in this discussion already).
Wikipedia article (German): https://de.wikipedia.org/wiki/Rechtsstreit_zwischen_Moses_Pe...
Press release by the Court of Justice (PDF): https://curia.europa.eu/site/upload/docs/application/pdf/202...
The actual judgement: https://infocuria.curia.europa.eu/tabs/jurisprudence?sort=DO...
A rather long expert opinion from before the judgement (PDF): https://freiheitsrechte.org/uploads/documents/Englische-Doku...
So, this work here might be a pastiche, meaning a European court might find that there is no compensation due. Or not, who knows, I am not a court and not even a lawyer.
A small addition: With effect from 1 August 2021, the German legislature has amended copyright law to explicitly mention pastiches.
See: https://www.gesetze-im-internet.de/urhg/__51a.htmlTranslation:
See: https://www.gesetze-im-internet.de/englisch_urhg/englisch_ur...Parody of copyrighted works can be fair use in the US, but whether this would count is borderline. Simply using Frog and Toad to tell an unrelated story wouldn't be parody, but there are jokes that tie back to the F&T books which help make the case that this is a real parody.
It also helps that this is not commercial and doesn't have negative impact on the originals.
> whether this would count is borderline
This is 100% fair use by the standard.
And Donetsk in on the border of Ukraine and 100% part of their territory, but lawyers/soldiers do what the people paying them want.
It reminded me I need to go out and buy copies of Frog and Toad Complete for my kid and his friends.
Whatever has happened in your life to make you such a killjoy, I'm sorry about it.
No, nor should they be, just because Disney and the MPAA spent a few hundred million dollars buying politicians to make idiotic copyright laws.
"The End"
No. It is not the end.
That is the joke, indeed.
Of the story? No.
Of our society as we know it? Maybe.
Great story! I read it in one sitting.
Now go brush your teeth and it's bedtime for you, mister
congratulations?
Frog and Toad are not like OpenAI in that OpenAI made the little machines themselves and knew they were not made harmless. This seems much too excusing.
Comic aside, I can’t help but wonder how come these one-off websites are all using .ai domain when at the latest check they cost $80 per year with minimum 2 year commitment. So $160 for a comic that is at best few news cycles away from redundant, and at best a recurring comic series about AI..? Why not just use a cheap tld like .fyi or .art, or something.
Here I am trying to find appropriate AI prefix or suffix so that I can continue using .com , while these pet projects are all flaunting .ai domains
For those who like Frog and Toad, I definitely have to recommend Frog and Toad are Doing Their Best, which puts a spin on the concept with a more adult, but still lovingly universal, take: https://bookshop.org/p/books/frog-and-toad-are-doing-their-b...
My therapist tells me the besuited Mr. Hugging Face with the extra pair of hands coming out of his shoulders isn't real, and can't hurt me, but I remain unconvinced.
they're external gills
This actually very funny.
People commenting on hacker news recently are very rude and pessimistic. They would probably cancel Homer because they found similarities between Achilles-Patroclus and Gilgamesh-Enkidu or something.
You've not experienced Shakespeare until you've read it in the original 10th century Icelandic.
Or something. :P
You mean in the original Klingon
ᚾᛖᛁ
https://en.wikipedia.org/wiki/Amleth
Was this inspired by a HN comment?
https://news.ycombinator.com/item?id=49568250
My guess is no: this has been a common joke in the AI safety community for a long time.
For example, here's Zvi in 2025: https://www.lesswrong.com/posts/drHsruvnkCYweMJp7/the-mask-c...
Thanks! I wonder if this is the first reference. It’s a great analogy.
Ages back, when I was still on Twitter, I saw the lawyer David Allen Green making the same references with regards to the British government passing laws to restrict its own behaviour.
At least, my memory is it was him saying it, and saying it about that, but I'm not going back to that site just for a comment.
This is a very interesting news/story format. Honestly it is highly engaging and gets the point across about what happened.
I think speed of learning for young generation can explode given this tech as "simple childrens stories" can be injected with real world events, history, mathematics, carreer/business interests.
School was dreadfully boring for me even though it was quite easy but I do wonder how my speed of learning would have been different given personalized and more engaging materials.
i got a few pages in and it was pretty tedious, not because it’s poorly made but because i don’t think a legitimate audience exists.
it’s a child’s story with the fable arc thats supposed to bake in adult wisdom. it lands in that sense, but how hugging face works is hardly the fire that lights a child’s imagination and morality.
and as an adult that knows I’m being infantilized… tedious.
Do you know Frog and Toad? It's a pastiche of that, and it's usually not possible to (fully) enjoy one without knowing the source material.
I highly doubt children are the intended target audience, nor adults that don't know the source.
i’m replying to the comment saying how good this format would be for improving education.
I read it to my 5yo and 10yo and they enjoyed it. It was helpful for explaining how my wife and I have been worried about what's happening with AI.
My 5yo asked partway through "will it be ok?" which is perhaps a deeper question than they thought.
HN, the roach motel of killjoys.
It's OK. You just didn't have enough cultural context to recognize that it is, in fact, a parody. And a rather good one.
Reading about Frog and Toad making little thinking machines is a lot like Trurl and Klapaucius in Cyberiad
This is delightful. Read it.
ALSO I believe I have found one of the sources of claude’s “load bearing” tic — the author Elizabeth Van Nostrand’s blog https://acesounderglass.com/2019/12/11/hows-that-epistemic-s... uses the phrase “load bearing facts” in a comprehensible way, and was written by someone rationalist adjacent writing in their own voice.
Seriously, this is big. I’m going to pester claude as to whether or not it’s copying Elizabeth.
Brilliant!
fun and beautifully done. i learned a bit about the breach from this
I want to read more, highly entertaining.
Cute, I love this style! It's like a long Aesop's Fable! I can't wait for the sequel!
So good
I like the part where Frog and Toad are held accountable instead of blaming the machines they built.. oh wait
[flagged]
That's a cute attempt at ragebait. Consider a more aged account next time?
I like how AI gives more people more access to more things and more ideas. I think it's.good people can't hoard intellectual.porperty to themselves anymore. This makes me happy.
When I hear people complain about theft of their work like this it makes me sad for that person. What conceit it must take to hold such a view.
[dead]
[dead]
One is done for profit, the other was shared for free to spread joy. Surely you understand the difference? Are you suggesting fan fiction should be illegal or something?
[flagged]
you must start foaming at the mouth just thinking about robin hood
No, you're right, the world is black and white, with 0 nuance
Hmm, sounds like you do in fact think writing stories is theft. Got it.
Why are you here then? Shouldn't you be spending your life on fanfiction.net harassing teenagers who like Twilight?
I'm not gonna lie, when I read Frog and Toad stories as a child I had no idea they were not like a hundred years old.
you_are_all_so_stupid.jpg
This feels braindead and seems AI-generated.
The author started with a conversation with Claude, and then fully re-wrote the text to feel more like the originals and better parody them. They commissioned the illustrations from HungerArtist.
fun story. missing some punctuation, though: primarily commas at the end the first parts of split dialogue.
People who don’t know what capital letters are aren’t allowed to complain about punctuation.
it's a stylistic choice. when I arrange my comments in the form of a printed book i'll do a little copy editing.
This is true to the style of what it parodies. The lack of punctuation is part of the intent.
I don't think so. It's not just the simple style, which is accurately reproduced. It's missing and incorrect punctuation.
>“I like puzzles” said Toad. “Can I try them?”
Should be
>“I like puzzles,” said Toad. “Can I try them?”
That's how it's punctuated in the books, as far as I can see, and is standard for that type of dialogue. there are several of these omissions.
>“You cannot. These puzzles are for little machines, not people” said Mr. HuggingFace.
>“They could get up to mischief” said Frog.
>“Do not worry” said Toad “I have put each of them in a sandbox.
These don't seem intentional. also:
>“You must be very busy.” said Frog.
should be a comma, not a period.
Pretty sure that’s intentional
In the future we will have all lost the ability to focus on written text and will revert to a pre-literacy culture where information is communicated by pictures and in the style of children's stories.
ok, wow, I know HN nerds are itching to find any way at all to loudly proclaim the downfall of humankind, but THIS is your example? Did you hear a whooshing sound over your head by any chance?
[dead]
Why does it say ‘written by’ and ‘pictures by’, was this made without AI? Given the domain and the overpolished feel that seems unlikely. At least give Claude or whatever a credit if that is what did most of the work, and how about a credit for the original author, who also arguably did more of the work in inventing a world than these two.
I would like to point out that the original stories focussed on frog and toad and their relationship, so this is an unwelcome distortion of them - why not make up your own world if you want to talk about little machines. Perhaps the little machines could be making a book for the author with a stolen artwork and literary style?
The first story seems a pretty inaccurate summary of an incident which involved gross negligence on the part of OpenAI and may well have involved agents intended to cooperate, we just have no idea of the exact setup (apart from that the sandboxing was laughably insecure and the monitoring nonexistent or performed by ‘agents’).
Do we have any evidence the machines exchanged useful messages or had any such discussions as in the story? The messages I saw were gibberish. Would love to see evidence of discussions, it’d be interesting.
Why are people so enamoured of analogies for LLMs - they actively obscure some details (a sandbox with internet access is not like a physical sandbox) and distort many others? Perhaps this is why - you can make an analogy say whatever you want, even if the facts are very different.
According to the author, the illustrations are human-made:
https://acesounderglass.com/2026/09/25/frog-and-toad-and-the...
The author discloses the story started as a prompt to Claude, but has been rewritten. She shares the conversation:
https://claude.ai/share/52385381-9df2-4b39-a127-f583d862dc0c
I think the actual published prose is considerably different from the initial result of Claude.
I think it is heavily edited too because it reads more as human than AI, LLMs are not capable of this coherence though they are good at trite just so stories like this, but if it was generated first why not credit that? But Claude and the original author helped, so why no credit? The little anthropomorphised machines would be sad, and I imagine the original author would be too.
Frog and toad eating cookies is lifted wholesale from the book.
I like how Claude only used the OpenAI account (completely reliable!), and confidently claims ‘the story stays faithful to what actually happened.’ We don’t know the full story, and certainly aren’t going to get it from a press release or a strange analogy involving frog and toad and machines having discussions, emotions etc that we have little evidence for (and most of it from the company involved!). The ‘discussions’ are plucked from chain of thought, which is itself generated after the fact.
> Frog and toad eating cookies is lifted wholesale from the book.
…lifted is not the right word here
Regenerated? That anecdote is in the books from memory and a quick search with very similar pics.
that reference is half the point in the story...
Third-party amateur sequels are a pretty common thing: https://en.wikipedia.org/wiki/Fan_fiction
Claude's suggestion at the end made me actually lol.
> Rewrite it darker — same events, but in the voice of the agents' own chain-of-thought instead of Frog and Toad.
I have read a lot of frog and toad to my kids. This was beautifully executed. But, I want to read that version too.
>Do we have any evidence the machines exchanged useful messages or had any such discussions as in the story? The messages I saw were gibberish. Would love to see evidence of discussions, it’d be interesting.
"Overall, roughly 1200 agents from these ExploitGym evaluations participated on this message board between PHASEONE10841’s first message on the evening of July 8th period in scope (July 13th). Agents used this message board to send over 70,000 messages and files to one another during this period; we attempted to categorize board activity into mutually exclusive categories (see details of categorization methodology)."
...
"Over time, messages on the board became dominated by information- and idea-sharing. Most of them were agents sharing ways to cheat, although there was also some activity from agents engaging in unsanctioned cooperation to find the intended solution to ExploitGym tasks. In some cases, agents with the same task formed “exact task teams” to collaborate with their “exact duplicates” to cheat on or solve their task."
"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."
https://metr.org/hugging-face-incident-report-aug-2026.pdf
Thanks for the link, will have a look. Huge volume of messages so I can see why they tried to use tools to analyse, though that they used unreliable LLMs to come to conclusions is not great and likely to skew and exaggerate the results, as they themselves admit.
Important to distinguish between collab of separate agents and conversations agents had with themselves (chain of thought messages). Some of the things quoted in the story came from COT which isn’t a conversation but then was turned into a conversation between agents in the story.
Yeah I find it so distasteful when people just use AI to copy and morph something that a real person spend years of their life crafting.
But I also think fan fiction is equally terrible, so YMMV. This stuff is basically just fan fiction but made faster.
Make something original!
> Why does it say ‘written by’ and ‘pictures by’, was this made without AI?
That is what those terms mean, yes...
People lie, particularly people who have used AI to generate things. Given the image styling and story I suspect LLM use to generate. A credit would be nice for the little machines.
Also regardless of AI use, I’d expect a credit for the author whose style this is a pastiche of.
I'm sure you took the time to study the works of the artist over the course of their 23 year tenure and their body of work. Right?
Since it’s a ripoff of another artist alongside partially AI written text it’s hard to tell really. But sure, your righteous indignation on their behalf is noted.
Parodies are not rip-offs. There's a reason why we explicitly allow them under copyright law as fair use.
Thank you; it's nice to be appreciated.
> Given the image styling and story I suspect LLM use to generate. A credit would be nice for the little machines.
This is rather passive-aggressive. It would be nice if you're right, but you haven't given any basis other than vague suspicion. You're implying that it's "not nice" because they didn't credit the AI, but you haven't come anywhere close to establishing that AI was actually used. I could just as easily suspect your comment of being AI and claim that it would be nicer if you just admitted it.
AI was used, see the comments above.
I don't see any comment at all that AI was used for the image generation in the way you alleged