As I looked through the images I was unimpressed entirely, at first. But, then I started thinking, these look a little... "childish" to me.
Childish as in... A newish artist who is drawing a concept rather than light / forms (Which is something artists typically do as they understand drawing more and more).
The rose in the vase specifically - some models understood that there was supposed to be shading, reflections, the concept of refraction - others just drew "blue = glass" and "green = stem" and "red = rose".
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge. I'm expecting these to get better as models improve, and perhaps the artistic progression will be there along with it...
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge.
The Grok ones in particular gave me that thought. Most of them really look like what a kid would do when given the same tools, while the other models' output have distinctly more "AI-ness" to them (for lack of a better term.)
What's interesting is that the way in which they're childish is actually extremely human. In fact, one of the ways that you're often taught to draw more realistically is to stop thinking of the concepts as icons you're drawing the outlines of, and instead sort of blur your eyes and see things as they are: hues and values. In other words, become a camera or a printer that has no idea what it's capturing or printing other than a grid of values. That is how you achieve realism.
The fact that it has clearly iconified these concepts in its mind and is tracing the outlines of the things it thinks/expects to go where is very human.
Yes, but Parrots don't mimic the stages of learning speech development like children do. They just memorize a phrase.
These SOTA LLMs aren't trying to mimic existing children's drawings, but interestingly they're following somewhat similar progression that human children do as they develop.
LLMs come complete they don't train on the fly based on interaction with the world. LLMs are wikipedia if they stopped allowing edits and all of the previous edits are training.
The most interesting result for me is that they apparently prompted the models to optimize for SSIM, but many of the models trend worse over time. I suppose because viewing the canvas always comes after drawing, and they didn't give "revert to previous" capability as part of the toolkit.
Which in turn kinda jives with my experience of using these models for code: to some extent they only seem to have a concept of "forward", which invariably leads to "write more code to fix previous problems created", rather than taking a step back and removing broken things entirely.
GPT 5.6 Sol had the best two drawings (rose and starry nights) but even more impressive was how efficient it was RE cost/time/tokens vs Fable (3.4M vs 14.6M / $7.74 vs $161!). OpenAI has quietly innovated around inference - this is will be a growing differentiator even against open models.
I do think they are going to stretch their lead in value if Anthropic doesn't wake up and stop YOLOing tokens. Kimi is an amazing achievement, but it has the same (or worse) kitchen sink approach as Fable.
At work, even if Fable is technically better I much prefer Sol because it is so much faster and concise.
The Grok ones are amusing, almost comically bad. However, whenever I've tried to pass an image creation request to any of the Opus models, it's been far worse, like first week of using Microsoft Paint bad (while ChatGPT would create social media quality images using the same prompts)
Where do these models even have this data to learn from. There must be massive computer use datasets? I have a hard time believing it’s emergent if it’s able to do something this good.
Seriously, what's going on there ? Why is it so different from others? Is it just behind technologically/training wise or it's using something fundamentally different?
It performs much better than composer2.5 (while being as fast). It's not Opus, but I think it's not that far off. Definitely better than sonnet for what I've been doing.
On the other hand, I think they probably heavily adapted the training data so that it really is extremely focused on code. I just recently ran my personal "poetry benchmark" on it (where I give it ~850 poems I've written over my life and ask it to comment the corpus as a whole), and it's whack. It tries to write in portuguese (most of the poems are portuguese) and code-switches constantly and mixes up words to the point of making what it writes almost unreadable (e.g. it writes stuff like "You can't QoS that that look for beast poems", in portuguese, all messed up). The quality of the analysis is also quite bad (I'd say it's definitely behind Sonnet).
So I really think they either threw away data that wasn't tied to coding so that they could fine-tune it to that, or somehow they've got such an unbalanced dataset that coding ends up dominating either way. To me, its disastrous performance in this drawing "competition" fits this narrative.
Some of my poetry has clear IT jargon, but it's a very small portion of it (<1%). Some of my teenage poetry revolved a lot around the idea of wanting to become science, knowledge, and machine, and be rid of feeling altogether (to become an idea that has no body, or to be come the mathematical equations that define the world), with some very amateur odes written glorifying science and machine (a clear pastiche of Álvaro de Campos with a modern twist). But, again, this is not the majority of the work, far from it.
For some reason, OpenAI models, Gemini (and apparently Grok too), love to latch onto this and obsess over this idea that it's "programming poetry" or "poetry for the IT crowd". Often OpenAI and Gemini try to write the "equations of my poetry" (granted, I do write about a cyclical relationship between thinking, feeling and writing a lot, and I do have ONE poem which ends with a Q.E.D.).
I'm giving this context to say that it is very bizarre. It's as if they latch onto it and act as if it's a core or highly distinguished part of the poetry, when it really isn't. Anthropic models, on the other hand, absolutely do not do this, and have never done it.
I really don't understand why this happens. Maybe it's because it has a lot of portuguese, I don't know. And even though the "QoS" is clearly the wrong token being generated, I have had situations where gemini spoke of some phase of my poetry as the "Q&A part" (really, no joke...)
In any case, it's why it's my personal benchmark after all :D
I must admit, I have a natural tendency to overlook C&Bs, but solid catch -- it's there. Perhaps that explains the Mona Lisa redaction; Grok probably put more effort into that one.
Did Elon tell some poor engineer to give grok a prompt injection for drawing,"make it look like one of my childhood drawings!" Just like the Tesla truck?
- the best image related stuff I've seen is where the harness is constantly cropping and looking closer at things (likely helps a lot for computer vision in general)
- also it would be interesting because the harness could almost have its own "palette" as if it could play with blank squares and different strokes over lapping or blending before applying to the main canvas
If it is, it's a horrible ad. My main takeaway is that all of them were god awful at image generation and Fable was 20x more expensive and still awful.
Claude was clearly 'pushing back' on the coziness of the cabin. But I think it did best with the cat. Grok, I fear, is making a case for euthanasia. It's suffering and I think it would be cruel to let it continue. Someone pull the plug. .
Not useless. LLMs are the most general purpose computer algorithms ever created. They are getting smarter and cheaper at a geometric rate. What is a bad idea today could have useful applications tomorrow.
Citation needed - my company is paying more than ever for code generation. I have no reason to believe (given anecdotes) that anyone finds themselves in the opposite situation.
You’re paying more in total because the overall cost has gone down. This is normal economics when something useful gets cheaper. If you were to use original GPT-4 to do as much, you’d be paying an order of magnitude more.
I get a lot more use (read: cheaper per interaction) and much better quality results from the $20 that I spend on this stuff every month than I did several years ago.
(And several years before that, it was all essentially unobtanium.)
As I looked through the images I was unimpressed entirely, at first. But, then I started thinking, these look a little... "childish" to me.
Childish as in... A newish artist who is drawing a concept rather than light / forms (Which is something artists typically do as they understand drawing more and more).
The rose in the vase specifically - some models understood that there was supposed to be shading, reflections, the concept of refraction - others just drew "blue = glass" and "green = stem" and "red = rose".
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge. I'm expecting these to get better as models improve, and perhaps the artistic progression will be there along with it...
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge.
The Grok ones in particular gave me that thought. Most of them really look like what a kid would do when given the same tools, while the other models' output have distinctly more "AI-ness" to them (for lack of a better term.)
What's interesting is that the way in which they're childish is actually extremely human. In fact, one of the ways that you're often taught to draw more realistically is to stop thinking of the concepts as icons you're drawing the outlines of, and instead sort of blur your eyes and see things as they are: hues and values. In other words, become a camera or a printer that has no idea what it's capturing or printing other than a grid of values. That is how you achieve realism.
The fact that it has clearly iconified these concepts in its mind and is tracing the outlines of the things it thinks/expects to go where is very human.
Parrots can also sound extremely human, but it’s only mimicry.
How do we know the difference?
Yes, but Parrots don't mimic the stages of learning speech development like children do. They just memorize a phrase.
These SOTA LLMs aren't trying to mimic existing children's drawings, but interestingly they're following somewhat similar progression that human children do as they develop.
They don’t develop. We should stop anthropomorphizing LLMs.
AI labs are improving the ML techniques used to build better models; it’s a big difference.
Isn't LLM training, the development?
LLMs come complete they don't train on the fly based on interaction with the world. LLMs are wikipedia if they stopped allowing edits and all of the previous edits are training.
The LLMs themselves are not doing the developing or learning.
If you want to know if something is a human or a parrot, you could give it a paint brush and ask it to paint the Mona Lisa.
What's your parrot's take on the Jacobian Conjecture?
The most interesting result for me is that they apparently prompted the models to optimize for SSIM, but many of the models trend worse over time. I suppose because viewing the canvas always comes after drawing, and they didn't give "revert to previous" capability as part of the toolkit.
Which in turn kinda jives with my experience of using these models for code: to some extent they only seem to have a concept of "forward", which invariably leads to "write more code to fix previous problems created", rather than taking a step back and removing broken things entirely.
For capabilities reference:
I made a lower effort but similar scaffold for LLMs to do iterative drawing in Nov 2024, with Sonnet 3.5 as the artist: https://paritybits.me/llm-drawing-with-eyes-open/
Quite a difference.
GPT 5.6 Sol had the best two drawings (rose and starry nights) but even more impressive was how efficient it was RE cost/time/tokens vs Fable (3.4M vs 14.6M / $7.74 vs $161!). OpenAI has quietly innovated around inference - this is will be a growing differentiator even against open models.
I do think they are going to stretch their lead in value if Anthropic doesn't wake up and stop YOLOing tokens. Kimi is an amazing achievement, but it has the same (or worse) kitchen sink approach as Fable.
At work, even if Fable is technically better I much prefer Sol because it is so much faster and concise.
To me GPT is quite slow. I use it with max reasoning though but Fable at max as well
The Grok ones are amusing, almost comically bad. However, whenever I've tried to pass an image creation request to any of the Opus models, it's been far worse, like first week of using Microsoft Paint bad (while ChatGPT would create social media quality images using the same prompts)
Where do these models even have this data to learn from. There must be massive computer use datasets? I have a hard time believing it’s emergent if it’s able to do something this good.
Grok! LOL!
Seriously, what's going on there ? Why is it so different from others? Is it just behind technologically/training wise or it's using something fundamentally different?
Grok 4.5 is...something else.
It performs much better than composer2.5 (while being as fast). It's not Opus, but I think it's not that far off. Definitely better than sonnet for what I've been doing.
On the other hand, I think they probably heavily adapted the training data so that it really is extremely focused on code. I just recently ran my personal "poetry benchmark" on it (where I give it ~850 poems I've written over my life and ask it to comment the corpus as a whole), and it's whack. It tries to write in portuguese (most of the poems are portuguese) and code-switches constantly and mixes up words to the point of making what it writes almost unreadable (e.g. it writes stuff like "You can't QoS that that look for beast poems", in portuguese, all messed up). The quality of the analysis is also quite bad (I'd say it's definitely behind Sonnet).
So I really think they either threw away data that wasn't tied to coding so that they could fine-tune it to that, or somehow they've got such an unbalanced dataset that coding ends up dominating either way. To me, its disastrous performance in this drawing "competition" fits this narrative.
"You can't QoS that" sounds like the title of a nerdcore rap song.
Some of my poetry has clear IT jargon, but it's a very small portion of it (<1%). Some of my teenage poetry revolved a lot around the idea of wanting to become science, knowledge, and machine, and be rid of feeling altogether (to become an idea that has no body, or to be come the mathematical equations that define the world), with some very amateur odes written glorifying science and machine (a clear pastiche of Álvaro de Campos with a modern twist). But, again, this is not the majority of the work, far from it.
For some reason, OpenAI models, Gemini (and apparently Grok too), love to latch onto this and obsess over this idea that it's "programming poetry" or "poetry for the IT crowd". Often OpenAI and Gemini try to write the "equations of my poetry" (granted, I do write about a cyclical relationship between thinking, feeling and writing a lot, and I do have ONE poem which ends with a Q.E.D.).
I'm giving this context to say that it is very bizarre. It's as if they latch onto it and act as if it's a core or highly distinguished part of the poetry, when it really isn't. Anthropic models, on the other hand, absolutely do not do this, and have never done it.
I really don't understand why this happens. Maybe it's because it has a lot of portuguese, I don't know. And even though the "QoS" is clearly the wrong token being generated, I have had situations where gemini spoke of some phase of my poetry as the "Q&A part" (really, no joke...)
In any case, it's why it's my personal benchmark after all :D
They focused a little too much on Grok Imagine.
the starry night one is soo funny
The razor-wire at the bottom for Starry Night was clever, and very Grok. Really shows its military spirit.
Edit: I just don't see the point of redacting the Mona Lisa
I read the balls as “houses”, though the phallic spire emerging from them dead-center is also very on-brand
I must admit, I have a natural tendency to overlook C&Bs, but solid catch -- it's there. Perhaps that explains the Mona Lisa redaction; Grok probably put more effort into that one.
Did Elon tell some poor engineer to give grok a prompt injection for drawing,"make it look like one of my childhood drawings!" Just like the Tesla truck?
The Grok ones are so weird that they cross into uncanny and surreal. Truly bizarre and capable of eliciting feelings from me, if only bad feelings…
GPT-5.6 Sol is the knock out here. Some of those results are really human/charming.
The rectangular smudge tool is a weird tool in the first place, but it's cute to see the models try to use it.
I really like the cabin from Sol. It could be an art piece.
Fix this signup error please
https://www.tryai.dev/?error=server_error&error_code=unexpec...
Grok's look like a truly disturbed child. Like the drawings from that kid in "The Ring".
I wonder how much better a harness could get for drawing
I think a lot better
- the best image related stuff I've seen is where the harness is constantly cropping and looking closer at things (likely helps a lot for computer vision in general) - also it would be interesting because the harness could almost have its own "palette" as if it could play with blank squares and different strokes over lapping or blending before applying to the main canvas
That was a good testcase. Pretty cool.
Would love to see the results if they had used /goal or similar
Would be great to see Kimi here.
I think this is just an ad.
If it is, it's a horrible ad. My main takeaway is that all of them were god awful at image generation and Fable was 20x more expensive and still awful.
Well, your takeaway is bad. This isn't image generation, it's tool use with a digital paintbrush.
And image tokenisation.
Why include Grok when it is clearly not even in the same category as the other three?
The difference in cost is pretty incredible.
Rather the difference of cost of Claude is pretty incredible. Then again, I guess they gotta make it rain while they can.
Claude was clearly 'pushing back' on the coziness of the cabin. But I think it did best with the cat. Grok, I fear, is making a case for euthanasia. It's suffering and I think it would be cruel to let it continue. Someone pull the plug. .
Should have compared against FLUX Klein and Z-Image Turbo...
Now add Deepseek, GLM and Kimi :-)
I just tried with Deepseek and got this for the Mona Lisa
It might not have been the most scientific testNow this is a cool test. Sol's the best in each one, while Grok looks like the work of that person who "fixed" that Jesus painting..
I very much enjoyed Grok 4.5's rendition of the Mona Lisa as Elon Musk with tentacles.
Article is AI-written.
Useless They are not image generation models.
That's what makes it an interesting challenge.
Not useless. LLMs are the most general purpose computer algorithms ever created. They are getting smarter and cheaper at a geometric rate. What is a bad idea today could have useful applications tomorrow.
> cheaper at a geometric rate
Citation needed - my company is paying more than ever for code generation. I have no reason to believe (given anecdotes) that anyone finds themselves in the opposite situation.
You’re paying more in total because the overall cost has gone down. This is normal economics when something useful gets cheaper. If you were to use original GPT-4 to do as much, you’d be paying an order of magnitude more.
It keeps getting cheaper and better -- for me.
I get a lot more use (read: cheaper per interaction) and much better quality results from the $20 that I spend on this stuff every month than I did several years ago.
(And several years before that, it was all essentially unobtanium.)
Tell them to draw LeBron James or Heisenberg. Everyone will refuse.