Having the ability to do real-time video generation on a single workstation GPU is mind blowing.
I'm currently hosting a video generation website, also on a single GPU (with a queue), which is also something I didn't even think possible a few years ago (my show HN from earlier today, coincidentally: https://news.ycombinator.com/item?id=46388819). Interesting times.
I think video-based world models like Genie 2 will happen and that they'll be shrunken down for consumer hardware (the only place they're practical).
They'll have player input controls, obviously, but they'll also be fed ControlNets for things like level layout, enemy placement, and game loop events. This will make them highly controllable and persistent.
When that happens, and when it gets good, it'll take over as the dominant type of game "engine".
I don't know how much they can be shrunk down for consumer hardware right now (though I'm hopeful), but in the near-term it'll probably all be done in the cloud and streamed as it is now. People are playing streamed video games and eating the lag, so they'll probably do it for this too, for now.
A very, very different mechanism that "just" displays the scene as the author explictly and manually drew it, and yet has to pull an ungodly amount of hacks to make that viable and fast enough, resulting in a far from realistic rendition...
This on the other hand happily pretends to match any kind of realism requested like a skilled painter would, with the tradeoff mainly being control and artistic errors.
> with the tradeoff mainly being control and artistic errors.
For now. We're not even a decade in with this tech, and look how far we've come in the last year alone with Veo 3, Sora 2, and Kling 4x, and Kling O1. Not to mention the editing models like Qwen Edit and Nano Banana!
This is going to be serious tech soon.
I think vision is easier than "intelligence". In essence, we solved it in closed form sixty years ago.
We have many formulations of algorithms and pipelines. Not just for the real physics, but also tons of different hacks to account for hardware limitations.
We understand optics in a way we don't understand intelligence.
Furthermore, evolution keeps evolving vision over and over. It's fast and highly detailed. It must be correspondingly simple.
We're going to optimize the shit out of this. In a decade we'll probably have perfectly consistent Holodecks.
I feel like this misses the point. Also, vision and image generation are entirely different things. Even for humans, with some people not being able to create images in their head despite having perfectly good vision.
Understanding optics instead of intelligence speaks to the traditional render workflow, a pure simulation of input data with no "creative processes". Either the massive hack that is traditional game render pipelines, or proper light simulation. We'll probably eventually get to the point where we can have full-scene, real-time ray-tracing.
The AI image generation approach is the "intelligence" approach where you throw all optics, physics and render knowledge up in the air and let the model "paint" according to how it imagines the scene, like handing a pencil to a cartoon/anime artist. Zero simulation, zero physics, zero roles - just the imagination of a black box.
No light, physics or existing render pipeline tricks are relevant. If that's what you want, you're looking for entirely new tricks: Tricks to ensure object permanence, attention to detail (no variable finger counts), and inference performance. Even if we have it running in real-time, giving up your control and definition of consistency is part of the deal when you hand off the role of artist to the box.
If you want AI in the simulation approach you'll be taking an entirely different path, skipping any involvement in rendering/image creation and instead just letting the model pupetteer the scene within some physics restraints. Makes for cool games, but completely unrelated to the technology being discussed.
Looks like there is some quality reduction, but nonetheless 2s to generate a 5s video on a 5090 for WAN 2.1 is absolutely crazy. Excited to see more optimizations like this moving into 2026.
those streams of text are often conditioned on the prompts - people are using it to learn about new concepts, and as a hyperpersonalised version of search. it can not only tell you of tools you didn't know existed, but it can show you how to use them.
I do like my buttons to stay where I left them - but that can be conditioned. instead of gnome "designers" telling me the button needs to be wide enough to hit with my left foot, I could tell the system I want this button to be small and in that corner - and add it to my prompt.
I feel like a lot of the above assumes the user knows what they want or what works best. I want an intelligent designer to figure out the best flow/story/narrative/game and create/present it, cause I'm a dumb user who doesn't know what is actually good.
that's called a default - I'm happy for a gnome designer to "design" the button to be large enough to hit with my foot with a blindfold on, but I'd like the option to change it to adjust to my workflow rather than adjust my workflow to the button.
Nevertheless it does seem that generating will fairly soon become fast enough to extend a video clip in realtime. Autoregressive by the second. Integrated with a multi modal input model you would be very close to an AI avatar that would be extremely compelling.
Video AI acceleration is tricky, where many of the currently in use acceleration loras and cache level accelerations have a subtle at first impact on the generated video, which renders these accelerations as poison for video work: the AI's become dumber to the degree they can't follow camera directions, and the character performances suffer, the lip sync becomes a lip flap, and the body motions are reduced in quality, and become repetitive.
Now, I've not tested TurboDiffusion yet, but I am very actively generating AI video, I probably did a half hour of finished video clips yesterday. There is no test for this issue yet, and for the majority it is yet to be realized as an issue.
Out of curiosity, what do you do with the footage? in a personal way I found it fun for the occasional funny situational video, or for some small background animations, but not so useful as a whole. I understand for things like making sketches from scripts and quick prototyping it's nice, but I am genuinely curious what's the use :)
I'm creating a college / corporate seminar level class with a 3D animated host and instructor. Each lesson begins with a brief animated intro, which the student then reads their lesson afterwards, and then there is a chatbot that understands that lesson that engages with the student afterwards. The course will have about an hour of 1-2 minute videos in the end, and because they are an animated "professor" it is then possible to create other ethic versions, and other language versions easier than otherwise. And for the curious, that hour of final in-use video will be sourced out of somewhere around 8 hours of final video; these video AIs are fine and dandy for short content, but when working on longer form media the consistency issues grow to Godzilla sized and really become the most consistent issue: trying to keep the likenesses of the characters, and the environment from drifting over time.
We are scarily close to realtime personalization of video which if you agree with this NeurIPS paper [1] may lead to someone inadvertently creating “digital heroin”
> We further urge the machine learning community to act proactively by establishing robust design guidelines, collaborating with public health experts, and supporting targeted policy measures to ensure responsible and ethical deployment
We’ve seen this play out before, when social media first came to prominence. I’m too old and cynical to believe anything will happen. But I really don’t know what to do about it at a person level. Even if I refuse to engage in this content, and am able to identify it, and keep my family away from it…it feels like a critical mass of people in my community/city/country are going to be engaging with it. It feels hopeless.
I tend to think that it leads to censorship, and then censorship at a broader level in the name of protecting our kids. See with social networks where you now have to give your ID card to protect kids.
The best way in that case is education of the kids / people and automatically flag potentially harmful / disgusting content and let the owner of the device set-up the level of filtering he wants.
Like with LLMs they should be somewhat neutral in default mode but they should never refuse a request if user asks.
Otherwise the line between technology provider and content moderator is too blurry, and tomorrow SV people are going to abuse of that power (or be coerced by money or politics).
At a person / parent level, time limits (like you can do with web filtering device for TikTok), content policy would solve and taking time to spend with the kids as much as possible and to talk to them so they don’t become dumber and dumber due to short videos.
But totally opposed that it should be done on public policy level: “now you have right to watch pornography but only after you give ID to prove you are adult” (this is already the case in France for example)
It can quickly become: “now to watch / generate controversial content, you have to ID”
That doesn't work when the Chinese produce uncensored open weight models, or ones that can easily be adapted to create uncensored content.
Censorship for generative AI simply doesn't work the way we are used to, unless we make it illegal to posess a model that might generate illegal content, or that might have been trained on illegal data.
> Censorship for generative AI simply doesn't work the way we are used to, unless we make it illegal to posess a model that might generate illegal content, or that might have been trained on illegal data.
Censorship doesn't work for stuff that is currently illegal. See pirated movies.
It saddens me to think that the efforts so far hasn't been it. Maybe I should try my hand at "closing the loop" for image generation models.
Could it destroy the society? The humanity had lived through bunch of such actual substances, and always got bored of it in matters of decades... those risk talks feel a bit overblown to me.
Fun fact: if you say the right prayers to the Myelin Gods it will fuse straight through sage3 at D/DQ like it's seen it before, which of course it has.
Now if someone could release an optimization like this for the M4 Max I would be so happy. Last time I tried generating a video it was something like an hour for a 480p 5-second clip.
I mean the baselines were deliberately worse and not how someone would be using these to begin with maybe noobs and the quoted number is only for DIT steps not for other encoding and decoding steps, which is actually quite high still. No actual use of FA4/Cutlass based kernels nor TRT at any point.
Having the ability to do real-time video generation on a single workstation GPU is mind blowing.
I'm currently hosting a video generation website, also on a single GPU (with a queue), which is also something I didn't even think possible a few years ago (my show HN from earlier today, coincidentally: https://news.ycombinator.com/item?id=46388819). Interesting times.
Computer games have been doing it for decades already.
I think video-based world models like Genie 2 will happen and that they'll be shrunken down for consumer hardware (the only place they're practical).
They'll have player input controls, obviously, but they'll also be fed ControlNets for things like level layout, enemy placement, and game loop events. This will make them highly controllable and persistent.
When that happens, and when it gets good, it'll take over as the dominant type of game "engine".
I don't know how much they can be shrunk down for consumer hardware right now (though I'm hopeful), but in the near-term it'll probably all be done in the cloud and streamed as it is now. People are playing streamed video games and eating the lag, so they'll probably do it for this too, for now.
This is also the VR killer app.
Are you sure it's not just polish on the porn that is already the "VR killer app"?
A very, very different mechanism that "just" displays the scene as the author explictly and manually drew it, and yet has to pull an ungodly amount of hacks to make that viable and fast enough, resulting in a far from realistic rendition...
This on the other hand happily pretends to match any kind of realism requested like a skilled painter would, with the tradeoff mainly being control and artistic errors.
> with the tradeoff mainly being control and artistic errors.
For now. We're not even a decade in with this tech, and look how far we've come in the last year alone with Veo 3, Sora 2, and Kling 4x, and Kling O1. Not to mention the editing models like Qwen Edit and Nano Banana!
This is going to be serious tech soon.
I think vision is easier than "intelligence". In essence, we solved it in closed form sixty years ago.
We have many formulations of algorithms and pipelines. Not just for the real physics, but also tons of different hacks to account for hardware limitations.
We understand optics in a way we don't understand intelligence.
Furthermore, evolution keeps evolving vision over and over. It's fast and highly detailed. It must be correspondingly simple.
We're going to optimize the shit out of this. In a decade we'll probably have perfectly consistent Holodecks.
Hmmm, future video's might just "compress" down to a common AI model and a bunch of prompts + metadata about scene order. ;)
I feel like this misses the point. Also, vision and image generation are entirely different things. Even for humans, with some people not being able to create images in their head despite having perfectly good vision.
Understanding optics instead of intelligence speaks to the traditional render workflow, a pure simulation of input data with no "creative processes". Either the massive hack that is traditional game render pipelines, or proper light simulation. We'll probably eventually get to the point where we can have full-scene, real-time ray-tracing.
The AI image generation approach is the "intelligence" approach where you throw all optics, physics and render knowledge up in the air and let the model "paint" according to how it imagines the scene, like handing a pencil to a cartoon/anime artist. Zero simulation, zero physics, zero roles - just the imagination of a black box.
No light, physics or existing render pipeline tricks are relevant. If that's what you want, you're looking for entirely new tricks: Tricks to ensure object permanence, attention to detail (no variable finger counts), and inference performance. Even if we have it running in real-time, giving up your control and definition of consistency is part of the deal when you hand off the role of artist to the box.
If you want AI in the simulation approach you'll be taking an entirely different path, skipping any involvement in rendering/image creation and instead just letting the model pupetteer the scene within some physics restraints. Makes for cool games, but completely unrelated to the technology being discussed.
Bob Ross did it, too.
1 frame of Bob Ross = 1,800s
So with 108,000 (60 X 1,800) Bob Ross PPUs (parallel painting units) we should be able to achieve a stable 60FPS!
Once you set up a pipeline, sure. They'd need a lot of bandwidth to ensure the combined output makes any kind of sense, not unlike the GPU I guess.
Otherwise it's similar to the way nine women can make a baby in a month. :)
The food/housing/etc bill for 108k Bob Ross er... PPU's seems like it would be fairly substantial too.
Looks like there is some quality reduction, but nonetheless 2s to generate a 5s video on a 5090 for WAN 2.1 is absolutely crazy. Excited to see more optimizations like this moving into 2026.
Efficient realtime video diffusion will revolutionize the way people use computers even more so than LLMs.
I actually think we are already there with quality, but nobody is going to wait 10 minutes to do a task with video that takes 2 seconds with text.
If Sora/Kling/whatever ran cool locally 24/7 at 60FPS, would anyone ever build a UI? Or a (traditional) OS?
I think it's worth watching the scaling graph.
> If Sora/Kling/whatever ran cool locally 24/7 at 60FPS, would anyone ever build a UI?
I like my buttons to stay where I left them.
Yeah, it’s like asking “why would anyone read a book today when LLMs can generate infinite streams of text”
those streams of text are often conditioned on the prompts - people are using it to learn about new concepts, and as a hyperpersonalised version of search. it can not only tell you of tools you didn't know existed, but it can show you how to use them.
I do like my buttons to stay where I left them - but that can be conditioned. instead of gnome "designers" telling me the button needs to be wide enough to hit with my left foot, I could tell the system I want this button to be small and in that corner - and add it to my prompt.
I feel like a lot of the above assumes the user knows what they want or what works best. I want an intelligent designer to figure out the best flow/story/narrative/game and create/present it, cause I'm a dumb user who doesn't know what is actually good.
that's called a default - I'm happy for a gnome designer to "design" the button to be large enough to hit with my foot with a blindfold on, but I'd like the option to change it to adjust to my workflow rather than adjust my workflow to the button.
I suppose if one only reads self-help books of the “You’re the best, trust your instincts!” kind, then LLMs are an appropriate replacement.
Or indeed, if one has a mind of their own and wants a tool to obey them, rather than submit to their "betters"'s opinions.
Please no, please no
That will be Windows 12 and perhaps 2 generations in of iOS :)
That’s not the actual time if you run it, encoding and decoding is extra
Nevertheless it does seem that generating will fairly soon become fast enough to extend a video clip in realtime. Autoregressive by the second. Integrated with a multi modal input model you would be very close to an AI avatar that would be extremely compelling.
this is probably the best tool for this stuff now: https://github.com/deepbeepmeep/Wan2GP
It has fastwan ... probably will have this soon. it's a request in multiple tickets: https://github.com/deepbeepmeep/Wan2GP/issues
Video AI acceleration is tricky, where many of the currently in use acceleration loras and cache level accelerations have a subtle at first impact on the generated video, which renders these accelerations as poison for video work: the AI's become dumber to the degree they can't follow camera directions, and the character performances suffer, the lip sync becomes a lip flap, and the body motions are reduced in quality, and become repetitive.
Now, I've not tested TurboDiffusion yet, but I am very actively generating AI video, I probably did a half hour of finished video clips yesterday. There is no test for this issue yet, and for the majority it is yet to be realized as an issue.
Out of curiosity, what do you do with the footage? in a personal way I found it fun for the occasional funny situational video, or for some small background animations, but not so useful as a whole. I understand for things like making sketches from scripts and quick prototyping it's nice, but I am genuinely curious what's the use :)
I'm creating a college / corporate seminar level class with a 3D animated host and instructor. Each lesson begins with a brief animated intro, which the student then reads their lesson afterwards, and then there is a chatbot that understands that lesson that engages with the student afterwards. The course will have about an hour of 1-2 minute videos in the end, and because they are an animated "professor" it is then possible to create other ethic versions, and other language versions easier than otherwise. And for the curious, that hour of final in-use video will be sourced out of somewhere around 8 hours of final video; these video AIs are fine and dandy for short content, but when working on longer form media the consistency issues grow to Godzilla sized and really become the most consistent issue: trying to keep the likenesses of the characters, and the environment from drifting over time.
Also interested, since that is my same impression.
We are scarily close to realtime personalization of video which if you agree with this NeurIPS paper [1] may lead to someone inadvertently creating “digital heroin”
[1] https://neurips.cc/virtual/2025/loc/san-diego/poster/121952
> We further urge the machine learning community to act proactively by establishing robust design guidelines, collaborating with public health experts, and supporting targeted policy measures to ensure responsible and ethical deployment
We’ve seen this play out before, when social media first came to prominence. I’m too old and cynical to believe anything will happen. But I really don’t know what to do about it at a person level. Even if I refuse to engage in this content, and am able to identify it, and keep my family away from it…it feels like a critical mass of people in my community/city/country are going to be engaging with it. It feels hopeless.
I tend to think that it leads to censorship, and then censorship at a broader level in the name of protecting our kids. See with social networks where you now have to give your ID card to protect kids.
The best way in that case is education of the kids / people and automatically flag potentially harmful / disgusting content and let the owner of the device set-up the level of filtering he wants.
Like with LLMs they should be somewhat neutral in default mode but they should never refuse a request if user asks.
Otherwise the line between technology provider and content moderator is too blurry, and tomorrow SV people are going to abuse of that power (or be coerced by money or politics).
At a person / parent level, time limits (like you can do with web filtering device for TikTok), content policy would solve and taking time to spend with the kids as much as possible and to talk to them so they don’t become dumber and dumber due to short videos.
But totally opposed that it should be done on public policy level: “now you have right to watch pornography but only after you give ID to prove you are adult” (this is already the case in France for example)
It can quickly become: “now to watch / generate controversial content, you have to ID”
That doesn't work when the Chinese produce uncensored open weight models, or ones that can easily be adapted to create uncensored content.
Censorship for generative AI simply doesn't work the way we are used to, unless we make it illegal to posess a model that might generate illegal content, or that might have been trained on illegal data.
> Censorship for generative AI simply doesn't work the way we are used to, unless we make it illegal to posess a model that might generate illegal content, or that might have been trained on illegal data.
Censorship doesn't work for stuff that is currently illegal. See pirated movies.
You just fell outrage bait so I doubt you will be able to identify AI.
Potentially interesting that the authors are primarily affiliated with NatWest - a British bank. I had to Google their names to find that out, though.
They highlight reduced workplace productivity as a risk, among other things.
It saddens me to think that the efforts so far hasn't been it. Maybe I should try my hand at "closing the loop" for image generation models.
Could it destroy the society? The humanity had lived through bunch of such actual substances, and always got bored of it in matters of decades... those risk talks feel a bit overblown to me.
Infinite Jest predicted this.
Fun fact: if you say the right prayers to the Myelin Gods it will fuse straight through sage3 at D/DQ like it's seen it before, which of course it has.
https://gist.github.com/b7r6/94f738f4e5d1a67d4632a8fbd18d347...
Faster than Turbo with no pre-distill.
Now if someone could release an optimization like this for the M4 Max I would be so happy. Last time I tried generating a video it was something like an hour for a 480p 5-second clip.
maybe wait for M5 Max and new MLX.
I mean the baselines were deliberately worse and not how someone would be using these to begin with maybe noobs and the quoted number is only for DIT steps not for other encoding and decoding steps, which is actually quite high still. No actual use of FA4/Cutlass based kernels nor TRT at any point.
I want to use this on a website!