This is incredible. One potential application that I'm thinking about already is the rapid iteration of video-game map blocking. Being able to drop in some 'initial state' configuration and then have it procedurally generate a handful of alternative configurations could make rapid prototyping a significantly quicker experience, especially if you wanted to see what a potential end result could look like.
Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.
I'm very excited about AI tooling moving forward if this is a glimpse into the future.
This seems like by far the best model yet for reconstructing 3D spaces from sparse images. It looks like you could reconstruct your whole house with pretty good fidelity from a dozen or so images taken on your phone.
They show it working with videos that have motion, but it seems like time is always frozen while the camera is moving, and they always return to a ground truth camera view before advancing time again. Maybe the temporal consistency isn't very good? This surprises me given how well it understands space. I guess modeling physics and time is the next step in the development of this kind of model.
In addition to frozen time, Atlas can handle some scene motion - see for example the candy New York City in the Camera-Controlled Generation section where cars are moving, or some subtle motion of the waves toward the end of the 1-minute video example.
But this is certainly one area we plan to improve going forward.
spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
Yes, as long as the input images are "poseable" -- if they were taken in the same space they need to have some overlap, where the same object or part of the scene is visible in multiple views so the pose can be predicted.
You can also manually position the input images in 3D space to create scenes generatively; we show examples of this in the "generating with spatial context" section
It's an overloaded term that different people use for different things. At World Labs we think about three major categories of world models:
- Renderers output pixels or views of a world, either the real world or a generated world
- Simulators predict how the world responds to actions
- Planners predict actions for an agent to make to affect some change in the world
Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.
It's an overloaded term for AI models that have spatial reasoning LLMs currently lack.
Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.
There's a bunch of different approaches people are trying:
- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)
- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)
- Google is betting on generative video
- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions
Small correction, World Labs isn't betting exclusively on explicit 3D -- many of the Atlas results in this post are generated frames without any explicit underlying 3D representation. But Atlas can (optionally) output explicit 3D for the cases where you know you want it.
However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.
The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.
There are other meanings. but essentially a world model is able to reason in 3d, rather than text.
It can operate in different modes for different applications.
If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.
But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.
> For robotics, reconstruction is only half the job: as a simulated robot moves through space, Atlas also generates the RGB and depth data its sensors would observe along the way. The world and the robot's view of it come from the same model.
Potentially very significant for accelerating the data flywheel challenge for robotics
Yes, one of the main goals of this model was to keep 3D consistency as the camera moves around; it can do this even without relying on an explicit 3D point cloud or Gaussian splat representation of the scene.
I believe so. This is not a model that generates pixels frame by frame from user input like genie 3. Instead, there’s an actual 3D scene / structure generated (point cloud, 3dgs) from the input images.
This is incredible. One potential application that I'm thinking about already is the rapid iteration of video-game map blocking. Being able to drop in some 'initial state' configuration and then have it procedurally generate a handful of alternative configurations could make rapid prototyping a significantly quicker experience, especially if you wanted to see what a potential end result could look like.
Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.
I'm very excited about AI tooling moving forward if this is a glimpse into the future.
This seems like by far the best model yet for reconstructing 3D spaces from sparse images. It looks like you could reconstruct your whole house with pretty good fidelity from a dozen or so images taken on your phone.
They show it working with videos that have motion, but it seems like time is always frozen while the camera is moving, and they always return to a ground truth camera view before advancing time again. Maybe the temporal consistency isn't very good? This surprises me given how well it understands space. I guess modeling physics and time is the next step in the development of this kind of model.
In addition to frozen time, Atlas can handle some scene motion - see for example the candy New York City in the Camera-Controlled Generation section where cars are moving, or some subtle motion of the waves toward the end of the 1-minute video example.
But this is certainly one area we plan to improve going forward.
I'm a cofounder at World Labs - happy to answer questions about Atlas!
spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?
Atlas project lead here.
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
can atlas also generate 3D without pose information attached to the input images?
Yes, as long as the input images are "poseable" -- if they were taken in the same space they need to have some overlap, where the same object or part of the scene is visible in multiple views so the pose can be predicted.
You can also manually position the input images in 3D space to create scenes generatively; we show examples of this in the "generating with spatial context" section
What exactly does world model mean? Ive seen it used so many times in so many ways to just describe SOTA anything its lost its meaning.
It's an overloaded term that different people use for different things. At World Labs we think about three major categories of world models:
- Renderers output pixels or views of a world, either the real world or a generated world
- Simulators predict how the world responds to actions
- Planners predict actions for an agent to make to affect some change in the world
Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.
Source: https://www.worldlabs.ai/blog/taxonomy-of-world-models
It's an overloaded term for AI models that have spatial reasoning LLMs currently lack.
Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.
There's a bunch of different approaches people are trying:
- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)
- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)
- Google is betting on generative video
- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions
Small correction, World Labs isn't betting exclusively on explicit 3D -- many of the Atlas results in this post are generated frames without any explicit underlying 3D representation. But Atlas can (optionally) output explicit 3D for the cases where you know you want it.
It means everything to everyone.
However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.
The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.
There are other meanings. but essentially a world model is able to reason in 3d, rather than text.
It means it builds internal representation of the world it understands (can do physics on/predict/modify) and then renders it.
Can a reconstruct distinct areas of a larger area with a "fog of war" in between, or will it hallucinate the unknown areas?
It can operate in different modes for different applications.
If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.
But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.
> For robotics, reconstruction is only half the job: as a simulated robot moves through space, Atlas also generates the RGB and depth data its sensors would observe along the way. The world and the robot's view of it come from the same model.
Potentially very significant for accelerating the data flywheel challenge for robotics
A camera moving through a 3D space the world model understands is getting much closer to real robotics applications
Any ideas if it keeps continuity - when you turn around, is the invented chair still there, or does everything shift like in a dream?
Yes, one of the main goals of this model was to keep 3D consistency as the camera moves around; it can do this even without relying on an explicit 3D point cloud or Gaussian splat representation of the scene.
Amazing! Thanks for explaining!
I believe so. This is not a model that generates pixels frame by frame from user input like genie 3. Instead, there’s an actual 3D scene / structure generated (point cloud, 3dgs) from the input images.
"Can reconstruct [scenes from Unreal Engine]"