Regarding the LAION aesthetic predictor footnote, I don't see why a modern model and nonlinear classifier won't do a good a job. Is there a fundamental technical problem with the idea?
What would you have in mind for a modern model? Like Dino-V3 or something of that ilk?
For the LAION classifier specifically, it's trained on-top of CLIP. The bottleneck for accuracy isn't the linear/non-linear readout, it's CLIP's specific knowledge of the world.
CLIP was trained with a contrastive learning objective, where all it has to learn is that images of the same category are closer to each other than images of other categories (e.g. two items are both dogs so they should be closer semantically, while say a dog and a computer are different and should be farther semantically.
As a result CLIP is really good at object definitions, but struggles at pretty much everything else. Modern multimodal LLM's perceptual modules are much stronger than CLIP; that's why we chose fine-tuning a LLM rather then training a MLP on-top of CLIP (which would be a direct iteration on the LAION aesthetic classifier).
Aah, for this we're just trying to filter not generate. When it comes to conditioning, you'll still need a model that understands text since the primary control is text.
In the original Stable Diffusion, CLIP doubled as part of this LAION Aesthetic Filter as well as the text control for the generative model. But in the last couple of years, the field has drifted away from CLIP towards LLM hidden states (e.g. Qwen, Mistral, etc.). It turns out that text-to-image and text-to-video models do not need the visual prior from CLIP; instead the hidden states of multimodal LLMs act as better conditioning mechanisms.
But you are on-to something when it comes to leveraging pre-trained vision models as priors in generative image/video.
Nowadays, models like Dino-v3 (which have the strong spatial priors, you're talking about with LingBot) are being used in 3 distinct ways:
1) REPA: The DiT in the generative model minimizes cosine distance to another vision prior, like Dinov3. This paper from earlier this year, iREPA (https://arxiv.org/pdf/2512.10794) digs into WHAT specifically makes a good regression target for generative image.
2) VA-VAE: Most generative image/video models work in a latent space that's pre-trained separately. We wrote a lot about this in an earlier blog post (https://www.linum.ai/field-notes/vae-reconstruction-vs-gener...). But basically, VAEs that overfit to the reconstruction task create brittle latent spaces for generative models. One solution folks have figured out is to regress to a strong spatial prior like Dino, during VAE training. This is discussed at length in the VA-VAE paper(https://arxiv.org/abs/2501.01423). Recent image models like Flux 2, likely achieve their improvements through this type of VAE training.
3) RAE (Representation Autoencoder): This is the latest/greatest right now. Where generative image models are trained directly in Dino-v3 space; and then they just train a decoder to read back out into pixel space. (https://arxiv.org/pdf/2605.18324)
I strictly meant using the embeddings for training a reward model, not the generator. By good for generation I just meant the reward model might find more visual cues for aesthetic preference and avoid some of the spurious semantic correlation CLIP has
Hi HN, one of the authors here. Lmk if you have any questions, and I'll try my best to answer them!
Regarding the LAION aesthetic predictor footnote, I don't see why a modern model and nonlinear classifier won't do a good a job. Is there a fundamental technical problem with the idea?
What would you have in mind for a modern model? Like Dino-V3 or something of that ilk?
For the LAION classifier specifically, it's trained on-top of CLIP. The bottleneck for accuracy isn't the linear/non-linear readout, it's CLIP's specific knowledge of the world.
CLIP was trained with a contrastive learning objective, where all it has to learn is that images of the same category are closer to each other than images of other categories (e.g. two items are both dogs so they should be closer semantically, while say a dog and a computer are different and should be farther semantically.
As a result CLIP is really good at object definitions, but struggles at pretty much everything else. Modern multimodal LLM's perceptual modules are much stronger than CLIP; that's why we chose fine-tuning a LLM rather then training a MLP on-top of CLIP (which would be a direct iteration on the LAION aesthetic classifier).
> Like Dino-V3 or something of that ilk?
Yes but for generation LingBot seems uniquely compelling https://technology.robbyant.com/lingbot-vision because it has a very strong spatial prior
Aah, for this we're just trying to filter not generate. When it comes to conditioning, you'll still need a model that understands text since the primary control is text.
In the original Stable Diffusion, CLIP doubled as part of this LAION Aesthetic Filter as well as the text control for the generative model. But in the last couple of years, the field has drifted away from CLIP towards LLM hidden states (e.g. Qwen, Mistral, etc.). It turns out that text-to-image and text-to-video models do not need the visual prior from CLIP; instead the hidden states of multimodal LLMs act as better conditioning mechanisms.
But you are on-to something when it comes to leveraging pre-trained vision models as priors in generative image/video.
Nowadays, models like Dino-v3 (which have the strong spatial priors, you're talking about with LingBot) are being used in 3 distinct ways:
1) REPA: The DiT in the generative model minimizes cosine distance to another vision prior, like Dinov3. This paper from earlier this year, iREPA (https://arxiv.org/pdf/2512.10794) digs into WHAT specifically makes a good regression target for generative image.
2) VA-VAE: Most generative image/video models work in a latent space that's pre-trained separately. We wrote a lot about this in an earlier blog post (https://www.linum.ai/field-notes/vae-reconstruction-vs-gener...). But basically, VAEs that overfit to the reconstruction task create brittle latent spaces for generative models. One solution folks have figured out is to regress to a strong spatial prior like Dino, during VAE training. This is discussed at length in the VA-VAE paper(https://arxiv.org/abs/2501.01423). Recent image models like Flux 2, likely achieve their improvements through this type of VAE training.
3) RAE (Representation Autoencoder): This is the latest/greatest right now. Where generative image models are trained directly in Dino-v3 space; and then they just train a decoder to read back out into pixel space. (https://arxiv.org/pdf/2605.18324)
Hopefully this helps!
I strictly meant using the embeddings for training a reward model, not the generator. By good for generation I just meant the reward model might find more visual cues for aesthetic preference and avoid some of the spurious semantic correlation CLIP has