This kind of work is important because it's an attempt to make computers understand the real world. LLM/GenAI do not understand the real world, and the more we run up against their limitations, the more the people and systems who depend on them will try to make the world more understandable to LLMs/GenAI. This means modifying your behavior so that you are more like a machine. LeCun is trying to make machines more human.
It's a bet beyond LLM and generative AI, to embrace other techniques and areas of research.
> The world is unpredictable. If you try to build a generative model that predicts every detail of the future, it will fail. JEPA is not generative AI. It is a system that learns to represent videos really well. The key is to learn an abstract representation of the world and make predictions in that abstract space, ignoring the details you can’t predict. That’s what JEPA does. It learns the underlying rules of the world from observation, like a baby learning about gravity. This is the foundation for common sense, and it’s the key to building truly intelligent systems that can reason and plan in the real world. The most exciting work so far on this is coming from academia, not the big industrial labs stuck in the LLM world.
Someone should just build an ANN as big as currently as possible with current hardware, while still having both inference and training to be as close to real-time as possible (micro-to milli-seconds), build the self-learning using some loose equivalents of pain/pleasure feedback in actual brains, plug sensors and actuators from some sort of robot, and just see what happens.
I think anything less than that is just a parlor trick.
That's still not enough. The biggest architectural hurdle to actualised artificial intelligence is the feed forward model. Unlike AI, all animals take in various inputs and produce various outputs asynchronously. That means the feed forward model of all NNs are fundamentally limited. Even recurrent NNs can't overcome this since they need a new input every iteration.
The counterpoint would be that when they started to build LLMs they must have clearly seen limitations of the approach and proceeded regardless, and achieved quite a bit. So the approach to introduce continuous (in-vivo if you will) self-guided training AND multiple sensors and actuators would still be limited but might yield some interesting results nevertheless.
ANN is an attempted model/ripoff (turned out to be extremely simplified but still) of a brain, why not go further? Continuous autonomous learning (which requires continuous feedback in a way of good/bad stimuli) is clearly what makes it work.
The current approach of guided pre-training and inference on essentially a "dead brain" clearly causes limitations.
For more information about his methods, broadly termed energy based models, check out the deep learning course he co-taught with Alfredo Canziani at NYU. https://atcold.github.io/NYU-DLSP20/
This kind of work is important because it's an attempt to make computers understand the real world. LLM/GenAI do not understand the real world, and the more we run up against their limitations, the more the people and systems who depend on them will try to make the world more understandable to LLMs/GenAI. This means modifying your behavior so that you are more like a machine. LeCun is trying to make machines more human.
It's a bet beyond LLM and generative AI, to embrace other techniques and areas of research.
> The world is unpredictable. If you try to build a generative model that predicts every detail of the future, it will fail. JEPA is not generative AI. It is a system that learns to represent videos really well. The key is to learn an abstract representation of the world and make predictions in that abstract space, ignoring the details you can’t predict. That’s what JEPA does. It learns the underlying rules of the world from observation, like a baby learning about gravity. This is the foundation for common sense, and it’s the key to building truly intelligent systems that can reason and plan in the real world. The most exciting work so far on this is coming from academia, not the big industrial labs stuck in the LLM world.
Someone should just build an ANN as big as currently as possible with current hardware, while still having both inference and training to be as close to real-time as possible (micro-to milli-seconds), build the self-learning using some loose equivalents of pain/pleasure feedback in actual brains, plug sensors and actuators from some sort of robot, and just see what happens.
I think anything less than that is just a parlor trick.
That's still not enough. The biggest architectural hurdle to actualised artificial intelligence is the feed forward model. Unlike AI, all animals take in various inputs and produce various outputs asynchronously. That means the feed forward model of all NNs are fundamentally limited. Even recurrent NNs can't overcome this since they need a new input every iteration.
Makes sense.
The counterpoint would be that when they started to build LLMs they must have clearly seen limitations of the approach and proceeded regardless, and achieved quite a bit. So the approach to introduce continuous (in-vivo if you will) self-guided training AND multiple sensors and actuators would still be limited but might yield some interesting results nevertheless.
Taking the biological approach to 11 there?
ANN is an attempted model/ripoff (turned out to be extremely simplified but still) of a brain, why not go further? Continuous autonomous learning (which requires continuous feedback in a way of good/bad stimuli) is clearly what makes it work.
The current approach of guided pre-training and inference on essentially a "dead brain" clearly causes limitations.
For more information about his methods, broadly termed energy based models, check out the deep learning course he co-taught with Alfredo Canziani at NYU. https://atcold.github.io/NYU-DLSP20/
Quantum inference. Mark my words and give it 20+ years.