LeCun's Vision for AI: World Models, JEPA, and Autonomous Intelligence

1. Motivation: Why LeCun Thinks Current AI Is Not Enough

The paper starts from a simple observation: humans and animals learn many things much more efficiently than current machine learning systems. A child does not need millions of driving trials, millions of falls, or millions of labeled examples to build useful expectations about the world.

LeCun argues that the missing ingredient is a learned world model: an internal model that allows an agent to predict what might happen, fill in missing information, imagine possible futures, and plan actions before trying them in the real world.

Simple definition: LeCun's vision is an AI system that learns a model of how the world works, uses that model to imagine future outcomes, evaluates those outcomes using costs, and chooses actions that lead to better futures.

The key problem

Most current AI systems are strong at recognizing patterns in data, but they often struggle with common sense, physical reasoning, long-term planning, and adapting to new situations with few examples. LeCun's paper proposes a path toward agents that learn more like humans and animals.

Student analogy: Imagine learning to ride a bicycle. You do not try every possible steering angle until you crash thousands of times. You use an internal sense of balance, gravity, speed, and danger. That internal sense is similar to a world model.

2. Paper at a Glance

A Path Towards Autonomous Machine Intelligence is a position paper. This means it is not mainly a benchmark paper or a report of one finished algorithm. Instead, it proposes a research direction and a possible architecture for future autonomous intelligent agents.

Title	A Path Towards Autonomous Machine Intelligence
Author	Yann LeCun
Version	Version 0.9.2, June 27, 2022
Type of work	Position paper / research vision
Main topic	Autonomous machine intelligence through world models, self-supervised learning, JEPA, and hierarchical planning.

The three big challenges

1. Learn by observation

Machines should learn a lot about the world by watching and predicting, not only through labels or rewards.

Self-supervised learning

2. Reason and plan

Machines should be able to search for good actions using differentiable models and cost functions.

Planning

3. Use hierarchies

Machines should represent the world at several levels, from detailed perception to abstract goals.

Abstraction

The paper's central message is not “one more neural network layer.” It is a proposal for a complete agent architecture: perception, memory, prediction, planning, action, costs, and executive control.

3. World Models: The Missing Component

A world model is an internal simulator. It helps an agent answer questions like: “What is happening now?”, “What is likely to happen next?”, “What would happen if I did this?”, and “Which action leads to the best outcome?”

Current situation + Possible action → Predicted future → Choose wisely

Why world models matter

Prediction: The agent can anticipate what may happen next.
Planning: The agent can compare different possible actions before acting.
Safety: The agent can avoid dangerous actions without trying them physically.
Common sense: The agent can reject impossible or unlikely interpretations.
Few-shot learning: The agent can learn new tasks faster because it already understands useful background structure.

World model versus ordinary classifier

System type	Typical question it answers	Limitation
Classifier	“What label belongs to this input?”	It may recognize an object but not understand how the situation can evolve.
Generative model	“What data could come next?”	It may try to predict too many irrelevant details, such as every pixel in a future video frame.
World model	“What future states are plausible, useful, or dangerous?”	It must handle uncertainty, abstraction, and task relevance.

Simple explanation: A world model is not just a memory of past examples. It is a system for imagining possible futures. That is why it is central to planning.

4. The Proposed Architecture for Autonomous Intelligence

LeCun proposes a modular architecture. Each module has a role, and the modules work together in a loop: perceive the world, predict possible futures, evaluate those futures, and choose an action.

Perception → World Model → Predicted Futures

Actor proposes actions → World Model → Cost / Energy

Short-term Memory ↔ Critic ↔ Intrinsic Cost

Configurator configures perception, world model, actor, and cost for the current task

4.1 Module-by-module explanation

Module	Simple meaning	Role in the architecture	Student analogy
Perception	Understands the current situation.	Transforms raw sensory input into useful state representations.	Looking at the road and recognizing cars, lanes, signs, and obstacles.
World model	Internal simulator.	Predicts possible future states from current states and imagined actions.	Imagining what happens if you brake, turn, or accelerate.
Actor	Action proposer.	Suggests possible action sequences and selects actions that reduce cost.	Choosing whether to slow down or change direction.
Cost module	Badness score.	Measures whether a state is desirable or undesirable using energy/cost.	Knowing that crashing, falling, or being hungry is bad.
Intrinsic cost	Built-in drives.	Represents basic non-learned objectives such as avoiding damage or discomfort.	Pain and fear of danger help guide behavior.
Critic	Future cost predictor.	Estimates long-term consequences without always running full planning.	Knowing from experience that a bad turn could lead to a crash later.
Short-term memory	Temporary workspace.	Keeps track of current and imagined states during reasoning.	Holding several possible routes in mind before choosing one.
Configurator	Executive controller.	Configures the other modules depending on the current task or subgoal.	Focusing attention differently when driving, reading, or searching for keys.

4.2 Why the configurator is important

The configurator is one of the most ambitious parts of the proposal. Instead of building a separate world model for every task, LeCun imagines one configurable world-model engine that can be adjusted to the current goal. This allows knowledge reuse across tasks.

Key intuition: The same person can use the same brain to drive a car, cook, read, and assemble furniture. The task changes, but much of the underlying world knowledge is shared. The configurator is the proposed mechanism for this kind of task-dependent reuse.

5. Mode-1 and Mode-2: Fast Reaction versus Deliberate Planning

The paper distinguishes between two broad modes of behavior. These are similar to the familiar distinction between fast intuitive action and slow deliberate reasoning.

Mode-1: Reactive behavior

The agent acts quickly, without deeply simulating many possible futures. This is useful when fast response is needed and the situation is familiar.

Fast
Automatic
Useful for practiced skills

Mode-2: Deliberate planning

The agent uses its world model to imagine action sequences, evaluate their outcomes, and choose a plan with lower expected cost.

Slower
More flexible
Useful for new or difficult situations

The planning loop

Current state + candidate actions → predicted future states → cost/energy evaluation → chosen action

A useful way to teach this is to say that Mode-2 can train Mode-1. Slow planning can generate good behavior, and then the system can learn a faster policy that imitates the planned behavior in familiar situations.

Student analogy: When you first learn mathematics, you solve slowly step by step. Later, familiar patterns become automatic. The same idea appears here: slow reasoning can eventually become fast behavior.

6. Self-Supervised Learning and Energy-Based Models

LeCun's proposal depends heavily on self-supervised learning. The agent should learn from the structure of the world itself, not only from labels or explicit rewards.

6.1 Self-supervised learning in simple terms

In self-supervised learning, part of the data is used to predict or evaluate another part. For example, a model may look at the beginning of a video and learn what kinds of future states are plausible. The key is that the data provides its own training signal.

Observed part x → Representation predicts / evaluates Unobserved part y

6.2 Energy-based models

An energy-based model does not have to produce one exact answer. Instead, it gives a low energy score to compatible pairs and a high energy score to incompatible pairs.

Energy value	Meaning	Example
Low energy	The pair is plausible or compatible.	A video of a ball rolling down a ramp after being released.
High energy	The pair is implausible or incompatible.	A ball disappearing and instantly reappearing somewhere else without cause.

Why this matters: The future is often not unique. Many futures can be valid. Energy-based learning allows the system to say “this future is plausible” rather than forcing it to predict every exact pixel.

7. JEPA: Joint Embedding Predictive Architecture

JEPA is the centerpiece of the paper. It is a way to learn predictive representations without reconstructing every raw detail of the future.

7.1 The problem with raw prediction

Predicting the future at the raw pixel level is extremely difficult. In a video, the exact movement of leaves, shadows, water, or clothing may be impossible to predict precisely. But those details may not matter for planning.

Simple explanation: If you are crossing the road, you do not need to predict every pixel of every car. You need to know whether a car is coming, how fast it is moving, and whether crossing is safe.

7.2 JEPA's main idea

JEPA does not predict the raw future observation. It predicts a representation of the future observation. This representation should keep useful, predictable information and discard irrelevant unpredictable details.

x → Encoder → s_x → Predictor → predicted s_y compared with actual s_y

Energy = distance( representation of y, predicted representation of y from x )

7.3 Generative prediction versus JEPA prediction

Approach	What it tries to predict	Main difficulty	Why JEPA is attractive
Generative model	Raw data, such as pixels, audio samples, or tokens.	Must model many details, including irrelevant unpredictable ones.	Useful for generation, but not always ideal for planning.
JEPA	Abstract representation of the target.	Must learn representations that preserve what matters and ignore what does not.	Better aligned with prediction, abstraction, and world modeling.

7.4 Why JEPA is non-generative

JEPA is called non-generative because it is not designed to reconstruct the exact missing data. Instead, it measures compatibility in representation space. This is why it can be useful for common sense and planning: the agent can focus on what matters for action.

8. Hierarchical JEPA: Abstraction Across Levels and Time

Hierarchical JEPA, or H-JEPA, extends JEPA to multiple abstraction levels. Lower levels handle short-term detailed information. Higher levels handle longer-term abstract information.

High level: abstract, long time scale

Goal → Subgoals → Long-term plan

Middle level: objects, relations, events

Objects → Trajectories → Interactions

Low level: detailed, short time scale

Sensory details → Short prediction → Motor control

Why hierarchy matters

Long-term planning cannot be done efficiently at the level of raw pixels or muscle movements. To plan a trip, you think in terms of destinations, routes, and constraints, not every step of every muscle movement. H-JEPA is designed to support this kind of abstraction.

Level	What it represents	Example in driving
Low level	Fine-grained sensory and motor details.	Steering angle, lane position, nearby obstacles.
Middle level	Objects and local events.	A pedestrian may cross; a car is slowing down.
High level	Abstract goals and long-term plans.	Reach the destination safely and efficiently.

Key intuition: Abstraction is not just compression. It is what makes long-term prediction and planning possible.

9. Planning Under Uncertainty

The real world is uncertain. A world model cannot always predict exactly what will happen. Other agents may act unpredictably, sensors may miss information, and physical systems may be noisy or chaotic.

9.1 How the agent plans

The actor proposes possible actions.
The world model imagines possible future states.
The cost module evaluates those futures.
The actor chooses an action sequence with low expected cost.
The agent executes the first action and repeats the process.

Action A → Future 1 Future 2 Future 3 → Compare costs

9.2 Latent variables

LeCun uses latent variables to represent uncertainty. A latent variable can be understood as a hidden choice that selects one plausible future among many. Different latent values correspond to different possible outcomes.

Student analogy: When you see a person near a crosswalk, several futures are possible: they may cross, wait, turn back, or look at their phone. A good world model should represent these alternatives instead of predicting only one future.

9.3 Cost and risk

In uncertain situations, the best action is not always the action with the best average outcome. Safety-critical systems may need to avoid actions with small probability but very high cost. This is why planning must consider both expected cost and risk.

10. LeCun's Position in Major AI Debates

The paper also positions this architecture against several popular claims in AI: that scaling is enough, that reward is enough, and that symbolic reasoning must be added as a separate hard-coded system.

“Scaling is enough”

LeCun argues that simply scaling generative transformer models is not sufficient, especially for continuous high-dimensional modalities such as video and for goal-directed planning.

LLMs Video

“Reward is enough”

He argues that scalar rewards provide too little information. Most learning should come from predicting the world, not from trial-and-error reward maximization alone.

RL Sample efficiency

“Symbols are required”

The paper suggests reasoning can be framed as energy minimization and constraint satisfaction, though discrete search may still be needed in some cases.

Reasoning Search

Balanced interpretation

LeCun is not saying that language models, reinforcement learning, or symbolic methods are useless. Rather, he argues that none of them alone provides a complete path to autonomous machine intelligence. The proposed alternative is a learned world-model architecture trained largely by self-supervised prediction.

11. Teaching Interpretation: How to Explain the Paper to Students

The easiest way to teach the paper is to present it as a move from “AI that recognizes patterns” toward “AI that understands enough about the world to predict, plan, and act.”

11.1 One-sentence explanation

LeCun's vision: Future autonomous AI should learn hierarchical world models from observation, use them to imagine possible futures, evaluate those futures with costs, and choose actions through planning.

11.2 A simple classroom story

Suppose a robot is asked to place a glass on a table. A purely reactive system may map camera input directly to motor commands. A world-model-based system would understand objects, surfaces, gravity, possible collisions, and the goal state. It could imagine what happens if it moves too quickly, if the glass tilts, or if the table is too far away.

See glass → Imagine actions → Avoid bad futures → Place safely

11.3 Common student misunderstandings

Misunderstanding	Correction
“JEPA is just another autoencoder.”	JEPA predicts representations, not necessarily raw input reconstruction. The point is to keep useful abstract information.
“A world model must generate perfect future video.”	The paper argues the opposite: useful world models should often avoid predicting irrelevant details.
“Intrinsic cost is the same as external reward.”	Intrinsic cost is built into the agent's architecture and can guide behavior without waiting for sparse rewards from the environment.
“H-JEPA is already a finished AGI system.”	The paper is a research agenda. Many parts remain unsolved, especially training H-JEPA from real video and designing the configurator.

12. Limitations and Open Research Problems

The paper is intentionally ambitious, but it leaves many difficult problems open. This is important for students: the proposal is not a completed recipe. It is a map of possible research directions.

Training H-JEPA from video

Can a hierarchical JEPA actually learn useful world models from real-world video at scale?

Representation quality

How can the system learn representations that keep useful information and discard irrelevant detail?

Latent-variable control

How can latent variables represent uncertainty without allowing the model to cheat or collapse?

Configurator design

How can a system learn to decompose complex tasks into subgoals and configure its own modules?

Planning efficiency

How can an agent search over future actions quickly enough for real-time tasks?

Safety and values

How should intrinsic costs be designed so that behavior remains safe, robust, and aligned with human goals?

Critical note: The most mysterious part of the proposal is the configurator. It is supposed to set subgoals, configure attention, and coordinate modules, but the paper does not provide a complete learning algorithm for it.

13. Conclusion

LeCun's vision can be understood as a proposal for moving beyond systems that only classify, generate, or react. The goal is to build autonomous agents that learn abstract world models, reason over possible futures, and act according to internally computed costs.

The central technical idea is prediction in representation space. JEPA predicts useful abstractions rather than reconstructing raw data. H-JEPA extends this idea across multiple levels of abstraction, enabling long-term planning through subgoals.

For lecture purposes, the strongest explanation sequence is: motivation → world models → architecture → Mode-1/Mode-2 → energy-based learning → JEPA → H-JEPA → planning → debates → limitations.

Final takeaway: The paper is best read as a research agenda for autonomous intelligence: learn the world, abstract what matters, imagine futures, minimize bad outcomes, and act intelligently.

References and Key Concepts

LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. Version 0.9.2, June 27, 2022. OpenReview.
Craik, K. (1943). The Nature of Explanation. Early psychological account related to internal models of the world.
Sutton, R. S. (1991). Dyna, an Integrated Architecture for Learning, Planning, and Reacting. Related to model-based reinforcement learning and planning.
Hinton, G. E. (2006). To Recognize Shapes, First Learn to Generate Images. Related to representation learning and prediction.
Bardes, A., Ponce, J., & LeCun, Y. (2021). VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. Relevant to non-contrastive self-supervised learning.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021). Barlow Twins: Self-Supervised Learning via Redundancy Reduction. Relevant to non-contrastive representation learning.
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021). Reward is Enough. A contrasting position discussed in LeCun's paper.