An adolescent learns to drive in about 20 hours of practice. A self-driving system, Yann LeCun writes, may need thousands of reinforcement-learning trials (training by reward and penalty) to learn that taking a curve too fast ends badly. The teenager never has to skid off the road to find out, because she already has a rough sense of what happens to a moving object that turns too sharply.

LeCun, then at Meta's Fundamental AI Research lab and New York University, built a research program on that gap. His June 2022 paper "A Path Towards Autonomous Machine Intelligence" argues that current AI systems lack an internal model of how the world works, and it sketches an architecture that would learn one. The paper matters here because it states in one place what its author believes language models cannot do and what he would build instead. It is a proposal, not a result: LeCun calls it a position paper rather than a scholarly one, and the text reports no experiments.

How information moves through the architecture

Six modules, one loop. The slow mode runs the loop; the fast mode skips it.

Information flow in LeCun's proposed architecture The world feeds observations to perception. Perception passes a state to the world model. The actor proposes actions to the world model, which predicts states and passes them to the cost module. The cost module returns energy and its gradient to the actor, which sends the first action to the world. A short-term memory exchanges states and costs with the cost module. A configurator sets the task for every module. Configurator Sets the task and the cost weights for every module Perception encodes x into state s[0] World model predicts future states built from stacked JEPAs Cost Intrinsic cost · fixed Critic · trained Actor Mode 1: policy Mode 2: optimizer Short-term memory read and written by the world model and the critic The world state s[0] predicted states s[1…T] proposed actions a[0] … a[T] energy + gradient states + costs Mode 1: state → action observation x first action a[0] 1 2 3 4
Information flow in LeCun's proposed architecture The world feeds observations to perception. Perception passes a state to the world model. The actor proposes actions to the world model, which predicts states and passes them to the cost module. The cost module returns energy and its gradient to the actor, which sends the first action to the world. A short-term memory exchanges states and costs with the cost module. A configurator sets the task for every module. Configurator Sets the task and the cost weights Perception encodes x into state s[0] World model predicts future states built from stacked JEPAs Cost Intrinsic cost · fixed Critic · trained Actor Mode 1: policy Mode 2: optimizer Short-term memory read and written by the world model and the critic The world state s[0] predicted states s[1…T] energy + gradient proposed actions a[0] … a[T] states + costs Mode 1: state → action observation x first action a[0] 1 2 3 4
  1. The actor proposes a sequence of actions.
  2. The world model predicts the states those actions would produce.
  3. The cost module scores the predicted states. The gradient of that score tells the actor how to change the actions. Steps 1 to 3 repeat until the score stops improving.
  4. The agent carries out only the first action, observes the result and starts over.

Mode 1 skips the loop. The policy maps the current state straight to an action, the dashed path in the diagram.

  • Information flow
  • Mode 1 shortcut
  • Storage and retrieval
  • Configured by the configurator

Based on section 3 of Yann LeCun, "A Path Towards Autonomous Machine Intelligence" (Version 0.9.2, 2022). Layout and labels are this article's own.


A world model lets an agent test an action before taking it

LeCun describes common sense as a collection of world models, internal models that tell an agent what is likely, plausible or impossible. An agent that has one can simulate the consequences of an action and discard the dangerous ones before the world punishes it.

His architecture has six modules: perception, a world model, a cost module, an actor, a short-term memory and a configurator that sets the system up for the task at hand. In the fast mode, the actor maps what it perceives straight to an action, which LeCun compares to Daniel Kahneman's System 1, the quick and intuitive kind of thinking. The slow mode is the more interesting one. The actor proposes a sequence of actions, the world model predicts the states they would produce, and the cost module scores them. The actor then adjusts the sequence to lower the score, carries out the first action and starts over. Control engineers have used this loop, called model-predictive control, since the 1950s.

The cost module is where goals live. One part is hard-wired and cannot be trained: it computes a single number for the agent's "discomfort," standing in for pain, hunger and similar drives. A second part, a trainable critic, learns to predict future discomfort. LeCun states that the design of the hard-wired part determines how the agent behaves.


JEPA predicts abstractions and leaves the details out

The world is only partly predictable, and LeCun treats that as the hardest problem in building a world model. Many futures follow from the same present. A system that generates a full prediction, down to every pixel of the next video frame, has to commit to details nobody could know.

His answer is the Joint Embedding Predictive Architecture, or JEPA. Two encoders turn the observed input x and the target y into representations, compact summaries in vector form. A predictor estimates the representation of y from the representation of x, and the prediction error, measured in representation space, serves as the energy: low when the two fit, high when they do not. Because the encoder of y may discard what cannot be predicted, the model never has to reproduce it. LeCun's example is a video of a car approaching a fork in the road. The representations keep position, orientation and speed, and they drop the trees and the texture of the sidewalk. A latent variable, a hidden switch that carries information x lacks, selects whether the car takes the left or the right branch.

The price is stated openly. A JEPA cannot easily generate y; it can only judge whether a y is compatible with a given x.

Technical aside

Why training can collapse (§4.5)

If both encoders output a constant, the prediction error drops to zero and the model has learned nothing. LeCun calls this collapse. Contrastive training, which uses mismatched pairs as negative examples, avoids it, but he argues it becomes inefficient in high dimensions.

In 2022 he proposed four criteria instead: each representation should carry as much information as possible about its input, the representation of y should be predictable from that of x, and the latent variable should carry as little information as possible. The method he described in detail, VICReg (Bardes et al., 2021), keeps the variance of each representation component above a threshold and pushes the covariance between components toward zero.

Later work replaced that recipe with something simpler. LeJEPA (Balestriero and LeCun, 2025) uses a single regularizer, SIGReg, which pulls the distribution of the representations toward an isotropic Gaussian. The authors prove that this rules out collapse, and the method needs only one trade-off setting instead of a stack of training tricks.


Stacked JEPAs plan at several time scales

Since each JEPA drops what it cannot predict, JEPAs can be stacked. The first level makes short-term predictions from low-level representations. A second level takes those representations as input and predicts further ahead at a coarser grain. LeCun's example is a commute. "Get to work" breaks into driving to the station and catching a train, driving breaks into walking out and starting the car, and the chain runs down to muscle control, which can only be filled in once the local conditions, such as obstacles and traffic lights, are perceived. Hierarchical planning usually needs a hand-defined vocabulary of actions. LeCun wants the intermediate levels learned.


Later work closed some of the open parts, not all of them

The architecture had gaps of its own, and the paper's section 8.1 lists them. Some have since been tested. In June 2025, Meta's research lab released V-JEPA 2, a 1.2-billion-parameter JEPA trained on internet video. A variant that also takes actions as input, V-JEPA 2-AC, was post-trained on less than 62 hours of unlabeled robot video and then picked up and placed objects with robot arms in two different labs, with no task-specific training and only an image of the goal. That is the slow loop from the diagram running on hardware, with two differences: it searches for actions by sampling candidates rather than following a gradient, and it needs about 16 seconds per action. In March 2026, LeWorldModel showed that a small JEPA can be trained stably end to end from raw pixels on a single GPU, and it planned up to 48 times faster than a comparable world model built on a pretrained encoder.
Other gaps remain open. V-JEPA 2 plans at a single level and over short horizons. Its authors report that errors accumulate as predictions are chained forward, and they name long-horizon planning as future work. The learned intermediate levels of the hierarchy LeCun describes have not been shown. In both systems a human supplies the goal as an image, and nothing yet plays the configurator's role of breaking a task into subgoals. LeCun left Meta in November 2025 and now pursues the program at AMI Labs in Paris, which raised $1.03 billion in March 2026 to build JEPA world models.


Sources and further reading

The link has been copied!