In What I’m Working On, I used “world models” to describe our work at Parisi. Depending on who you ask, that phrase can mean several different things.
There’s overlap, but I think the most useful definition starts a layer below the demos.
A name with a few different promises
The most visible interpretation is some shape of generative AI that makes spatial worlds. World Labs generates, reconstructs, and simulates environments. Google’s Genie creates interactive worlds that evolve as you explore them.
In robotics, the emphasis often flips toward anticipating physical consequences. The model helps a robot predict what happens when it moves or interacts with an object.
In my mind, these are quite symbiotic. If you can make a world, you can generate training data within it. Assuming that data is useful and transfers to reality, it can feed the models operating in physical worlds.
I’m also seeing the term popping up more and more around process extraction and automation, often with graph-based representations of a system. A lot of ‘thing specific’ world models point in this direction.
Descriptions mix what a model produces, where it gets used, and what it has learned. That makes the category slippery.
The definition I find useful
To me, removing the applications and companies, a world model learns a representation of an environment and its dynamics. What exists, how it relates, and how it changes over time.
Representing the current state of some complex thing is the starting point. The bigger ambition is to use those learned relationships across different questions: what can we infer, what could happen next, and what changes if we take an action?
The concept isn’t new. We’ve modeled systems and simulated scenarios for a while now. Learned dynamics alos have a history; the 2018 World Models paper is one example.
The three ingredients I’m interested in now are generality, efficiency, and continual learning. I think most world modeling companies and approaches probably look at some collection of these as their primary goals – in both the research and commercialization of what they build. JEPA, Yann’s brainchild, illustrates one approach: predicting representations from context without reconstructing every detail of the observation.
We’re really digging into those ideas at Parisi, applied to a different kind of environment.
It’s a Small World
Put an actor (a robot) in a room with a table, a box, and a closed door.
For a task inside that room, that can be the entire world it needs to account for. The wallpaper probably contributes little. The friction between the box and the floor could contribute a lot. The representation needs enough of the system to anticipate what is possible within its constraints.
Prediction is useful on its own. Planning uses those predictions to compare possible actions. If the robot pushes the box, will it move? How far? Which routes become available afterward?
The environment determines the outcome of the action, and the action changes the environment. With incomplete information, we need to account for a range of possible outcomes. The next decision starts from whatever actually happened.
Now let’s open the door.
Another space comes into view. New objects and possible interactions enter the picture. The boundary follows what can influence the task, including things the robot can’t directly see.
“World” doesn’t have to mean the literal world. It could be a room, a logistics network, or a power grid.
As an aside: imagine a fish that has only ever known the river. One day, a fisherman pulls it out. For a moment, the world is much bigger than the water. I like that as a way of thinking about how environments shape the things within them.
The bigger world already exists
We don’t have to wait for a robot to graduate into an enormous open world before this becomes interesting.
The bigger world already exists. It has its own walls, boxes, chairs, and floors. They just take different forms.
Factories, networks, and grids already contain interacting objects, constraints, and people making decisions. Power systems are where some of my interest comes from, but the idea extends much further.
That’s why I prefer this definition. The interesting question is what the model has learned about a system … then, what that learning lets us do.