Actions are both conditions for prediction and targets for learning.
Two experts. One causal conversation.
Know the beginning. Learn the rest.
A clean prefix conditions future-video prediction and denoising of the remaining action suffix.
A₁A₂A₃A₄
Clean conditionNoisy denoising target
The observed video stays clean. Both experts read the same instruction through their own text attention.
03 / IN THE REAL WORLD
From imagining to doing.
Precise placement. Careful alignment. Everyday manipulation, on a real robot.
01
Build, one block at a time.
02
Find the perfect fit.
03
A little breakfast ritual.
Full robot recordings · Original speed · Silent
04 / IMAGINED FUTURES
An action. And the world it changes.
Watch how the predicted future follows a sequence of actions.
Prediction examples
Loading recorded examples…
Same actions, two futures. Drag to rotate the recorded trajectory; scrub to follow the motion.
About the videos and actions +
Each clip starts with an observed frame, followed by eight predictions. Reference video and conditioning actions come from the same sample. The 3D path shows recorded end-effector positions in metres; it is not a predicted path. Action values are raw controller commands, not displacements in metres. Sample details include timing and model information.
✧✦✧
05 / A NEW FINDING
Dream, act, repeat.
Give OneWM an initial observation and a task. It predicts the next actions, imagines their consequences, then uses its own predicted view to decide what to do next.
An Endless Dream
After the first observation,OneWM sees only what it imagines.
OneWM · LIBERO
An imagined journey
Round 01 / 16
Scene viewWrist view
Initial observation · held
0.0 / 25.6 s
1
Predict the next actions
32 controls from the current view.
2
Imagine their effect
8 future frames, guided by those actions.
↺
Use the last imagined frame
Feed it back. The next round begins.
Unified causal training connects policy and world prediction: OneWM can carry a task forward inside a world it generates itself.
About this finding
Selected task:
16 rounds · 512 predicted actions · 25.6 s in imagination
Each LIBERO example starts from one dual-camera observation and a task instruction. The same OneWM checkpoint alternates action prediction and action-conditioned video prediction. Every later visual input is its own last generated frame; no later recorded observations or recorded actions enter the loop.
The animation reveals each saved 32-action prediction in temporal order, then plays its corresponding eight future frames. The 3D path accumulates the predicted translation commands within each round; it is a relative control path, not a measured end-effector trajectory. Values use the experiment’s denormalized actions. The pauses illustrate the loop, not inference speed.
All six examples retain the full 16 rounds. This experiment evaluates imagined task progression. Simulator execution and automatic completion detection are outside this experiment; long rollouts can show drift and generation artifacts.
06 / WHAT WE FIND
Better at both.
Matched comparisons support reciprocal benefits.
WORLD → ACTION
Policy success
vs. WAMmask
94.95%LIBERO +2.50 pp
59.27%LIBERO-Plus +5.90 pp
36.68%RoboCasa365 +8.36 pp
ACTION → WORLD
Better PSNR, SSIM & LPIPS
Across all three benchmarks, against the same-architecture world model.
A closer look at the numbers +
Real-world success: 48.9% → 66.7%, WAMmask → OneWM. Three tasks; 15 trials per task and model.
World prediction, PSNR ↑: LIBERO 22.33 → 22.95 · LIBERO-Plus 20.54 → 21.05 · RoboCasa365 24.72 → 25.21. Same MoT architecture, WM → OneWM.
Selected paper results. Distributional metrics and individual perturbation categories show a more nuanced picture.
✳
Understanding the world by learning to act in it.
Explore the architecture, experiments, and findings in the paper.
