OneWM

Can World and Action Modeling Benefit Each Other under Unified Causal Training?

One causal model learns to predict the world
and act in it.

An illustrated white robot arm arranging rose and blue blocks in a sunlit studio
imagine ↔ act

Yue Su1,2,6,*Bokai Lin2,4,*Xinkai Wang2,4,*Chubin Zhang3Xiangnan Wu5Kuan Zhang3Weiyang Jin1Jiwen Yu1Yiming Li3Cewu Lu2,4Lixin Yang2,4,†Xihui Liu1,†

1 MMLab@HKU2 MVIG@SJTU3 THU4 SII5 UCAS6 IAAI* Equal contribution   † Corresponding authors

THE FILM

Imagine. Act. Learn together.

OneWM in 2 minutes 52 seconds.
English narration · 1080p

From the idea to real robots and imagined futures.Watch on YouTube ↗
Read the narration

This is OneWM: one model that learns to imagine the world, and to act in it.

Robots need two kinds of understanding. A world model asks: if I take this action, what happens next? A policy asks the reverse: given what I see, what should I do? These are usually learned apart. So, can learning one make the other better, in both directions?

That's where OneWM comes in. It brings world prediction and action generation into one causal framework, where actions play two roles: a condition that says what will be done, and a target to learn.

Inside, a video expert and an action expert share joint causal attention, and both read the task instruction.

That attention follows time. Future frames read the past and their aligned actions, while actions read only what has been observed, never the future.

Training mixes three action regimes. With every action given, the model learns their visual consequences. With only the first few, it completes the rest. With none, it generates every action from what it sees.

One objective learns future video and unknown actions together.

So, does learning the world help the robot act? Under matched comparisons, yes. Success rises on all three benchmarks, with the largest gain, over eight points, on the most challenging one.

On a real robot, across three tasks, overall success climbs from 48.9 to 66.7 percent.

Now, the other direction. Given the same actions, does learning to act help the model predict the future?

It does. OneWM improves all three image-quality metrics on every benchmark, so predicted futures stay closer to reality.

Put the two together, and OneWM can dream. From a single observation, it predicts actions, imagines their effects, and feeds that imagined view back in to act again. Sixteen rounds, over five hundred actions, all inside its own imagination.

OneWM. Understanding the world by learning to act in it. Explore the paper and the project page.

For the exact baselines and scope of the video’s claims, see the matched comparisons below and the paper. The Dream–Act examples demonstrate imagined progression, not closed-loop simulator success.

Download English captions

Abstract

World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.

01 / THE IDEA

Prediction and control.
A two-way relationship.

A world model learns what happens when a robot acts.
A policy learns which action to take.
Can learning one help the other?

How much action is known?

World model

All-clean actions

Predict the effect of known actions.

WAMnoisy

All-noisy actions

Predict with noisy action conditions.

WAMmask

Actions masked

Predict without reading action tokens.

OneWM action conditioning
Outlined cells show the action-to-video connections. Every configuration preserves temporal causality.
WORLD → ACTION

Clean-action prediction strengthens control and generalization.

ACTION → WORLD

Learning actions improves future-video prediction.

02 / HOW IT WORKS

One architecture.
Three ways to condition.

Actions are both conditions for prediction
and targets for learning.

Two experts. One causal conversation.

Animated OneWM training pipelineA VAE encodes video frames, retaining a clean observed prefix and adding noise to future latents. Action chunks are embedded. Separate video and action experts exchange information through causal joint attention. Both experts receive the same instruction through their own text attention and feed-forward blocks. The action regime can be changed below. VIDEO EXPERTACTION EXPERT Framesclean prefix VAE+ε Q / K / V Joint attentionTEMPORALLY CAUSAL Text attention+ feed-forward Instruction ℓ Future latentsprefix retained Action chunkclean + noisy Embed Q / K / V Text attention+ feed-forward Action chunkcondition + target

Know the beginning. Learn the rest.

A clean prefix conditions future-video prediction and denoising of the remaining action suffix.

A₁A₂A₃A₄
Clean conditionNoisy denoising target
The observed video stays clean. Both experts read the same instruction through their own text attention.

03 / IN THE REAL WORLD

From imagining
to doing.

Precise placement. Careful alignment.
Everyday manipulation, on a real robot.

01

Build, one block at a time.

02

Find the perfect fit.

03

A little breakfast ritual.

Full robot recordings · Original speed · Silent

04 / IMAGINED FUTURES

An action.
And the world it changes.

Watch how the predicted future follows a sequence of actions.

Same actions, two futures. Drag to rotate the recorded trajectory; scrub to follow the motion.

About the videos and actions +

Each clip starts with an observed frame, followed by eight predictions. Reference video and conditioning actions come from the same sample. The 3D path shows recorded end-effector positions in metres; it is not a predicted path. Action values are raw controller commands, not displacements in metres. Sample details include timing and model information.

05 / A NEW FINDING

Dream, act, repeat.

Give OneWM an initial observation and a task. It predicts the next actions, imagines their consequences, then uses its own predicted view to decide what to do next.

An Endless Dream

After the first observation, OneWM sees only what it imagines.

OneWM · LIBERO

An imagined journey

Round 01 / 16
Initial observation · held
0.0 / 25.6 s
  1. Predict the next actions

    32 controls from the current view.

  2. Imagine their effect

    8 future frames, guided by those actions.

  3. Use the last imagined frame

    Feed it back. The next round begins.

Unified causal training connects policy and world prediction: OneWM can carry a task forward inside a world it generates itself.

About this finding

Selected task:

16 rounds · 512 predicted actions · 25.6 s in imagination

Each LIBERO example starts from one dual-camera observation and a task instruction. The same OneWM checkpoint alternates action prediction and action-conditioned video prediction. Every later visual input is its own last generated frame; no later recorded observations or recorded actions enter the loop.

The animation reveals each saved 32-action prediction in temporal order, then plays its corresponding eight future frames. The 3D path accumulates the predicted translation commands within each round; it is a relative control path, not a measured end-effector trajectory. Values use the experiment’s denormalized actions. The pauses illustrate the loop, not inference speed.

All six examples retain the full 16 rounds. This experiment evaluates imagined task progression. Simulator execution and automatic completion detection are outside this experiment; long rollouts can show drift and generation artifacts.

06 / WHAT WE FIND

Better at both.

Matched comparisons support reciprocal benefits.

WORLD → ACTION

Policy success

vs. WAMmask
94.95%LIBERO +2.50 pp
59.27%LIBERO-Plus +5.90 pp
36.68%RoboCasa365 +8.36 pp
ACTION → WORLD

Better PSNR, SSIM & LPIPS

Across all three benchmarks, against the same-architecture world model.
A closer look at the numbers +

Real-world success: 48.9% → 66.7%, WAMmask → OneWM. Three tasks; 15 trials per task and model.

World prediction, PSNR ↑: LIBERO 22.33 → 22.95 · LIBERO-Plus 20.54 → 21.05 · RoboCasa365 24.72 → 25.21. Same MoT architecture, WM → OneWM.

Selected paper results. Distributional metrics and individual perturbation categories show a more nuanced picture.

✳

Understanding the world
by learning to act in it.

Explore the architecture, experiments, and findings in the paper.

Read the paper

Citation

If you find this work useful, please cite: Download BibTeX

@article{OneWM,
  title={Can World and Action Modeling Benefit Each Other under Unified Causal Training?},
  author={Yue Su and Bokai Lin and Xinkai Wang and Chubin Zhang and Xiangnan Wu and Kuan Zhang and Weiyang Jin and Jiwen Yu and Yiming Li and Cewu Lu and Lixin Yang and Xihui Liu},
  journal={arXiv preprint arXiv:2610.21095},
  year={2026},
  eprint={2610.21095},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  doi={10.48550/arXiv.2610.21095},
  url={https://arxiv.org/abs/2610.21095}
}