Praxis-1 Predicts What a Robot Will See Next—and the Motion to Get There

A robot arm asked to place a mug upside down on a tray has to do more than map words to joint angles. It must infer where the mug is, anticipate how its movement changes the scene and recover when its camera sees something different from the prediction. Runway’s Praxis-1 makes that connection explicit: it predicts future video and actions together, in short chunks, and then observes again.

The November 11 release makes weights, inference and fine-tuning code available under Apache 2.0, including commercial use, through GitHub and Hugging Face. The system is a 27-billion-parameter mixture-of-experts diffusion transformer. Runway says the backbone has about 26.9B parameters, with roughly 2.8B active per token, alongside a separate action expert. The total model is about 54.4 GB in bfloat16 and the report says it runs on one 80 GB GPU. ^{[1] [2]}

Video prediction becomes part of the control loop

Many robot policies learn a direct mapping from camera frames and a task instruction to a sequence of actions. Praxis-1 also models what the scene should look like after those actions. A video backbone predicts future visual chunks; an action expert reads those features and predicts motion. Each robot has small input and output projections for its own pose and action representation, while the larger backbone and expert can be shared across embodiments. ^[2]

At inference, a call returns 24 action rows at 15 Hz—1.6 seconds of motion—alongside a predicted video chunk. The robot executes the chunk, sends fresh camera frames and calls the model again. This is a receding observation loop: the prediction is not a complete plan for a long task but a short commitment followed by another chance to correct. ^[2]

That design choice reflects a physical constraint. A video generator can use future frames while synthesizing a clip; a robot at time t cannot observe what happens at t+1. Praxis-1’s technical report describes changing attention to a chunk-causal setup, so a chunk can attend to earlier observations and itself, but not future chunks. Training and deployment must respect the same information boundary or the model would learn from clues unavailable to the real robot. ^[2]

General video may help, but embodiment still matters

Robot demonstrations are expensive to collect. Runway’s approach uses large-scale human and general video to learn how objects and tasks unfold, then adds embodied video and robot action data. The report describes staged training: video pretraining, embodied-video mid-training, joint video-and-action training, and fine-tuning for a target robot. Per-robot projections translate each machine’s pose and action dimensions into the shared model representation. ^[2]

That can lower the amount of robot-specific data required, but it does not erase the differences among robots. A gripper, camera placement, joint limits, payload, surface friction and coordinate frame all matter. The open weights are a starting point for adaptation; they are not a universal controller that can safely operate any arm from a text prompt.

Runway reports 555 successful episodes out of 1,200 on NVIDIA’s RoboLab-120 benchmark for a checkpoint fine-tuned on DROID. The company’s release compares that score with other models on the leaderboard. The number is useful as a reproducible benchmark claim when the task protocol and checkpoint match, but it should not be generalized to warehouse throughput, rare hazards or unscripted long-horizon work. The results are company-authored and require independent replication. ^{[1] [3]}

What an engineering evaluation should include

If you are considering Praxis-1, first pin the exact checkpoint and inspect its license, inference requirements and data lineage. Then test a narrow task in simulation with your camera views and action space. Measure completion, collisions, dropped objects, recovery after disturbances and latency—not only whether the model eventually reaches the goal.

Move to hardware only with an independent safety layer: speed and force limits, workspace boundaries, a physical emergency stop, guarded operation where needed and a deterministic controller able to halt on invalid actions. Test interruptions, stale observations and out-of-distribution objects. A model that predicts plausible video can still produce unsafe motion if its predicted actions are wrong.

The valuable question is not whether a world model can make robots general. It is how much robot-specific data and engineering can be saved without losing reliable control. Praxis-1 makes the research legible and modifiable for more teams. The practical answer will come from carefully instrumented tests on the robot and tasks a team actually intends to deploy. Compare its video-to-action approach with our coverage of retargeting human motion to robot trajectories; the latter concerns motion transfer, not a learned feedback policy.