I tried running NVIDIA Cosmos 3 on DGX Spark
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
I've been strongly feeling lately that NVIDIA is at the forefront of Physical AI (embodied AI). As the foundation supporting robot control and factory simulation, the importance of World Foundation Models has risen significantly.
I had the opportunity to run the next-generation "Cosmos 3" from the NVIDIA Cosmos series on the DGX Spark™, so I'd like to share what kind of model it is. Cosmos 3 has a structure that handles everything in one model — from robot observation, predictive video generation, to control command generation — and the parts that previously required connecting 2 to 3 pipeline stages are increasingly being absorbed into the world foundation model side.
Cosmos 3 Architecture — MoT 2 Towers
In the previous Cosmos series, models were provided individually by use case, such as Cosmos Predict 2.5 for video generation and Cosmos Reason 2 as a VLM for video understanding. Cosmos 3 significantly changes this configuration. The core of Cosmos 3 is the Omni model, with a structure where two towers — the Reasoner Tower and the Generator Tower — run in parallel within the same MoT (Mixture-of-Transformers) architecture.
The Reasoner Tower is the understanding-focused VLM responsible for "reading and judging" video and text, while the Generator Tower is the generation-focused diffusion expert responsible for "creating and moving" images, videos, audio, and actions. The key point is that rather than placing these two as separate models side by side, they are connected via shared latent representations, allowing the generation tower to directly receive the intermediate representations derived by the understanding tower as conditions. Text is decoded autoregressively by predicting the next token in sequence, while images, videos, audio, and actions are generated through iterative denoising — a design that allows the most suitable generation method for each modality to be used within a single framework.
Cosmos 3 provides this MoT-configured omnimodel in two sizes: Nano (15.17B) and Super (63.99B). During inference, it is also possible to extract only the Reasoner Tower and run it as a VLM, so the same model can be used either as a Reasoner for understanding-focused applications, or as the full omnimodel when generation is also needed. In this article, we focus our validation on Nano.
Validation Environment on DGX Spark
Validation was performed on an NVIDIA DGX Spark (GB10 / ARM64 / 128 GB unified memory, CUDA 13.0, Ubuntu 24.04). The model used was Cosmos3-Nano (full Omni configuration, approximately 30 GB in BF16).
The official inference code is built so that the environment is ready with a single uv sync, installing torch 2.10.0+cu130, natten (Blackwell wheel), lerobot, and more. The visual tokenizer for Cosmos 3 uses Alibaba's Wan 2.2 VAE, which is automatically retrieved from Hugging Face on the first inference.
Running 4 Use Cases
Here is the main content. Using the Cosmos 3 Omni model, I ran four modes on DGX Spark: text-to-image, text-to-video, image-to-video, and Policy Model. Execution in all cases simply involves specifying an official sample JSON, making it straightforward.
Generating Commercial-Quality Robotics Scenes from Text
Text-to-image generates images of robotics scenes from long prompts. When given content such as "a modern laboratory with white walls and a gray floor, with a metal-finished robotic arm mounted on a white workbench," the actual measurements on DGX Spark showed 960×960 / 35 steps, 22 seconds after model residency, with approximately 30 GB GPU memory, producing an image that included most of the elements described in the prompt. At 22 seconds per image, it's quite moving to think that an open-source world foundation model can run on a single DGX Spark. Since it's built on training data from the Physical AI domain, it seems well-suited for VSS and manufacturing synthetic scene materials, as well as data augmentation for PPE training.
Generating Videos of Grasping Motions from Text
For text-to-video, I verified operation with the prompt "a gripper grasps a red cube and slowly lifts it." With a light setting of 256p / 24 frames / 12 fps, inference time was 22 seconds. In the generated video, the robot arm structure was temporally consistent, and the action sequence of "descent → contact → grasp → lift" was arranged in a physically plausible order. The fact that it doesn't break down structurally even at low resolution settings is characteristic behavior of a model trained on the Physical AI domain.
Generating Physically Conservative Videos from Existing Images
Image-to-video generates video starting from a conditioning image. When given the prompt "the right arm slowly reaches toward the board in the center and returns to its original position" with the official sample's conditioning image (two robot arms side by side with a wooden board), the result cleanly showed only the right arm moving as instructed while keeping both arms intact. Inference time was 17 seconds, shorter than text-to-video, suggesting that image conditioning stabilizes diffusion convergence.
What I personally found interesting is how strongly it respects the physical state of the conditioning image. Since it has a clear stance of "preserve what's in the image, don't introduce what isn't," for use cases that predict "what would happen if this state were left unattended" — like surveillance footage — its faithfulness in not spontaneously producing non-existent objects seems reliable.
Simultaneously Generating Video and Control Commands with the Policy Model
The central focus of this article is the Policy Model. This is the flagship mode of Cosmos 3, outputting a predicted video and a robot action sequence simultaneously from an observation video and a natural language task instruction. The parts that previously required connecting separate pipelines for "observation," "planning," "generation," and "control" are now completed in a single inference.
For validation, I used the official samples as-is. The observation video is from the Bridge dataset in LeRobot v3 format (WidowX kitchen robot), and the prompt in English is "Put the pot to the left of the purple item." Running on DGX Spark, it output a 640×480 × 17-frame predicted video and 16 steps × 10-dimensional actions in 21 seconds after model residency.
The prompt's instruction is properly reproduced in the video — "pot" is interpreted as the stainless bowl, which corresponds to a "portable container" in the scene, and the flow of grasping it and moving it to the left plays out correctly.
Here is the highlight. The Policy Model outputs numerical values for "how to move the robot arm" together with the video. A sequence of 16 steps of arm movements (a numerical sequence combining hand position, orientation, gripper open/close, etc.) is produced at a precision level that can be directly passed to the arm for execution. The samples include a "reference motion" for comparison with your own output. Since the official threshold for acceptance ("less than 0.05 error") is also defined, pass/fail is clear.
The overall error was 0.013194, falling to less than a quarter of the passing threshold of 0.05. Of the 16 steps, 14 steps matched the reference almost exactly, with slight deviation only at the moment the gripper opens and closes. The ability to simultaneously produce a predicted video and a passing-level motion in 21 seconds using only an observation video and natural language instructions gives a sense that this could practically serve as the foundation for "verbally instructing an arm" on small robots like the Reachy Mini or SO-ARM101.
Differences from Before and Summary
Finally, let me summarize what has changed compared to the previous Cosmos series.
Previously, when building Physical AI applications, it was necessary to build the observation part and the generation part using separate models. The form was to connect two pipelines — Cosmos Reason 2 as a VLM for observation, and Cosmos Predict 2.5 as a diffusion model for generation. Since each consumed approximately 17–40 GB in BF16, co-hosting two models sometimes required careful resource management. With Cosmos 3, this observation and generation are completed in a single inference. In this validation, the Policy Model output a predicted video and 16 steps × 10-dimensional action sequence together in 21 seconds.
The range of practical applications also appears to be expanding. For small robots like the Reachy Mini or SO-ARM101, since the Policy Model generates "language → video + action" end-to-end, there is a sense that a single model could handle parts that previously required separately training GR00T or ACT. For factory footage, use cases are also coming into view, such as starting from an anomalous event extracted by VSS and visualizing as a video "what would happen if this state were left unattended."
To summarize once more, the flow is as follows.
- Roles that were previously separate — such as video generation and video understanding — are consolidated in Cosmos 3 into a 2-tower MoT of Reasoner + Generator
- Text-to-image / text-to-video / image-to-video produced practical-quality output in approximately 35 steps and 22 seconds
- Image-to-video exhibited conservative behavior prioritizing the physical state of the conditioning image, making it seemingly reliable for safety-critical applications
- The Policy Model simultaneously generated a predicted video and action sequence from observation video and task instructions, clearing the official golden benchmark with MSE 0.013
The biggest change I felt this time was that the idea of completing "observation → planning → generation → control" within a single world foundation model can now run at a size that fits on one DGX Spark.
Cosmos 3 was officially released (GA) at Computex. I hope to continue exploring Cosmos 3 Reasoner's Physical AI inference, environment setup details, and further in-depth validation of the Policy Model in separate articles.
Reference Links
- NVIDIA Cosmos Platform Overview
- Wan 2.2 VAE (Wan-AI/Wan2.2-TI2V-5B)
- Qwen3-VL technical report (arXiv:2511.21631)
- LeRobot v3 dataset format
- Related Cosmos article series by the author
- Related VSS article series
