Tried Running NVIDIA Cosmos World Foundation Model on DGX Spark
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
Many people might be curious when they hear "input text, and a video following the laws of physics is generated." Cosmos, published by NVIDIA, is exactly the platform designed to realize such "world foundation models." It consists of multiple models, including Predict 2.5, which generates videos from text and images, and Reason2, which understands video content and reasons about causal relationships and spatial structures.
However, world foundation models consume a large amount of memory, and running them has traditionally required data center GPUs such as A100 or H100. So this time, I decided to verify how far things could go on a DGX Spark with 128GB of unified memory. To cut to the chase, the 2B model ran comfortably, while the 14B model was abandoned after 3 system freezes...
This article covers both video generation with Cosmos Predict 2.5 and video understanding with Cosmos Reason2, sharing the benefits and limits of 128GB unified memory with concrete numbers.
Overview of the Cosmos Family
Cosmos is a platform designed to accelerate "Physical AI" development, with multiple models available for different purposes. Let me organize the whole family, focusing on the two I tested this time.
"Predicting the Future" with Predict 2.5
Predict 2.5 is a model that takes text or images as input and generates physically plausible videos. It was developed with autonomous driving simulation and robotics training data generation in mind, but it's also interesting as a pure text-to-video tool.
There are three modes depending on the type of input: Text2World, which creates videos from text alone; Image2World, which adds motion starting from an image; and Video2World, which generates a continuation of a video. Model sizes come in two variants—2B and 14B—with the 14B expected to produce higher quality output.
"Understanding Video" with Reason2
Reason2 is a VLM (Vision-Language Model) that analyzes video content and reasons about causal relationships, temporal transitions, and spatial structures. It is fine-tuned based on Qwen2.5-VL and answers in natural language questions such as "what will happen next in this video" or "what is the positional relationship between objects." Two sizes are available: 2B and 8B.
Transfer 2.5 Skipped This Time
I also attempted Transfer 2.5, which generates realistic footage from control signals (edge maps, depth maps, etc.), but even the 2B model requires 65.4GB of VRAM, and the process was killed by OOM on the DGX Spark. I'd like to try this again another time.
Test Environment
| Item | Value |
|---|---|
| Device | NVIDIA DGX Spark |
| SoC | Grace Blackwell GB10 |
| Memory | 128GB unified memory (CPU/GPU shared) |
| CUDA | 13.0.2 |
| Driver | 580.126.09 |
| OS | Ubuntu 24.04 (DGX OS) |
The defining feature of the DGX Spark is its "unified memory architecture," where the CPU and GPU share memory. In a typical GPU server, GPU memory (VRAM) and CPU memory (RAM) are physically separate, but the DGX Spark allows flexible allocation of 128GB. This means that models which would run into OOM on an RTX 4090 with 24GB VRAM might just work as-is.
Running Cosmos Predict 2.5
Setup
The official repository provides two setup methods: Docker and uv. I used uv this time.
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5.git
cd cosmos-predict2.5
git lfs pull
uv sync --extra=cu130
Because the DGX Spark uses ARM64 (aarch64) + CUDA 13.0, specifying --extra=cu130 is necessary. It won't work with CUDA 12.8 (for x86_64).
Downloading the model requires agreeing to NVIDIA's Open Model License on HuggingFace. You'll need to agree to the license on the Cosmos-Predict2.5-2B model page in advance and set HF_TOKEN as an environment variable.
The inference script used is examples/inference.py. The official README also mentions scripts/diffusers_inference.py, but at the time of writing, diffusers 0.35.2 did not have the pipeline for Cosmos 2.5 implemented, resulting in an ImportError.
Generating Video with Text2World
First, let's generate a video from text alone. Input is specified via a JSON file.
{
"prompt": "Water slowly pouring from a glass bottle into a clear glass on a wooden table. Close-up view, natural lighting from a window.",
"inference_type": "text2world",
"num_steps": 35,
"seed": 42
}
.venv/bin/python examples/inference.py \
--model 2B/post-trained \
--batch_input_path inputs/water_pour.json \
--batch_output_path outputs/ \
--disable-guardrails
--disable-guardrails is an option that disables input/output filtering by Cosmos-Guardrail1. When guardrails are enabled, input prompts are safety-checked using a Blocklist and Qwen3Guard (LLM-based), and generated videos are also inspected by a VideoContentSafetyFilter, with detected faces automatically blurred. The Guardrail model requires separate license agreement and download; since my purpose here was to verify video generation, I skipped it.
On first run, the model download will take a moment. The 2B Text2World checkpoint is approximately 5GB.
Generation proceeds through 36 denoising steps, taking approximately 53 seconds per step, for a total of about 32 minutes.
| Metric | Value |
|---|---|
| Generation time | Approx. 32 min (36 steps) |
| GPU memory | 27.7GB |
| GPU utilization | 96% |
| Power consumption | 88W |
| GPU temperature | 82°C |
| Output resolution | 1280x704 |
| Frame count | 93 frames (16fps) |
32 minutes for a single video is by no means fast, but being able to generate videos with a world foundation model on a desktop-sized device without an A100 or H100 is quite appreciated. Memory usage was 27.7GB, leaving plenty of headroom in the 128GB budget.
Animating Images with Image2World
Next, I tried Image2World, which generates a video from a still image as input. I tested with two images: an official sample bus terminal image and a robot welding image.
.venv/bin/python examples/inference.py \
--model 2B/post-trained \
--batch_input_path inputs/bus_terminal_i2w.json \
--batch_output_path outputs/ \
--disable-guardrails
In Image2World, motion is added according to the prompt while preserving the feel of the input image. From the bus terminal image, a video was generated of a bus slowly departing, and from the welding robot image, a video was generated of sparks continuing to scatter.
Generation time and file sizes for the two materials were as follows.
| Material | Generation time | File size |
|---|---|---|
| Bus terminal | Approx. 31 min | 823KB |
| Robot welding | Approx. 33 min | 1.7MB |
Generation time is about the same as Text2World. Having an input image gives the impression that the composition and color palette are more stable. For use cases like "I want to animate a photo I took myself," Image2World seems more suitable.
The Story of Trying the 14B Model and Freezing 3 Times
Since 2B was running stably, I moved on to challenging the 14B model. At the time of my research, the VRAM requirement for 14B Text2World was listed as approximately 49GB, so I thought it should run with 128GB of unified memory.
To cut to the chase, I tried three times and the system froze all three times, forcing me to unplug the power cable each time.
| Attempt | Conditions | Result |
|---|---|---|
| 1st | No options | OOM during model load → system freeze |
| 2nd | --offload-tokenizer, stopped unnecessary services |
Model load successful (51GB) → freeze immediately after inference started |
| 3rd | --offload-diffusion-model + --offload-tokenizer |
Model load successful (106GB/121GB) → freeze immediately after inference started |
On the 3rd attempt, I even used the option to offload the diffusion model to CPU, and the model load itself succeeded. However, the moment inference started, intermediate tensors from autoregressive generation caused memory to overflow and the system became unresponsive.
With DGX Spark's unified memory, when GPU memory is exhausted, the entire system goes down with it. On a typical GPU server, only the GPU process would be killed while the OS stays alive, but on the DGX Spark, since the GPU and CPU share memory, OOM on the GPU side triggers a kernel-level hang. Sometimes even a soft reboot fails to recover the GPU due to GSP firmware initialization failure, requiring a cold reboot by physically cutting the power...
The conclusion is that 128GB is not enough for the 14B model to run. It would likely require 256GB or more of unified memory, or a multi-node configuration.
Understanding Video with Cosmos Reason2
Setting Up the Environment on DGX Spark
If Predict 2.5 is the model that "creates videos," Reason2 is the model that "reads videos." Reason2-8B is a VLM based on Qwen2.5-VL that reasons about causal relationships and temporal transitions in video events.
Running Reason2 on DGX Spark required a bit of ingenuity. Reason2 is a multimodal model based on Qwen2.5-VL, and since the FlashAttention sm_121a kernel is not provided, it cannot be started in vLLM server mode. I adopted a configuration that performs inference directly using transformers.
from transformers import Qwen2_5_VLForConditionalGeneration
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"nvidia/Cosmos-Reason2-8B",
torch_dtype=torch.bfloat16,
attn_implementation="eager", # FlashAttention not supported for sm_121a
).to("cuda")
The key is attn_implementation="eager". The default SDPA and FlashAttention don't have kernels provided for the DGX Spark's GPU (Blackwell, sm_121a) and crash with a cuDNN error. Switching to eager attention works around this. The model size is approximately 17.5GB in BF16, with a load time of about 89 seconds.
Running Inference on 3 Types of Material
I tested the capabilities of Reason2 with three types of videos: causal reasoning, temporal transition, and spatial reasoning.
Causal Reasoning with a Water Overflowing Video
I input a video of continuously pouring water into a glass until it overflowed and asked "what is happening in this video."
Reason2 described the process of the glass filling with water in chronological order, returning a response that included detailed observations such as the water surface rising while forming bubbles until it reached the rim, and light refraction.
| Language | Token count | Inference time | TPS |
|---|---|---|---|
| English | 73 | 21.4 sec | 3.4 |
| Japanese | 197 | 33.9 sec | 5.8 |
Reason2 Japanese response (water pouring video)
A transparent glass placed in a silver sink is being filled with water flowing down from above. Some water is already in the glass, and the new flow pushes it up while adding more. The water surface slowly rises while forming bubbles, eventually reaching the rim of the glass. In this process, water fills the interior space of the glass as it enters, and as a result, the water level can be observed rising. Additionally, a few water droplets remain on the outside of the glass, and the effects of light through reflection and refraction are visible on its surface.
Temporal Transition with an Egg Cracking Video
In a video of cracking an egg into a frying pan, it recognized the egg yolk and white already in the pan, and read the action of another hand cracking the next egg in. It captured the situation of someone trying to cook multiple eggs simultaneously, suggesting an understanding of temporal progression.
| Language | Token count | Inference time | TPS |
|---|---|---|---|
| English | 142 | 28.7 sec | 5.0 |
| Japanese | 80 | 21.0 sec | 3.8 |
Reason2 Japanese response (egg cracking video)
Another egg yolk and white have accumulated in the frying pan. Meanwhile, another hand is holding an already cracked shell and putting in the next first egg. The remaining half is still being added. At this point, they are trying to make two eggs at once, so both will solidify shortly after.
Spatial Reasoning with an Aerial Parking Lot Video
In a video shot from above a parking lot, the overall layout (arrangement of car rows, green strips between rows, sidewalks, surrounding plants, etc.) was read fairly accurately. However, the specific count (76 cars) may not be accurate, and to be honest the material itself may have been a bit unsuitable...
| Language | Token count | Inference time | TPS |
|---|---|---|---|
| English | 512 | 67.9 sec | 7.5 |
| Japanese | 228 | 37.1 sec | 6.1 |
Reason2 Japanese response (aerial parking lot video)
76 cars are visible in the image. These cars are parked in an orderly arrangement. From left to right, they are lined up vertically. There are green spaces between each row, with trees planted in them. There is also a sidewalk around the parking lot, and people walking along it are visible. On the perimeter of the parking lot, grass and other plants can be seen.
The 8B model's GPU memory usage was a stable 18.1GB, lighter than Predict 2.5's 2B (27.7GB). There is ample headroom in 128GB to use both models alternately.
Summary
Let me look back at the results of running the Cosmos family on the DGX Spark's 128GB unified memory.
The 2B model of Predict 2.5 ran stably for both Text2World and Image2World. Although it takes about 30 minutes to generate a single video, being able to try world foundation model video generation on a desk-side device without using an A100 or H100 was, personally, quite an interesting experience.
On the other hand, the 14B model was abandoned after 3 system freezes. Even offloading to CPU with --offload-diffusion-model, the intermediate tensors during inference exceed 128GB, making it impossible to run on a single DGX Spark. Running the 14B would likely require an environment with 256GB or more, or a multi-node configuration.
Reason2's 8B ran smoothly at 18.1GB, showing practical quality in causal reasoning and temporal transition analysis. The high quality of Japanese output was also a pleasant discovery.
The 128GB of unified memory provides an environment where you can try models at your desk that "can't run both at once on even an RTX 5090, but don't quite need an H100." With Predict 2.5's 2B and Reason2's 8B combined, it only uses about 45GB, leaving plenty of room. I hope this is helpful as a reference for those interested in world foundation models when choosing their environment.
