I tried running NVIDIA Cosmos world foundation models on DGX Spark
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
I think many people would be intrigued to hear that "you can generate videos that follow the laws of physics just by entering text." Cosmos, published by NVIDIA, is a platform designed to realize exactly this kind of "world foundation model." It consists of multiple models, including Predict 2.5, which generates videos from text and images, and Reason2, which understands video content and infers causal relationships and spatial structures.
However, world foundation models consume a large amount of memory, and until now, data center GPUs such as A100 or H100 have been required to run them. So this time, I decided to verify how far things could go on a DGX Spark with 128GB of unified memory. To cut to the chase, the 2B model ran comfortably, while the 14B model was abandoned after 3 system freezes...
This article covers both video generation with Cosmos Predict 2.5 and video understanding with Cosmos Reason2, sharing the benefits and limitations of 128GB unified memory along with concrete figures.
Overview of the Cosmos Family
Cosmos is a platform designed to accelerate the development of "Physical AI," and multiple models are available for different use cases. Let me organize the entire family, focusing on the two I tested this time.
Predicting the Future with Predict 2.5
Predict 2.5 is a model that generates physically plausible videos from text or image inputs. It was developed with autonomous driving simulation and robotics training data generation in mind, but it's also interesting simply as a tool for creating videos from text.
There are three modes depending on the type of input: Text2World, which creates videos from text alone; Image2World, which adds motion starting from an image; and Video2World, which generates a continuation of a video. Model sizes come in two varieties, 2B and 14B, with the 14B expected to produce higher-quality output.
Understanding Videos with Reason2
Reason2 is a VLM (Vision-Language Model) that analyzes video content and reasons about causal relationships, temporal transitions, and spatial structures. Fine-tuned on top of Qwen2.5-VL, it responds in natural language to questions like "what will happen next in this video?" or "what is the positional relationship between objects?" Two sizes are available: 2B and 8B.
Transfer 2.5 Was Skipped This Time
I also attempted Transfer 2.5, which generates realistic imagery from control signals (edge maps, depth maps, etc.), but even the 2B model requires 65.4GB of VRAM, and the process was killed with OOM on the DGX Spark. I'd like to try this again on another occasion.
Test Environment
| Item | Value |
|---|---|
| Device | NVIDIA DGX Spark |
| SoC | Grace Blackwell GB10 |
| Memory | 128GB unified memory (shared CPU/GPU) |
| CUDA | 13.0.2 |
| Driver | 580.126.09 |
| OS | Ubuntu 24.04 (DGX OS) |
The defining feature of DGX Spark is its "unified memory architecture," where the CPU and GPU share memory. In a typical GPU server, GPU memory (VRAM) and CPU memory (RAM) are physically separate, but on the DGX Spark, the 128GB can be flexibly allocated. This means models that would result in OOM on an RTX 4090 with 24GB VRAM have a chance of running as-is.
Running Cosmos Predict 2.5
Setup
The official repository offers two setup methods: Docker and uv. I used uv this time.
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5.git
cd cosmos-predict2.5
git lfs pull
uv sync --extra=cu130
Since DGX Spark uses the combination of ARM64 (aarch64) + CUDA 13.0, specifying --extra=cu130 is required. It does not work with CUDA 12.8 (for x86_64).
Downloading the model requires agreeing to the NVIDIA Open Model License on HuggingFace. You'll need to agree to the license on the Cosmos-Predict2.5-2B model page in advance and set HF_TOKEN as an environment variable.
The inference script uses examples/inference.py. The official README also mentions scripts/diffusers_inference.py, but at the time of writing, diffusers 0.35.2 does not have the Cosmos 2.5 pipeline implemented, resulting in an ImportError.
Generating Videos with Text2World
First, let's try generating a video from text alone. The input is specified via a JSON file.
{
"prompt": "Water slowly pouring from a glass bottle into a clear glass on a wooden table. Close-up view, natural lighting from a window.",
"inference_type": "text2world",
"num_steps": 35,
"seed": 42
}
.venv/bin/python examples/inference.py \
--model 2B/post-trained \
--batch_input_path inputs/water_pour.json \
--batch_output_path outputs/ \
--disable-guardrails
--disable-guardrails is an option that disables input/output filtering by Cosmos-Guardrail1. When guardrails are enabled, input prompts are safety-checked using a Blocklist and Qwen3Guard (LLM-based), generated videos are inspected by VideoContentSafetyFilter, and any detected faces are automatically blurred. Guardrail models require separate license agreement and download, so I skipped them this time since the goal was to verify video generation.
The first run will download the model, so expect a short wait. The 2B Text2World checkpoint is approximately 5GB.
Generation proceeds through 36 steps of denoising, with approximately 53 seconds per step. The total time came to approximately 32 minutes.
| Metric | Value |
|---|---|
| Generation time | Approx. 32 minutes (36 steps) |
| GPU memory | 27.7GB |
| GPU utilization | 96% |
| Power consumption | 88W |
| GPU temperature | 82°C |
| Output resolution | 1280x704 |
| Frame count | 93 frames (16fps) |
32 minutes per video is certainly not fast, but it's great to be able to generate videos with a world foundation model on a desktop-sized device without using an A100 or H100. Memory usage was 27.7GB, leaving plenty of headroom against the 128GB total.
Animating Images with Image2World
Next, I tried Image2World, which generates a video using a still image as input. I tested with two images: the official sample bus terminal image and a robot welding image.
.venv/bin/python examples/inference.py \
--model 2B/post-trained \
--batch_input_path inputs/bus_terminal_i2w.json \
--batch_output_path outputs/ \
--disable-guardrails
Image2World preserves the atmosphere of the input image while adding motion according to the prompt. From the bus terminal image, a video of a bus slowly departing was generated, and from the welding robot image, a video of sparks continuously scattering was produced.
Generation times and file sizes for the two subjects are as follows.
| Subject | Generation time | File size |
|---|---|---|
| Bus terminal | Approx. 31 min | 823KB |
| Robot welding | Approx. 33 min | 1.7MB |
Generation time is roughly the same as Text2World. Having an input image gives the impression that the composition and color palette are more stable. For use cases like "I want to animate a photo I took myself," Image2World seems more suitable.
The Story of Attempting the 14B Model and Freezing 3 Times
With 2B running stably, I next attempted the 14B model. At the time of research, the VRAM estimate for 14B Text2World was around 49GB, and I figured it should run on 128GB of unified memory.
To cut to the chase, all 3 attempts resulted in system freezes, requiring me to unplug the power cable each time.
| Attempt | Conditions | Result |
|---|---|---|
| 1st | No options | OOM during model load → system freeze |
| 2nd | --offload-tokenizer, stopped unnecessary services |
Model load succeeded (51GB) → freeze immediately after inference started |
| 3rd | --offload-diffusion-model + --offload-tokenizer |
Model load succeeded (106GB/121GB) → freeze immediately after inference started |
On the 3rd attempt, I even used the option to offload the diffusion model to CPU, and the model load itself succeeded. However, the moment inference began, intermediate tensors from autoregressive generation overflowed memory, causing the system to become unresponsive.
With DGX Spark's unified memory, when GPU memory is exhausted, the entire system is taken down with it. On a typical GPU server, only the GPU process would be killed while the OS survives, but on DGX Spark, since the GPU and CPU share memory, a GPU-side OOM triggers a kernel-level hang. There were even cases where a soft reboot failed to initialize the GSP firmware and the GPU could not recover, requiring a cold reboot by physically cutting the power...
The conclusion is that 128GB is not enough to run the 14B model. It would likely require 256GB or more of unified memory, or a multi-node configuration.
Understanding Videos with Cosmos Reason2
Setting Up the Environment on DGX Spark
If Predict 2.5 is a model that "creates videos," then Reason2 is a model that "reads videos." Reason2-8B is a Qwen2.5-VL-based VLM that reasons about causality and temporal transitions of events captured in video.
Running Reason2 on DGX Spark required a bit of work. Reason2 is a multimodal model based on Qwen2.5-VL, and since the FlashAttention sm_121a kernel is not available, it cannot be launched in vLLM server mode. I adopted a configuration that performs inference directly with transformers.
from transformers import Qwen2_5_VLForConditionalGeneration
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"nvidia/Cosmos-Reason2-8B",
torch_dtype=torch.bfloat16,
attn_implementation="eager", # FlashAttention not supported on sm_121a
).to("cuda")
The key is attn_implementation="eager". The default SDPA and FlashAttention kernels are not available for DGX Spark's GPU (Blackwell, sm_121a), causing crashes with cuDNN errors. Switching to eager attention works around this. The model size is approximately 17.5GB in BF16, with a load time of approximately 89 seconds.
Running Inference on 3 Types of Material
I tested Reason2's capabilities on three types of videos: causal inference, temporal transitions, and spatial reasoning.
Causal Inference with a Video of Water Overflowing
I input a video of water being poured into a glass until it overflows and asked "what is happening in this video?"
Reason2 described the process of the glass filling with water in chronological order, including detailed observations such as the surface rising with bubbles until it reached the rim, and light refraction.
| Language | Token count | Inference time | TPS |
|---|---|---|---|
| English | 73 | 21.4 sec | 3.4 |
| Japanese | 197 | 33.9 sec | 5.8 |
Reason2 Japanese response (video of pouring water)
The video shows a transparent glass placed in a silver sink being filled with water flowing down from above. Some water is already in the glass, and the new stream of water pushes it up while adding more. The water surface rises slowly with bubbles, eventually reaching the rim of the glass. In this process, it can be observed that as water enters the glass, the internal space is filled, causing the water level to rise. Additionally, slight water droplets remain on the outside of the glass, and the effects of light through reflection and refraction can also be seen on its surface.
Temporal Transition with a Video of Cracking Eggs
For the video of cracking eggs into a frying pan, Reason2 recognized the egg yolk and white already in the pan, and also read the action of another hand cracking the next egg into it. It captured the situation of trying to cook multiple eggs at the same time, demonstrating an understanding of temporal progression.
| Language | Token count | Inference time | TPS |
|---|---|---|---|
| English | 142 | 28.7 sec | 5.0 |
| Japanese | 80 | 21.0 sec | 3.8 |
Reason2 Japanese response (video of cracking eggs)
Another egg yolk and white have accumulated in the frying pan. Meanwhile, another hand is holding the already-cracked shell and placing the next egg in. The remaining half is still being added. At this point, an attempt is being made to cook two eggs at once, so both will set shortly after.
Spatial Reasoning with an Overhead Video of a Parking Lot
For the overhead video of a parking lot, the overall layout (rows of cars, green strips between rows, sidewalks, surrounding plants, etc.) was read fairly accurately. However, the specific count (76 cars) may not be accurate, and in retrospect the material may have been a bit inappropriate...
| Language | Token count | Inference time | TPS |
|---|---|---|---|
| English | 512 | 67.9 sec | 7.5 |
| Japanese | 228 | 37.1 sec | 6.1 |
Reason2 Japanese response (overhead video of a parking lot)
76 cars are visible in the image. These cars are parked in an orderly arrangement. They are lined up vertically from left to right. There are green spaces between each row, with trees planted in them. There is also a sidewalk around the parking lot, where people can be seen passing by. Grass and other plants can be seen around the perimeter of the parking lot.
GPU memory usage for the 8B model was stable at 18.1GB, lighter than Predict 2.5's 2B (27.7GB). Even using both models alternately, there is plenty of headroom within 128GB.
Summary
Here is a review of the results of running the Cosmos family on DGX Spark's 128GB unified memory.
Predict 2.5's 2B model ran stably for both Text2World and Image2World. While it takes about 30 minutes to generate one video, being able to try world foundation model video generation on a desk-side device without an A100 or H100 was, personally, quite an interesting experience.
On the other hand, the 14B model was abandoned after 3 system freezes. Even with --offload-diffusion-model to offload to CPU, intermediate tensors during inference exceed 128GB, making it impossible to run on a single DGX Spark. Running the 14B model would likely require an environment with 256GB or more, or a multi-node configuration.
Reason2's 8B ran smoothly at 18.1GB, delivering practical quality for causal inference and temporal transition analysis. The high quality of Japanese output was also a pleasant discovery.
128GB of unified memory is an environment where you can try desk-side models that "can't run both at the same time even on an RTX 5090, but don't quite need an H100." Even combining Predict 2.5's 2B and Reason2's 8B only amounts to about 45GB, leaving plenty of room to spare. I hope this serves as a helpful reference for those interested in world foundation models when choosing their environment.
