Verification of video2video using Cosmos3

Verification of video2video using Cosmos3

Through work related to Physical AI, I used NVIDIA's cloud environment Brev and conducted verification of video-to-video data synthesis using Cosmos. I will introduce the environment setup procedure and implementation details.
2026.07.15

This page has been translated by machine translation. View original

It's been a while. I'm Kent, an intern.
While working on Physical AI-related tasks, I had the opportunity to use NVIDIA's cloud environment Brev and verify video-to-video data synthesis with cosmos3, so I'd like to share what I found.

About NVIDIA Brev

For the GPU cloud environment this time, I used NVIDIA Brev.
This service is similar to vast.ai, which became a topic of conversation a while ago, and allows you to rent GPUs from providers around the world for however long you need.
The fact that it's a service provided by NVIDIA itself made me feel comfortable using it.
The advantages include being cheaper than hosting on AWS or Google Cloud and the ease of setting up the environment.
In practice, with AWS and similar services you need to establish connection routes after launching an instance, but with Brev, Jupyter Notebook is available from the start, making it easy to use even for those unfamiliar with the cloud.

Setting Up cosmos3 (vLLM-Omni)

This time, I set it up on an instance equipped with an H100 (driver 580.x), serving Cosmos3-Nano via vLLM-Omni in a .venv native environment. The steps are as follows. Basically, following the setup instructions from Nvidia/Cosmos on Github should be fine, but some additional steps are required. Also, please note that the cosmos framework provided by NVIDIA does not support video input as of the time of writing (July 15, 2026).

1. Preparing uv

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv --version

2. Creating a venv (Python 3.13)

cd /home/shadeform
uv venv --python 3.13 --seed --managed-python

3. Logging in to Hugging Face

Cosmos3-Nano requires a safeguard for video and image generation, and since that safeguard is a gated model, you need to agree to the license on Hugging Face in advance. The model used is here: Cosmon-1.0-Guardrail
After agreeing, log in using a token. Since this is a rented cloud rather than a personally owned PC, it would be safer to grant only READ permission for the token you generate.

uvx --from huggingface_hub hf auth login --token $HF_TOKEN

4. Installing vLLM-Omni from the main branch

uv pip install --torch-backend=cu130 \
  "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"

5. Installing vLLM itself

Since vLLM itself is not added to the runtime environment in step 4, you need to install it here. This time, we're using version 0.25.0 to match the vLLM-Omni version.

uv pip install --torch-backend=cu130 "vllm==0.25.0"

6. Installing the Guardrail (Safety Checker)

If you haven't agreed to the model's terms of use on Hugging Face, an error will occur in a later step, so please refer to this page if necessary.

uv pip install --torch-backend=cu130 cosmos-guardrail

7. Verification

.venv/bin/vllm serve --help 2>&1 | grep -icE "Failed to load plugin|Traceback"

.venv/bin/python -c "import torch,vllm,vllm_omni; \
  print('torch',torch.__version__,'cuda',torch.cuda.is_available()); \
  print('vllm',vllm.__version__,'omni',vllm_omni.__version__)"

The expected output is as follows.

0
torch 2.11.0+cu130 cuda True
vllm 0.25.0 omni 0.25.0rc2.dev10+...

The 0 on the first line indicates that no plugin load errors occurred, and lines 2–3 confirm that torch recognizes the GPU with a CUDA 13 build and that the vllm and omni versions are aligned.

8. Starting the Server

source .venv/bin/activate
export HF_HUB_DISABLE_XET=1
vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800

Once Application startup complete. is displayed, the API will be available at http://localhost:8000.

9. Calling video-to-video

Use POST /v1/videos/sync (synchronously returns an MP4 directly) or POST /v1/videos (asynchronously returns a job ID). The video_reference is not passed as a file upload, but as a JSON string in the form {"video_url": "data:video/mp4;base64,..."}.

Converting video to a base64 data URL

python -c 'import base64,json; b=open("sample_input.mp4","rb").read(); \
open("vref.json","w").write(json.dumps({"video_url":"data:video/mp4;base64,"+base64.b64encode(b).decode()}))'

Running video-to-video

curl -X POST http://localhost:8000/v1/videos/sync \
  -F "model=nvidia/Cosmos3-Nano" \
  -F "prompt=Replace the background with a snowy mountain landscape at sunset, keep the main subject unchanged, cinematic lighting" \
  -F "negative_prompt=blurry, distorted, low quality" \
  -F "video_reference=<vref.json" \
  -F "num_frames=17" \
  -F "num_inference_steps=20" \
  -F "fps=8" \
  -F "seed=42" \
  -o sample_output.mp4

No description of video-to-video could be found in the official documentation, but I was able to call it using the method above.

Share this article