I tried running NVIDIA Nemotron 3 Nano Omni on DGX Spark

Introduction

Hello, I'm Mori Shigeru from Classmethod's Manufacturing Business Technology Department.

NVIDIA released Nemotron 3 Nano Omni on April 28, 2026. It is an Omni model that handles four types of input—text, image, audio, and video—within a single model, using a 30B-A3B Mamba+Transformer hybrid MoE architecture. NVIDIA's official blog highlights "7.4× multi-document and 9.2× video throughput compared to other open Omni models" and "20% improvement in multimodal intelligence," and the release was accompanied by simultaneous support across Hugging Face, OpenRouter, build.nvidia.com, Ollama 0.22, and SageMaker JumpStart all on the same day.

https://developer.nvidia.com/blog/nvidia-nemotron-3-nano-omni-powers-multimodal-agent-reasoning-in-a-single-efficient-open-model/

Since DGX Spark and DGX Station are explicitly listed as officially supported GPUs in the vLLM official blog, I immediately ran it on my DGX Spark. Three quantization variants—NVFP4 (~21 GB), FP8 (~33 GB), and BF16 (~62 GB)—were published simultaneously, so I compared startup behavior, latency, and output quality across each while testing all four modalities: text, image, audio, and video.

Let me share my hands-on impressions upfront. The NVFP4 variant (~21 GB) on DGX Spark handles all four modalities with short turnaround times—image (PPE detection in 2 seconds), audio (transcribing "Mary had a little lamb" in 0.85 seconds), and video (understanding a 15-second conveyor belt scene in 0.64 seconds) all worked normally. I also sent the same PPE image to Gemma 4 31B IT NVFP4 (vLLM 0.20.0) under the same quantization and backend conditions, which returned in 9.78 seconds. Even with the same NVFP4, the difference of 0.67 seconds vs. 9.78 seconds is the key takeaway of this early report for getting a feel for the Omni model's efficiency.

Overview of Nemotron 3 Nano Omni

Model Family

All three variants accept text + image + audio + video as input and produce text as output, with a 256K context. They use a Mamba2 + Transformer hybrid architecture, integrating CRADIO v4-H as the image encoder and Parakeet as the audio encoder. The encoder portions are kept in BF16 without quantization, so input quality is not degraded.

Architecture Highlights

The hybrid MoE activates only 3B out of the total 30B parameters during inference, achieving high throughput while preserving expressive capacity. Video frames are pruned using Conv3D + EVS (Efficient Video Sampling) to remove redundant frames, with --video-pruning-rate 0.5 as the default recommendation. On the generation side, Reasoning mode is ON by default, and passing --reasoning-parser nemotron_v3 allows separation of reasoning content and the final answer using <think>...</think>. Tool calling is configured by specifying --tool-call-parser qwen3_coder, which returns output in a Qwen3-compatible format.

License

NVIDIA Nemotron Open Model License. Commercial use is permitted, and weights, datasets, and training techniques are publicly available.

https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

Deployment Options

Nearly all options were available from day one of release. This article focuses on DGX Spark + vLLM 0.20+, but here is an overview of all available options.

On the SageMaker JumpStart side, the model ID is huggingface-vlm-nvidia-nemotron3-nano-omni-30ba3b-reasoning-fp8, with recommended instances of ml.p4d.24xlarge or ml.p5.48xlarge. It can be launched in just a few lines with JumpStartModel(...).deploy(accept_eula=True). For details, the AWS Machine Learning Blog article is helpful. If you need to keep data governance in-house or handle data that cannot leave the premises for licensing reasons, the local DGX Spark route is ideal; if you need peak scaling or multi-tenant operation, the JumpStart route is likely the better fit.

Test Environment

Setup Steps on DGX Spark

Installing vLLM 0.20.0

The vLLM official blog explicitly specifies vllm[audio]==0.20.0. Without the audio extras, sending audio to /v1/chat/completions will result in a ValidationError, so be sure to include the extras when specifying the package.

mkdir -p ~/works/nemotron3-omni && cd ~/works/nemotron3-omni
uv venv --python 3.12
source .venv/bin/activate
uv pip install "vllm[audio]==0.20.0"

Starting NVFP4

NVFP4 at ~21 GB fits comfortably within the DGX Spark's 128 GB UMA. The Hugging Face model card recommends specifying --moe-backend flashinfer_cutlass for NVFP4.

vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
  --served-model-name nemotron-omni \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.5 \
  --media-io-kwargs '{"video":{"num_frames":256,"fps":2}}' \
  --video-pruning-rate 0.5 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3 \
  --moe-backend flashinfer_cutlass \
  --trust-remote-code

Switching to FP8

Since --moe-backend flashinfer_cutlass is exclusive to NVFP4, remove it and update the model ID and memory settings.

# FP8 (balanced, ~33 GB)
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-omni \
  --gpu-memory-utilization 0.6 \
  --max-model-len 32768 \
  --media-io-kwargs '{"video":{"num_frames":256,"fps":2}}' \
  --video-pruning-rate 0.5 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3 \
  --trust-remote-code

BF16 Is Out of Scope for This Article

The BF16 variant carries a ~62 GB model body plus the image encoder (CRADIO v4-H, maintained in BF16) and audio encoder (Parakeet, maintained in BF16). On top of that, the KV cache (roughly 10 GB even with fp8_e4m3 at max_model_len 32768) is also required. Considering the remaining capacity after subtracting the OS and cache from DGX Spark's 128 GB UMA, even adding --cpu-offload-gb 16 makes it difficult to reach practical performance.

Quantization Variant Benchmark Comparison

The same prompt ("Explain in 3 short paragraphs how a hybrid Mamba+Transformer mixture-of-experts architecture differs from a dense Transformer LLM, focusing on inference efficiency.") was run for 5 rounds each, and latency and output throughput were measured. Reasoning mode was left ON at its default.

NVFP4 is a 4-bit format that runs natively on the DGX Spark's Blackwell GB10 (sm_121), delivering the smallest memory footprint while maintaining output quality. The numbers show approximately 16% lower latency and approximately 12% higher throughput compared to FP8. Since the encoder portions are kept in BF16, the input quality for images, audio, and video is not degraded.

Multimodal Operation Verification

Using NVFP4 as the backend, I sent all four modalities sequentially via the OpenAI-compatible /v1/chat/completions endpoint by mixing image_url, input_audio, and video_url within the messages[].content array. Since Reasoning mode tends to produce longer responses, for short-answer tasks like numerical comparisons or transcriptions, it is practical to pass chat_template_kwargs.enable_thinking=false.

For example, sending an image in base64 would look like this with curl:

IMG=$(base64 -w0 ppe-sample.jpg)
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"nemotron-omni\",
    \"messages\": [{\"role\": \"user\", \"content\": [
      {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,$IMG\"}},
      {\"type\": \"text\", \"text\": \"List the personal protective equipment visible in this image.\"}
    ]}],
    \"max_tokens\": 800,
    \"chat_template_kwargs\": {\"enable_thinking\": false}
  }"

Simply changing the type to input_audio (specifying the format as wav, etc.) or video_url (data:video/mp4;base64,...) enables audio and video input as well.

Text

Briefly describe what makes a multimodal omni model different from a text-only LLM.

Response in 4.87 seconds (277 tokens). An excerpt from the beginning:

A multimodal omni model differs from a text-only LLM in several key ways:

1. Multimodal Input/Output: Unlike text-only LLMs that only process and generate text,
   multimodal omni models can handle various data types, such as images, audio, video, and text.
2. Contextual Understanding: Multimodal omni models can better understand context by integrating
   information from different modalities. ...

Image (PPE Detection)

I used the same image as in the previous Cosmos-Reason2 structured reasoning article.

Response in 2.04 seconds (108 tokens).

Based on the image provided, the following personal protective equipment (PPE) is visible:

- Hard hats: Both individuals are wearing hard hats. The woman is wearing a white hard hat, and the man is wearing a yellow one.
- High-visibility vest: The woman is wearing a bright, lime-green high-visibility vest over her shirt.
- Safety glasses: The woman is wearing safety glasses, which are visible under her hard hat. The man is also wearing safety glasses, though they are less distinct.

The model described colors and wearing conditions in detail.

Audio (English, approximately 16 seconds)

A vLLM official asset (mary_had_lamb.ogg) was converted to 16 kHz mono WAV and sent.

Response in 0.85 seconds (43 tokens).

A man speaks in the original phonograph a little piece of practical poetry, Mary had a little lamb,
its fleece was white as snow, and everywhere that Mary went, the lamb was sure to go.

The model simultaneously transcribed and summarized the content, including the historical context of a phonograph recording.

Video (Conveyor Belt, 15 seconds)

I used a video of a cardboard box on a conveyor line from the previous VSS Event Reviewer verification article.

Response in 0.64 seconds (15 tokens).

A cardboard box is moving on a conveyor belt that is curved.

The response is concise and stays within the scope of a two-sentence summary of a short video.

Comparison with Gemma 4 31B

While the model architectures are fundamentally different, I wanted to see "how the Omni's size and latency compare when running a different VLM family under the same backend and quantization on DGX Spark." So I sent the same PPE sample image with the same prompt (instructing JSON output with ppe_items and overall_compliance) to both Nemotron 3 Nano Omni (NVFP4) and Gemma 4 31B IT NVFP4.

For launching Gemma 4 31B IT NVFP4 with vLLM, --reasoning-parser gemma4 --tool-call-parser gemma4 --max-num-batched-tokens 4096 was passed. Since the image encoder produces a large number of tokens, max_num_batched_tokens must be raised from 2048 to 4096 or the model will be rejected at initialization (see the vLLM official Gemma 4 recipe).

Speed and Size

There was approximately a 15× difference in actual inference time. Since both used NVFP4 + vLLM under identical conditions, this gap is almost entirely attributable to differences in model architecture. Omni uses a MoE (Mixture-of-Experts) structure where, despite holding 30B parameters, only 3B are actually traversed per token generated. Gemma 4 31B, being a dense model, traverses all 31B parameters for every token. Since the "number of active parameters per token" differs by roughly 10×, a ~15× latency difference is a natural outcome. This clearly demonstrates the MoE goal of "large model capacity, small inference cost" in the benchmark numbers.

Output

Both returned JSON, but with slight differences in detail and formatting.

// Nemotron 3 Nano Omni (NVFP4, 29 tokens, 0.67 s)
{ "ppe_items": ["hard hat", "safety vest"], "overall_compliance": "compliant" }

// Gemma 4 31B IT NVFP4 (57 tokens, 9.78 s)
{
  "ppe_items": ["hard hat (white)", "hard hat (yellow)", "high-visibility safety vest"],
  "overall_compliance": "non_compliant"
}

Gemma 4 returned the output wrapped in a Markdown code fence and broke down helmets by color. Omni straightforwardly returned JSON as instructed.

The compliant vs. non_compliant discrepancy reflects different interpretations of the image. Omni judged "the PPE visible in the image is all present → compliant," while Gemma 4 judged "non_compliant including equipment not visible in the image, such as safety glasses." Both are reasonable judgments, and this is the kind of difference where, when integrating into a production workflow, you need to explicitly define "what constitutes compliance" in the prompt—otherwise verdicts will differ between VLMs.

Usage Guidelines

Handling Japanese

According to the Hugging Face model card, Nemotron 3 Nano Omni was trained only on English, and no Japanese-derived version (such as the past nvidia/Nemotron-Nano-9B-v2-Japanese) has been released this time. Since Japanese tasks are not included in the benchmark tables, past articles such as the Gemma 4 benchmark or the Nemotron 9B-v2-Japanese hands-on are more informative for Japanese-specific scores (specific figures are shown in the table later).

However, when I tried sending Japanese prompts, the model responded normally in Japanese. I ran two queries—one text and one image—using the same NVFP4 setup (with enable_thinking=false).

Text (25.08 s, 279 tokens)

Response to the prompt "Please briefly explain how a multimodal integrated model like Nemotron 3 Nano Omni differs from a text-only LLM." (excerpt from the beginning):

Nemotron 3 Nano Omni のようなマルチモーダル統合モデルは、テキストのみの LLM と以下の点で異なります：

1. 多様な入力モードの処理：テキスト以外の入力（画像、音声、動画など）を理解・生成できる。
2. 統合的推論：テキストと他のモードを同時に処理し、文脈に応じた回答や生成が可能（例：画像の説明＋テキスト生成）。
3. 文脈の豊かさ：視覚・聴覚情報を組み込むことで、より豊かで正確な文脈理解が可能。
4. 応用領域の拡大：視覚的質問回答、動画解説、音声認識連携など、テキストLLMでは不可能なタスクを実行。

簡潔に：テキストLLMは「言葉だけ」の世界、マルチモーダルモデルは「言葉＋視覚・聴覚」の世界を扱う点が主な違いです。

Proper nouns remained in English while explanations were returned in structured Japanese.

Image (PPE Detection, 1.17 s, 31 tokens)

Sending the same PPE sample image with the prompt "Please list the personal protective equipment (PPE) visible in this image in Japanese using bullet points." returned:

- 白い安全ヘルメット
- 緑色の安全ベスト
- 黄色い安全ヘルメット

The model listed items including colors in natural Japanese. The level of detail did not seem to drop significantly compared to the English response ("hard hat / safety vest").

Although this is a model trained for English benchmarks and no official Japanese performance scores have been measured, the fact that simply switching the prompt to Japanese yields normal Japanese responses is reassuring for anyone looking to use English-focused tutorials directly in Japanese-language workflows.

Reference: Running JCommonsenseQA Scores

Since "it responded in Japanese" alone relies on subjective impressions, I ran all 1,119 validation questions from JCommonsenseQA v1.1 on both Omni and Gemma 4 31B IT NVFP4 using the same 3-shot prompts used in the previous Gemma 4 series. Both used vLLM 0.20.0 + NVFP4 as in this article.

It was confirmed that Gemma 4 31B IT retains nearly all its accuracy at 97.9% → 97.77% even when quantized to NVFP4. Omni achieved 88% despite being trained only on English with relatively small active parameters (3B), demonstrating a reasonable ability to handle Japanese commonsense reasoning. While it falls short of Nemotron-Nano-9B-v2-Japanese (91.2%), it was honestly surprising to see Omni—which handles text, image, audio, and video in a single model—reach this level. A full Japanese multimodal evaluation (Heron-Bench, JMMMU, etc.) is something I'd like to cover in a separate article.

Pitfalls

--media-io-kwargs Values Differ Between Sources

The vLLM official blog specifies num_frames=512, fps=1, while the Hugging Face model card specifies num_frames=256, fps=2—the recommended values differ depending on the source. How many frames are ingested per video and at what fps directly affects VRAM usage and inference latency, so it's best to lock in these values early based on your hardware constraints. In this case, I used the HF card's 256 / fps=2 and was able to run the 15-second conveyor belt video without VRAM pressure.

Insufficient max_tokens in Reasoning Mode Can Result in Empty Content

Nemotron 3 series generates reasoning content within <think>...</think> before producing the final answer. However, when I requested a two-sentence video description with max_tokens=300, the reasoning consumed all 300 tokens and the content came back empty (null). Passing chat_template_kwargs.enable_thinking=false immediately returned A cardboard box is moving on a conveyor belt that is curved. in just 13 tokens. The practical approach is to disable thinking for short-answer verification and transcription tasks, and to enable thinking with around max_tokens=1500 when you want to exercise the model's reasoning capabilities.

Summary

I was able to verify on DGX Spark that all four modalities work on the day of release. NVFP4 fits within ~21 GB, making it the most balanced variant among the three for daily use in terms of startup, latency, and throughput. The 0.67-second vs. 9.78-second difference compared to Gemma 4 31B IT running under the same NVFP4 + vLLM conditions clearly demonstrates in benchmark numbers the MoE characteristic of maintaining 30B-scale model capacity while limiting inference cost to just the active 3B. The JCommonsenseQA results—88% for Omni and 97.77% for Gemma 4 31B IT at NVFP4 (only 0.13 percentage points below BF16)—show negligible accuracy degradation from quantization, and it was a pleasant discovery to find that even an English-trained Omni model can handle Japanese commonsense reasoning at a reasonable level.

Reference:

• NVIDIA Developer Blog: Nemotron 3 Nano Omni
• vLLM official blog: Nemotron Omni
• Hugging Face: Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4
• AWS Machine Learning Blog: Nemotron 3 Nano Omni on SageMaker JumpStart