I tried running NVIDIA Nemotron 3 Super on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

On March 11, 2026, NVIDIA released Nemotron 3 Super. It is a reasoning model for agents that adopts a hybrid architecture with 120B total parameters and 12B active parameters. For details on the architecture and a trial run on Cloudflare Workers AI, please refer to Oguri-san's article.

https://dev.classmethod.jp/articles/nvidia-nemotron-3-super-cloudflare-workers-ai/

This time, we focus on local execution. In a previous article, we ran Nano (30B-A3B), the lightweight model from the same Nemotron 3 family, on DGX Spark. This time, we scale up from there to take on the 120B Super. Since we're also curious about the performance difference from Nano, we compared them using the JCommonsenseQA benchmark.

Overview of Nemotron 3 Super

Nemotron 3 is a new generation model family that NVIDIA trained from scratch with its own original architecture. It belongs to a different lineage from the Nemotron Nano 9B v2 Japanese (a Llama-based fine-tune) covered in a previous series.

Super is a hybrid configuration that alternately stacks three types of blocks: Mamba-2, Transformer Attention, and Latent MoE (Mixture of Experts). Each is designed to handle long context processing, accurate reference retrieval, and 4x expert utilization at the same cost, respectively. It also has a built-in multi-token prediction (MTP) head, enabling acceleration through native Speculative Decoding without an external draft model.

An important point to consider when thinking about running it on DGX Spark is that it was pre-trained from scratch in NVFP4 (4-bit floating point). Rather than quantizing after the fact, because it was trained from the start under the constraints of 4-bit precision, it is natively well-suited to the NVFP4 optimizations of the Blackwell architecture. However, as described later, as of March 2026 there were issues on the inference engine side in my environment, so I verified using Ollama's GGUF version.

Setting Up on DGX Spark

Test Environment

The NVFP4 Version Didn't Work

NVIDIA's official cookbook provides instructions for launching with vLLM 0.17.1 + NVFP4 checkpoints. However, the DGX Spark (GB10) has CUDA Compute Capability 12.1 (sm_121), and torch 2.10.0+cu128, which vLLM 0.17.1 depends on, only supports up to sm_120. Since the CUTLASS kernel crashes at runtime, it was not possible to run the NVFP4 variant with vLLM at this time.

I also tried NGC containers (26.01, 26.02), but 26.01 uses vLLM 0.13.x and does not support the Nemotron 3 Super architecture (nemotron_h), while 26.02 uses vLLM 0.15.x and fails to interpret the MIXED_PRECISION quantization.

TRT-LLM has an official configuration for DGX Spark (Config C), but since it requires building from the main branch rather than a release version, I passed on it this time. I hope this will be resolved in NGC 26.03 or later, or with an official TRT-LLM release.

Running with Ollama

With the GGUF format, CUDA kernel compatibility issues can be avoided. Ollama has published nemotron-3-super with Q4_K_M quantization, and it ran without issues on DGX Spark.

# Pull the model with Ollama (approx. 87GB)
ollama pull nemotron-3-super

Nano can be pulled in the same way.

# Also pull Nano for comparison (approx. 24GB)
ollama pull nemotron-3-nano

As long as Ollama is running, all you need to do is call the API. The initial model load takes about 1–2 minutes, but once loaded, responses come back smoothly.

Memory Usage

Since the DGX Spark's unified memory is shared between CPU and GPU, it's difficult to check accurate usage with nvidia-smi. When I checked using Ollama's ps command, Super was using approximately 87GB and Nano approximately 24GB. Against the 128GB unified memory, there is sufficient headroom for Super alone, but loading both models simultaneously totals around 111GB, so you need to be mindful of other running processes.

Running Inference

Reasoning Mode (with thinking)

Nemotron 3 Super has a Reasoning mode that outputs the reasoning thought process, similar to DeepSeek-R1. In Ollama, this can be controlled with the think parameter.

curl -s http://localhost:11434/api/chat -d '{
  "model": "nemotron-3-super",
  "messages": [
    {"role": "user", "content": "Explain the key innovations of Mixture of Experts architecture in 3 sentences."}
  ],
  "stream": false
}' | jq '{model, eval_count, eval_duration_ns: .eval_duration,
         tok_per_sec: (.eval_count / (.eval_duration / 1e9))}'

By default, it operates with Reasoning ON (with thinking), unfolding the thought process internally before generating a response. Because more tokens are generated, latency increases, but accuracy improves for complex reasoning tasks.

No-thinking (nothink) Mode

If you want to skip the thought process and get a direct answer, specify think: false.

curl -s http://localhost:11434/api/chat -d '{
  "model": "nemotron-3-super",
  "messages": [
    {"role": "user", "content": "What is 2+2?"}
  ],
  "think": false,
  "stream": false
}'

This mode is suitable for situations where short answers are required, such as benchmarks and classification tasks.

Throughput Measurement

We compared the throughput of Nano and Super using the same prompt.

The fact that Nano is approximately 4x faster in generation speed closely matches the 4x difference in active parameters between 3B and 12B. On the other hand, Super generates more tokens due to the long thought process in Reasoning mode, which leads to a difference in the quality of answers.

JCommonsenseQA Benchmark

Why This Benchmark?

JCommonsenseQA is a 5-choice dataset that tests common sense reasoning in Japanese. In a previous series, we evaluated Nemotron Nano 9B v2 Japanese (previous generation) on the same DGX Spark, providing a comparison baseline. Since the Nemotron 3 generation was trained mainly on English training data, it also serves as a gauge of how well it can handle Japanese tasks.

Measurement Conditions

Results

Nemotron 3 Super recorded the highest accuracy at 94.4%. The previous generation's 9B v2 Japanese (with Japanese-specific fine-tuning) scored 91.2%, so it's a bit surprising that it was surpassed without any Japanese-specific training. It feels like the reasoning power of the 12B active parameters is directly reflected.

Nano also performed admirably at 87.0%. In terms of latency, Nano is overwhelmingly faster at 0.30s/question, which is 3x faster than Super's 0.92s/question. However, since Super also finishes under 1 second per question, it was within a range that felt stress-free to use.

Precision for Super, speed for Nano. The distinction by use case is very clear.

Expectations for vLLM + NVFP4 and MTP

This time we verified using Ollama + GGUF, but NVIDIA's officially recommended configuration is vLLM 0.17.1 + NVFP4 checkpoints. TRT-LLM Config C is also available for DGX Spark.

Furthermore, Nemotron 3 Super has a multi-token prediction (MTP) head built into its checkpoint, enabling native Speculative Decoding without an external draft model. In vLLM, it can be enabled simply by adding the --speculative-config option, and the official documentation claims up to 3x speedup for structured generation tasks.

# To enable MTP (vLLM)
vllm serve $MODEL_CKPT \
  --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}'

Once vLLM + NVFP4 becomes operational on DGX Spark, higher throughput than the GGUF version can be expected, and MTP-based acceleration can also be tested. I look forward to the next version of the NGC container and the official TRT-LLM release.

June 2026 Update: NVFP4 Reaches Practical Speed with the Official NGC Container

In March, I ended with the expectation that "things should improve as the NGC official container and vLLM 0.17+ gain sm_121 support," and about three months later, the situation has changed considerably. When I revisited the DGX Spark for an upcoming event, the NVFP4 version was working out of the box with the NGC official vLLM container, so I'm adding the results here.

I used nvcr.io/nvidia/vllm:26.03.post1-py3. One of the causes of the failure in March—the MIXED_PRECISION quantization interpretation failure in the NGC 26.02 series—was resolved in 26.03, and I was able to load nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 directly.

docker run --rm --gpus all --ipc=host --network host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_FLASHINFER_MOE_BACKEND=latency \
  nvcr.io/nvidia/vllm:26.03.post1-py3 \
  vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --trust-remote-code --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --kv-cache-dtype fp8 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

What reassured me when looking at the startup log was that FLASHINFER_CUTLASS was selected as the MoE backend for NVFP4. In March, it had fallen back to Marlin weight-only and dropped to 4.8 tok/s, but this time FlashInfer CUTLASS was selected, even though Marlin was among the candidates. The float32 setting for mamba_ssm_cache_dtype is now handled automatically by 26.03, so the option I had to pass manually in March is no longer needed.

Measured Throughput

Results re-measured in a warm state.

Compared to 4.8 tok/s via Marlin in March, Super is approximately 4x faster. Enabling MTP (multi-token prediction) improved from 14.8 to 18.6 tok/s, an increase of about 26%. Since Super has an MTP head built into its checkpoint, it can be used simply by passing --speculative-config.

Summary

There is real impact in being able to run a 120B parameter model on a single desktop machine. Scoring 94.4% on JCommonsenseQA without Japanese-specific fine-tuning demonstrates the high baseline Japanese language capability.

It's also interesting that the distinction is clear—Nano for speed, Super for accuracy—and on DGX Spark you can load both and switch between them depending on your use case.

This time the NVFP4 version did not work due to sm_121 compatibility issues, but with Ollama + GGUF it was possible to run inference at a sufficiently practical speed. As native NVFP4 support progresses, further performance improvements—including MTP-based acceleration—can be expected.

I'll also be keeping an eye on further news about the Nemotron 3 family at GTC 2026 (3/16-19).

Reference Links

• NVIDIA が最新オープンモデル Nemotron 3 Super を発表したので Cloudflare Workers AI で試してみた（DevelopersIO）
• DGX Spark で Nemotron 3 Nano を日本語ファインチューニングしてみた（DevelopersIO）
• Introducing Nemotron 3 Super（公式ブログ）
• Inside Nemotron 3: Techniques, Tools, and Data（技術解説）
• Nemotron 3 Super 技術レポート
• NVIDIA-NeMo/Nemotron（GitHub、cookbook）
• HuggingFace: NVFP4 モデル