I thought about an LLM configuration that partners with one DGX Spark
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
NVIDIA has released nvidia/Qwen3.6-27B-NVFP4, a NVFP4 quantization of Qwen3.6-27B. On X, a benchmark has been circulating claiming it's "40% faster in a single session than the Unsloth version on DGX Spark," and many people seem to be curious about it.
I use one DGX Spark as a support machine that I access over the network from my laptop. I don't intend to dedicate it exclusively to LLMs — I also want to co-locate a vector DB and embedding model for RAG, and occasionally image generation with ComfyUI, all on the same machine. In other words, rather than simply hunting for the fastest model in a one-off benchmark, I want to decide "if I'm going to operate this one machine as a partner, what model configuration should I use?" I had tried fast models, smart models, and Japanese-capable models separately, but had never measured them side by side under the same conditions. Qwen3.6-27B-NVFP4 turned out to be a good opportunity to measure the candidates together and think it through.
To state the conclusion upfront: on a single DGX Spark, generation speed is almost entirely determined by "the amount of weights loaded per token." MoE models with small active parameters have the advantage, and Nemotron Nano 30B-A3B came out ahead at 62 tok/s single. On the other hand, the more dense-like Qwen3.6-27B only reaches 12 tok/s in its base state. However, Qwen ships with an MTP module for speculative decoding, and enabling it boosts speed to 38 tok/s — three times faster. The "fast" in the benchmark at the beginning refers to this number with MTP included.
I wrote an article on 2026-06-28 comparing models on DGX Spark by Japanese language performance. This time I've added Qwen3.6-27B-NVFP4 to that comparison and re-measured with a focus on the speed dimension.
In this article, using Qwen3.6-27B-NVFP4 as a starting point, I'll think through the model configuration for operating a single DGX Spark as a partner, along the axes of speed, Japanese language quality, and how well it can coexist with other use cases. I hope this resonates with those who want to run a local LLM persistently but aren't sure which model to choose.
What kind of model is Qwen3.6-27B-NVFP4?
Let me first confirm the background of the main subject. The base is Alibaba's Qwen3.6-27B, which NVIDIA quantized to NVFP4 using their in-house quantization tool ModelOpt (nvidia-modelopt 0.45.0). NVFP4 is a 4-bit floating point format; being compressed from 16-bit to 4-bit, disk and GPU memory usage is reduced to roughly 2.5 times smaller. The checkpoint I actually downloaded was 3 files totaling 20.4GB.
Looking inside, there were several notable characteristics.
First, it is not a pure dense model but a hybrid configuration. Of the 64 total layers, every 4th layer is standard full attention, while the rest use linear attention called GDN (Gated Delta Net). Second, it is a reasoning model. When given a question, it first outputs its reasoning process in English before arriving at an answer. Third, one layer's worth of MTP (Multi-Token Prediction) module for speculative decoding is bundled in. This is the same mechanism I saw in the DGX Spark version of DeepSeek V4 Flash, and it comes into play later.
The quantization is not a single NVFP4 but MIXED_PRECISION. The linear attention layers use FP8, while the other linear layers use NVFP4, and KV cache is also FP8. Context length support goes up to 262K. The license is Apache 2.0, making it an easy-to-handle model in terms of provenance.
I'll also include the benchmark scores published officially. NVIDIA's claim is that it maintains accuracy nearly equivalent to the pre-quantization FP8.
| Benchmark | Score |
|---|---|
| MMLU Pro | 86.3 |
| GPQA Diamond | 85.5 |
| AIME 2025 | 92.7 |
| MMMU Pro | 74.3 |
| SciCode | 44.5 |
A score of 92.7 on AIME 2025 means it can solve quite a few difficult math problems. As befitting a dense-type model, it appears to be a model with high density of intelligence packed per token.
Running it on a single DGX Spark
Here's where the main content begins. The execution environment is the plain vllm/vllm-openai:nightly-aarch64, and the measurements in this article are based on the nightly (v0.23.1rc1 series) pulled at the time of writing (2026-07-05). Since improvements around NVFP4 on DGX Spark are moving fast, it's recommended that those trying this out use a relatively new nightly. No special community-built images are needed, and the startup command is just this:
vllm serve nvidia/Qwen3.6-27B-NVFP4 \
--host 0.0.0.0 --port 8000 \
--quantization modelopt \
--kv-cache-dtype fp8 \
--max-model-len 65536 \
--max-num-seqs 8 \
--gpu-memory-utilization 0.85 \
--enable-prefix-caching --trust-remote-code
I do apply a few environment variables. On DGX Spark, the FlashInfer sampler JIT build can fail, so it's recommended to pass VLLM_USE_FLASHINFER_SAMPLER=0. Looking at the startup logs, the hybrid linear attention uses Triton/FLA kernels, and the FP8 layers use FlashInfer kernels — each selected appropriately, and it ran without issues.
Japanese responses come back normally too. Calculations like "I bought 3 apples for 160 yen and 5 tangerines for 200 yen. What is the total and the price per tangerine?" and Python code generation both arrived at the correct answer after going through a reasoning process. Operation itself is stable.
How much does speed change between base and MTP?
Now for the crucial speed results. I forced generation of 256 tokens with the same prompt and measured pure decode speed excluding TTFT. First, in the base state without MTP.
Single is 12.47 tok/s. The benchmark at the beginning showed 26 tok/s, so it's only half that. This is where MTP comes to mind. This checkpoint optimized for DGX Spark includes an MTP module, so I added --speculative-config '{"method":"mtp","num_speculative_tokens":3}' and re-measured.
| Parallelism | Base tok/s | MTP enabled tok/s | Multiplier |
|---|---|---|---|
| single | 12.47 | 37.69 | 3.02 |
| 2 | 23.78 | 57.22 | 2.41 |
| 4 | 45.82 | 116.75 | 2.55 |
| 8 | 84.58 | 202.27 | 2.39 |
Single jumped from 12.47 to 37.69 tok/s — three times faster. This exceeds the 26 tok/s from the benchmark at the beginning, but that's due to vLLM build differences mentioned later; measuring with a slightly older build gives 28.31, which closely matches the benchmark figure. Either way, what I confirmed on actual hardware is that the "fast" from the introduction referred to numbers with MTP enabled, not the base speed.
Why are more dense-like models slow at baseline? DGX Spark decode speed is roughly determined by "memory bandwidth 273 GB/s ÷ amount of weights read per token." Dense models need to read all parameters every token, so reading 27B worth of parameters hits the bandwidth ceiling. MTP works by looking ahead and predicting multiple tokens in a single forward pass, so the effective speed increases by however many tokens it correctly guesses.
Measuring side by side with regular candidates
With just Qwen alone, it's hard to judge whether it's fast or slow. I measured other candidates that can run on a single DGX Spark with the same harness and lined them up. The comparison includes two NVIDIA Nemotron models (Nano for the speed slot and Super for the accuracy slot), Ornith which is also strong in Japanese, and the all-purpose Gemma. Single is the decode speed when flowing just 1 request, 4-parallel is the total when flowing 4 simultaneously, and for models with MTP, the 4-parallel value shown is with MTP enabled.
| Model | Configuration | Footprint | Base single | MTP single | 4-parallel |
|---|---|---|---|---|---|
| Nemotron 3 Nano 30B-A3B | MoE (Active 3B) | ~21GB | 61.76 | N/A | 173.50 |
| Gemma 4 26B-A4B | MoE (Active 3.8B) | ~18GB | 30.19 | N/A | 115.26 |
| Nemotron 3 Super 120B-A12B | MoE (Active 12B) | ~70GB | 17.07 | 45.95 | 49.50 |
| Ornith 1.0 9B | dense 9B (bf16) | ~18GB | 12.56 | N/A | 53.28 |
| Qwen3.6-27B-NVFP4 | dense-hybrid 27B | ~20GB | 12.47 | 37.69 | 116.75 |
Dark blue is the base speed, light blue is the gain with MTP enabled. Nano 61.8 leads by a clear margin, while Super and Qwen with MTP extend to 2.7–3x their base speed.
The fastest was Nemotron Nano. Although it's a large 30B, it's MoE and uses only 3B Active parameters per token, running at 62 tok/s at baseline. Compared to Super's 46 tok/s with MTP enabled, it's still more than 30% faster. What differentiates speed is not the total number of parameters but the amount of weights actually read per token. Looking at base speed, MoE models with small active parameters — Nano and Gemma — are at the top, followed by Super with Active 12B, then dense Ornith and Qwen. The reason Ornith doesn't scale up despite being 9B is that it's not quantized and remains in bf16, meaning it has 4 times more bytes per parameter than NVFP4.
MTP is a bonus only for models that include it. Qwen went from 12.47 to 37.69 — 3x — and Super went from 17.07 to 45.95 — 2.7x. Super in particular is a pleasant surprise: despite packing 120B, it's faster than Qwen with MTP included. However, Super's MTP scaling at parallelism is unstable, and the total for 4-parallel sometimes fell below 2-parallel (varying between 43–62 tok/s each measurement). The single speed is genuine, but for parallel workloads, Nano scales more cleanly. Nano, Gemma, and Ornith don't include MTP modules, so their base numbers are their true performance.
In terms of quality, code generation and numerical calculations returned correct answers from all models. Japanese free-form writing quality felt a notch above from Ornith, which was also confirmed by ELYZA-tasks scores in my previous article. However, Ornith always outputs its reasoning process with no option to turn it off, which tends to increase latency, making it more of a quality slot model rather than a speed slot model.
Gemma 4 stood out in instruction following
Gemma occupied an interesting position. At 30 tok/s single it's second fastest after Nano, and TTFT at 0.05 seconds is among the best. What really stood out was instruction following — Gemma was the only one among 5 models that respected the constraint "explain in 50 characters or less." The others output lengthy reasoning processes and significantly exceeded the character limit. For use cases where short, precise responses are needed, Gemma stood out above the rest.
One prerequisite: the NVFP4 version of Gemma 4 requires a reasonably new vLLM. With slightly older builds, the modelopt quantization path doesn't support Gemma's weight sharing (tie_word_embeddings) and it won't start. The nightly I used as the baseline in this article works without issues. Also, the recipe differs slightly from other models — I followed community-reported working configurations, setting the attention backend to triton_attn and the parser to gemma4.
Incidentally, there's also a dense 31B-IT version of Gemma, which I also launched with the same vLLM. However, since it reads the full dense weights every token, single is 6.9 tok/s, which is slow, and for regular use on DGX Spark the MoE 26B-A4B is more sensible. Even within the same family, the principle of choosing the MoE with fewer active parameters holds.
vLLM build matters more than configuration tuning
There was another finding worth keeping in mind around speed. On DGX Spark, NVFP4 speed varies considerably with the vLLM build.
First, I tried to squeeze Qwen's MTP through startup flags. I increased max_num_batched_tokens, and added -O3, --async-scheduling, and --enable-chunked-prefill. The results barely changed. These are flags that improve parallelism and prefill efficiency, and they don't affect single-stream decode.
On the other hand, when I measured the same Qwen MTP with a build from two weeks ago, single was 28.31 tok/s — nearly 30% lower than today's 37.69. This closer figure is what matches the 26 tok/s from the benchmark at the beginning. Since base speed was around 12.4 in both cases unchanged, it appears that the speculative decoding kernels for DGX Spark (GB10) were improved over those two weeks. The lesson from actual measurement is that keeping the image up to date is far more effective than tweaking settings.
Footprint matters when sharing one machine with RAG and ComfyUI
So far I've talked about speed, but for regular use there's another axis: free memory.
My assumption is not to dedicate this DGX Spark exclusively to LLMs, but to co-locate it with a vector DB and embedding model for RAG, and occasionally image generation with ComfyUI. The image is to divide the 128GB unified memory roughly as follows:
| Use case | Estimate |
|---|---|
| OS and resident processes | ~8GB |
| RAG (vector DB + embedding model) | ~10GB |
| ComfyUI (when in use, SDXL ~ Flux) | ~10-15GB |
| Remaining for vLLM | ~90GB |
What comes into play here is --gpu-memory-utilization used in this verification. I raised it to 0.85 for benchmarking, but at this setting vLLM reserves over 100GB combined for weights and KV cache. In fact, when I started Nano at 0.85, the log showed KV cache alone reserving 81.6GiB. That leaves no room for RAG or ComfyUI. For use as a shared machine, it's more practical to lower this to 0.3–0.4 and leave the rest open for other uses.
Happily, lowering this value doesn't change single-stream speed. All that decreases is the number of simultaneous sessions and the context length that can be loaded. In fact, for this measurement I started Gemma 26B-A4B at 0.70 and 31B-IT at 0.50, and the single speeds for both were as expected.
From this perspective, let me line up "how far can gpu-util be lowered for each model when using it as a shared machine." The amount vLLM reserves is roughly the ~120GB visible to the GPU multiplied by this coefficient. As long as the weights are loaded and enough KV for a single laptop user is reserved, smaller models can go quite low. Note that this table is a rough estimate back-calculated from measured weight sizes, not values I actually re-launched and tested for all models.
| Model | Weights | Recommended gpu-util | vLLM reserve | Available for others | Co-location estimate |
|---|---|---|---|---|---|
| Ornith 1.0 9B | ~18GB | 0.30 | ~36GB | ~84GB | RAG + Flux with room to spare |
| Gemma 4 26B-A4B | ~18GB | 0.35 | ~42GB | ~78GB | RAG + Flux |
| Qwen3.6-27B-NVFP4 | ~20GB | 0.35 | ~42GB | ~78GB | RAG + Flux |
| Nemotron 3 Nano 30B-A3B | ~21GB | 0.35 | ~42GB | ~78GB | RAG + Flux |
| Nemotron 3 Super 120B-A12B | ~70GB | 0.70 | ~84GB | ~36GB | RAG + SDXL (Flux is tight) |
Blue is the vLLM reserved portion, green is the portion available for other uses. The 4 smaller models leave nearly 80GB free, but Super alone reverses this, with only 36GB remaining.
The 4 smaller models fit within gpu-util 0.3–0.35, leaving nearly 80GB free. A realistic approach is to keep the RAG vector DB and embedding model always resident, and launch ComfyUI with Flux only when needed. Super alone uses 70GB for weights, so without raising gpu-util to 0.7 there's no KV headroom, and available space is under 40GB. Running image generation simultaneously makes SDXL barely feasible and Flux tight.
Of course, running LLMs and ComfyUI simultaneously means both compete for GPU compute and both slow down. Embedding calls add latency on top of that. However, given the premise of a sub-machine that an individual occasionally accesses from their laptop, simultaneous execution won't be frequent. I think a practical division is to keep a small model resident at conservative gpu-util, have the LLM always responsive, and run image generation and index updates in the remaining capacity. This treats the single DGX Spark as a jack-of-all-trades, serving as a replacement for the internal GPU in a laptop.
The optimal answer varies by use case
Let me organize the results so far by use case. On a single DGX Spark, the "optimal model" doesn't narrow down to one, and it splits depending on what you want to do with it.
If you want to keep an agent running or have fast-paced conversations, Nemotron Nano 30B-A3B is the answer. In addition to single 62 tok/s, it scales up to 259 tok/s at 8-parallel, making it suitable for handling multiple requests. The impression was that speed-first means starting here.
If you want short, precise responses, Gemma 4 26B-A4B is the choice. In situations like tool calling or templated responses where you want it to return exactly as instructed without unnecessary preamble, Gemma was the most straightforward among the 5 models. Speed at 30 tok/s is also sufficient. However, as mentioned earlier, it requires a fairly recent vLLM.
If you want to pack high-density intelligence into one machine, Qwen3.6-27B-NVFP4 + MTP is the answer. The dense-type intelligence reflected in AIME 2025's 92.7 is brought up to a practical speed of 38 tok/s with MTP. It seems well-suited for heavy inference workloads from a single user.
If you need the accuracy of 120B, Super is the choice. With MTP enabled, it reaches 46 tok/s, which is sufficient speed even for an accuracy-focused model. However, as seen earlier, memory usage almost monopolizes a single machine. If Japanese text quality is the top priority, Ornith. These two are options when the goal is clear.
Beyond speed and quality, there are other selection axes. If you want to feed in images, Qwen and Gemma among this round's candidates support image input in their VL configurations (I only measured text in this article). Nano and Super are text-only, so if multimodal is a requirement, the options change. Usage patterns also matter. If you're only having conversations through a chat UI like OpenWebUI, single speed and text quality matter most; if you're calling with tool calling from an agent (in my case, Hermes Agent), instruction following and parallel performance become more dominant.
And honestly, this division varies greatly by use case and person. In my case, I rely on Claude Code or Codex CLI for serious coding and don't intend for local LLMs to carry everything. On the agent side, I use LLM routing to direct lightweight tasks locally and medium-or-larger tasks through OpenRouter to multiple cloud models. The local machine's role is to handle light templated processing, preprocessing, and data I don't want to send outside. I hope the numbers in this article serve as material for deciding "what to put in that local slot."
With that in mind, my top personal use candidate is Qwen. Since I'll inevitably want to rely on a single partner machine for all sorts of things, its balance of MTP pushing 38 tok/s, image input support, and the intelligence density of AIME 92.7 is compelling. I plan to start operating with a configuration that combines that with Nano for the always-on speed slot and Gemma for templated processing that requires instruction following.
Summary
Using Qwen3.6-27B-NVFP4 as a starting point, I measured local LLMs for regular use on a single DGX Spark side by side. What I found is that generation speed is almost entirely determined by the smallness of active parameters, and that Qwen's reputation for being "fast" referred to the state with the bundled MTP enabled. The gap of 12 tok/s at base vs. 38 tok/s with MTP is something you'd misjudge without knowing.
The Qwen3.6-27B-NVFP4 that sparked this investigation ran cleanly on a plain nightly image, and given its balance of intelligence, speed, and image input support, I plan to make it the core of my partner configuration. On the other hand, MoE Nano leads by a clear margin in raw speed and is strong for always-on use cases. Dividing use cases makes sense. Since NVFP4 speed and supported models change with vLLM builds, it's recommended to always use a recent version as your baseline.
Reference Links
- nvidia/Qwen3.6-27B-NVFP4
- nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
- nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
- deepreinforce-ai/Ornith-1.0-9B
- nvidia/Gemma-4-26B-A4B-NVFP4
- nvidia/Gemma-4-31B-IT-NVFP4
- MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies — Parallel benchmark of Gemma 4 26B-A4B running on DGX Spark
- vLLM on the DGX Spark
- NVIDIA TensorRT Model Optimizer
