I thought about an LLM configuration that partners with one DGX Spark

Introduction

Hello, I'm Morimo from Classmethod's Manufacturing Business Technology Division.

NVIDIA has released nvidia/Qwen3.6-27B-NVFP4, a NVFP4 quantization of Qwen3.6-27B. On X, benchmarks have been circulating claiming it is "40% faster in a single session than the Unsloth version on DGX Spark," and many people seem to be curious about it.

https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4

I use one DGX Spark as a support machine, accessed over the network from my laptop. I don't intend to dedicate it exclusively to LLMs — I also want to run a vector DB for RAG, embedding models, and occasionally image generation with ComfyUI, all on the same machine. In other words, rather than simply looking for the fastest model in a one-off benchmark, I want to decide "if I'm going to operate this single machine as a partner, what model configuration should I use?" I had touched various models individually — fast models, smart models, models good at Japanese — but had never measured them side by side under the same conditions. The release of Qwen3.6-27B-NVFP4 became a good opportunity, so I gathered the candidates, remeasured them all, and thought things through.

To state the conclusion upfront: generation speed on a single DGX Spark is largely determined by "the amount of weights read to produce one token." MoE models with small active parameters have the advantage, and in raw speed, Nemotron Nano 30B-A3B came out ahead at 61 tok/s single. On the other hand, the more dense-like Qwen3.6-27B is limited to 12 tok/s in raw mode. However, Qwen ships with an MTP module for speculative decoding, and enabling it boosts it up to 29 tok/s — a 2.4x increase. The "fast" figure in the benchmarks at the beginning referred to this MTP-enabled number.

I wrote an article on 2026-06-28 comparing models on DGX Spark by Japanese language performance. This time I'm adding Qwen3.6-27B-NVFP4 to that comparison and re-measuring with a focus on the speed axis.

https://dev.classmethod.jp/articles/ornith-1-0-dgx-spark-japanese-benchmark/

In this article, using Qwen3.6-27B-NVFP4 as a starting point, I think through the model configuration for operating a single DGX Spark as a partner, along the axes of speed, Japanese language quality, and how well it can coexist with other use cases. I hope this is useful for those who want to run a local LLM persistently but are unsure which model to choose.

What Kind of Model is Qwen3.6-27B-NVFP4?

Let me first confirm the background of the main subject. The base is Alibaba's Qwen3.6-27B, which NVIDIA has quantized to NVFP4 using their in-house quantization tool ModelOpt (nvidia-modelopt 0.45.0). NVFP4 is a 4-bit floating-point format, and compressing from 16-bit to 4-bit reduces disk and GPU memory usage by approximately 2.5x. The actual downloaded checkpoint consisted of 3 files totaling 20.4GB.

Looking inside, there were several notable characteristics.

First, it is not purely dense but a hybrid architecture. Out of 64 total layers, every 4th layer is standard full attention, while the rest use linear attention called GDN (Gated Delta Net). Next, it is a reasoning model. When given a question, it first outputs a thought process in English before arriving at an answer. And it ships with one layer of MTP (Multi-Token Prediction) module for speculative decoding. This is the same mechanism seen in the DGX Spark version of DeepSeek V4 Flash, and it becomes relevant later.

The quantization content was MIXED_PRECISION rather than a single NVFP4. The linear attention parts use FP8, other linear layers use NVFP4, and the KV cache is also FP8. Context support extends up to 262K. The license is Apache 2.0, so it's a relatively easy-to-handle model in terms of provenance.

I'll also include the benchmark scores published officially. NVIDIA's claim is that accuracy is maintained at nearly the same level as pre-quantization FP8.

A score of 92.7 on AIME 2025 means it can solve quite a few difficult math problems. Being more on the dense side, this seems to be a model with high density of intelligence packed into each token.

Running It on a Single DGX Spark

Here's the main topic. The execution environment is based on the vanilla vllm/vllm-openai:v0.24.0-aarch64 (stable). No special community-built image is needed, and the launch command is just this:

vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 --port 8000 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching --trust-remote-code

I do apply a few environment variables. On DGX Spark, the FlashInfer sampler JIT build can fail, so it's recommended to pass VLLM_USE_FLASHINFER_SAMPLER=0. Looking at the startup logs, the hybrid linear attention layers used Triton/FLA kernels, and the FP8 layers used FlashInfer kernels, each selected appropriately, and it ran without issues.

Japanese responses came back normally. Calculations like "I bought 3 apples for 160 yen and 5 mandarin oranges for 200 yen. What's the total and the price per mandarin orange?" as well as Python code generation both reached the correct answers after going through the thought process. The behavior itself is stable.

How Much Does Speed Change Between Raw and MTP?

Now for the crucial speed. I forcibly generated 256 tokens with the same prompt and measured the pure decode speed excluding TTFT. First, in raw mode without MTP:

Single at 12.13 tok/s. The benchmark at the beginning showed 26 tok/s, so I'm only getting half of that. This is where MTP comes to mind. This DGX Spark-optimized checkpoint ships with an MTP module, so I added --speculative-config '{"method":"mtp","num_speculative_tokens":3}' and re-measured.

Single jumped from 12.13 to 29.38 tok/s — a 2.4x increase. This also closely matches the 26 tok/s from the earlier benchmark. What I was able to confirm on actual hardware is that the "fast" figure was not the raw speed but the MTP-enabled speed.

Why is a more dense-like model slow in raw mode? The decode speed on DGX Spark is roughly determined by "memory bandwidth 273 GB/s ÷ amount of weights read per token." Dense models read all parameters every token, so reading 27B worth of parameters hits the bandwidth ceiling. MTP is a mechanism that speculatively reads ahead multiple tokens in a single forward pass, so the effective speed increases by however many tokens are correctly predicted.

Measuring Side by Side with Other Everyday Candidates

With only Qwen to compare, it's hard to judge whether it's fast or slow. I measured other candidates that run on a single DGX Spark using the same harness and lined them up. The candidates were two NVIDIA Nemotron models (Nano for the speed slot and Super for the accuracy slot), Ornith which is also strong in Japanese, and the versatile Gemma. "Single" is the decode speed when only 1 request is processed, and "4 parallel" is the total throughput when 4 requests are processed simultaneously. For models that can use MTP, the 4-parallel values show MTP-enabled figures. Gemma alone was measured on nightly due to compatibility reasons described later.

Dark blue is raw speed, light blue is the MTP gain on top. Gemma 62.4 with external drafter and raw Nano 61.2 are nearly tied, a clear step ahead of the rest.

The fastest in raw speed was Nemotron Nano. Though large at 30B, its MoE architecture uses only 3B active parameters per token, achieving 61 tok/s raw. What determines speed is not total parameter count but the amount of weights actually read per token. In raw speed, the MoE models with small active parameters — Nano and Gemma — come out on top, followed by Super with Active 12B, then dense Ornith and Qwen. The reason Ornith doesn't stretch much despite being 9B is that it remains in bf16 without quantization, meaning bytes per parameter is 4x that of NVFP4.

Incidentally, Ornith also has a MoE 35B-FP8 variant (Active 3B), which measured 37.5 tok/s in the previous article — 3x faster than the 9B. However, the Japanese free-form evaluation rated the 9B higher, so in this article I list the 9B as the quality candidate. The principle of "Active parameters × quantization determines speed" applies equally when choosing between variants within the same family.

MTP is an add-on only for models that support it. Of the checkpoints with bundled MTP modules, Qwen went from 12.1 to 29.4 — a 2.4x boost — and Super from 15.8 to 24.0 — a 1.5x boost. Super with 120B reaching 24 tok/s including MTP is sufficiently practical for an accuracy slot.

Then there's Gemma. The checkpoint doesn't include MTP, but Google officially distributes a drafter (a lightweight model for drafting) in a separate repository, and specifying it enables MTP. The additional download is only 832MB. This boosts it from 30.19 to 62.36 — a 2.1x increase — virtually tying with Nano for the top speed. Two speculative tokens are sufficient; increasing to 3 doesn't improve speed (the acceptance rate for deeper speculation drops and cancels out). Nano and Ornith don't have MTP mechanisms, so their raw numbers are their actual performance.

In terms of quality, all models returned correct answers for code generation and numerical calculations. The quality of Japanese free-form writing left Ornith a step ahead — something that was also confirmed by ELYZA-tasks scores in the previous article. However, Ornith always outputs its reasoning process and cannot be turned off. This tends to extend latency, making it more of a quality slot model than a speed slot one.

Gemma 4 Shines in Instruction Following and External Drafter

Gemma occupied an interesting position. Raw single at 30 tok/s, with TTFT also at 0.05 seconds, among the fastest. What really stood out was instruction following: the only model among the 5 that could comply with the constraint "explain in 50 characters or less" was Gemma. The others output lengthy reasoning processes, far exceeding the character limit. For use cases requiring short, precise responses, Gemma was clearly ahead.

Enabling MTP is just a matter of adding one line specifying the drafter repository to the launch command:

--speculative-config '{"method":"mtp","num_speculative_tokens":2,"model":"google/gemma-4-26B-A4B-it-assistant"}'

This drafter is the same one used in my earlier article verifying Gemma 4 MTP, where at the time (May version of vLLM) it was 47.9 tok/s. The same combination reaches 62.36 tok/s on today's nightly, with the acceptance rate also rising from 55% to 68%.

https://dev.classmethod.jp/articles/dgx-spark-gemma4-mtp-multi-token-prediction-bench/

Enabling MTP does not lose image input capability. Only the drafter side switches to text-only; the main model continues to run as VL. When I actually fed it an image with shapes drawn on it, it correctly returned "a red circle and a blue square" with speculative decoding still in effect.

One prerequisite: the NVFP4 version of Gemma 4 requires a relatively recent vLLM. In stable v0.24.0, the modelopt quantization path doesn't support Gemma's weight sharing (tie_word_embeddings) and won't launch; it works with the nightly build as of the time of writing. This is why only the Gemma measurements in this article were done on nightly.

Incidentally, Gemma also has a dense 31B-IT variant, which I also launched with the same vLLM. However, since it reads the full dense weights every token, it's only 6.9 tok/s single — quite slow. For everyday use on DGX Spark, the MoE 26B-A4B is the natural choice. Even within the same family, the principle of choosing the MoE variant with fewer active parameters holds.

Build Matters More Than Tuning Settings — But Not Always in the Expected Way

There was another finding worth keeping in mind around speed. On DGX Spark's NVFP4, vLLM build behavior varies quite a bit. And it wasn't as simple as "newer is faster."

First, I tried tuning Qwen's MTP through launch flags. Increasing max_num_batched_tokens, adding -O3, --async-scheduling, --enable-chunked-prefill. The results were essentially unchanged. These flags improve parallelism and prefill efficiency but don't affect single-stream decode speed.

Next, the build itself. Gemma's external drafter benefited straightforwardly from build improvements — the combination that was 47.9 tok/s on the May build reached 62.36 tok/s on today's nightly, with acceptance rate rising from 55% to 68%.

On the other hand, there was a pitfall with bundled MTP. Running Qwen and Super's bundled MTP on the latest nightly at time of writing, the apparent tok/s increases (Qwen 37.7 / Super 46.0), but the output becomes corrupted — just repeating the same tokens. Looking at the speculative decoding acceptance rate, it's stuck at either 0% or 100%, clearly different from the healthy distribution (55–92%, higher for the first speculation stage and lower for later stages in a staircase pattern). The corrupted repetitive output is "perfectly predicted" by the drafter, so it appears "fast" with 100% acceptance rate — that's the trick. Throughput benchmarks forcibly generate output and don't inspect the content, so chasing numbers alone would have been misleading.

On stable v0.24.0, the bundled MTP runs correctly with the same settings. Output is coherent, acceptance rate follows a staircase pattern, quality checks pass, and speed consistently achieves roughly 2x raw speed as shown in the table above. So this article uses v0.24.0 as the baseline (except Gemma, which uses nightly since v0.24.0 doesn't support it). Two lessons: for bundled MTP, stable is safer for now; and when using nightly, always verify that output is coherent. Also, tok/s numbers are dangerous to evaluate without looking at both the acceptance rate distribution and the actual output content.

Footprint Matters When Sharing One Machine with RAG and ComfyUI

So far it's been about speed, but thinking about everyday use brings up another axis: available memory.

My setup is to not dedicate this DGX Spark exclusively to LLMs, but to share it with a vector DB and embedding model for RAG, and occasionally ComfyUI for image generation. The 128GB unified memory would be roughly divided as follows:

This is where the --gpu-memory-utilization setting used in this benchmark becomes relevant. I had pushed it to 0.85 for benchmarking, but at this setting vLLM holds over 100GB combined for weights and KV cache. In fact, when I launched Nano at 0.85, the log showed KV cache alone reserving 81.6GiB. That leaves no room for RAG or ComfyUI. For shared use, it's more realistic to lower it to 0.3–0.4 and leave the rest free for other purposes.

Happily, lowering this value doesn't affect single-stream speed. What decreases is only the number of concurrent sessions and the supported context length. In practice, I ran Gemma 26B-A4B at 0.70 and 31B-IT at 0.50 for measurement, and single speeds for both were as expected.

From this perspective, here's a breakdown of "how low can gpu-util be lowered per model for shared machine use." The amount vLLM reserves is roughly this coefficient times the ~120GB visible from the GPU. As long as the weights load and there's enough KV for one laptop's worth of requests, that's sufficient, so smaller models can be set quite low. Note that this table is a rough estimate back-calculated from measured weight sizes. I did not re-launch all models at these values and verify them precisely.

Blue is vLLM's allocation, green is what's available for other uses. The 4 smaller models leave nearly 80GB free, but Super reverses this, with only 36GB remaining.

The 4 smaller models fit within gpu-util 0.3–0.35, leaving nearly 80GB free. Running a RAG vector DB and embedding model persistently, then spinning up ComfyUI with Flux only when needed — that's a realistic setup. Super alone uses 70GB just for weights, requiring gpu-util 0.7 or higher to have enough KV headroom, leaving only about 40GB free. Running image generation simultaneously would make SDXL barely feasible and Flux difficult.

Of course, running LLM and ComfyUI simultaneously means they'll compete for GPU compute and both will be slower. Embedding calls will add latency too. But given the assumption of a single personal user occasionally accessing from a laptop, simultaneous usage isn't that frequent. I think it's realistic to keep a small model persistently loaded at a modest gpu-util, with LLM always available to respond and image generation or index updates running in the freed-up space. Using the single DGX Spark as a jack-of-all-trades in place of a laptop's internal GPU, essentially.

The Optimal Choice Splits by Use Case

Let me organize the results by use case. On a single DGX Spark, there's no single "best model" — it depends on what you want to do.

For running persistent agents or having brisk conversations, Nemotron Nano 30B-A3B is the choice. In addition to single 61 tok/s, it scales to 254 tok/s at 8-parallel, suitable for handling multiple requests. Among the 5 models, it's the only one that achieves this speed in raw mode with no drafter, no flags, no additions.

For short, precise responses, Gemma 4 26B-A4B is the choice. In situations like tool calls or templated replies where you want answers without lengthy preambles, it was the most obedient among the 5. Adding the external drafter brings speed to 62 tok/s, on par with Nano, achieving both instruction compliance and speed. However, as mentioned, it requires a fairly recent vLLM.

For packing dense intelligence into one machine, Qwen3.6-27B-NVFP4 + MTP is the choice. The dense-style intelligence demonstrated by AIME 2025's 92.7 can be brought to a practical 29 tok/s with MTP. This seems suited for single-user heavy inference workloads.

For needing 120B-class accuracy, Super is the choice. With MTP it reaches 24 tok/s, which is practically fast enough for an accuracy slot. However, as seen earlier, it nearly monopolizes the memory of one machine. Ornith is for when Japanese text quality is the top priority. These two are choices for when you have a clear and specific purpose.

Beyond speed and quality, there are other selection axes. For image input capability, among today's candidates Qwen and Gemma are VL models supporting image input. Happily, MTP does not remove image input capability. I confirmed that both models respond correctly to image requests with speculative decoding active (speed measurements were done with text). Nano and Super are text-only, so multimodal requirements change the options.

Use patterns also matter. If you're just chatting through a UI like OpenWebUI, single speed and text quality dominate. If you're invoking tool calls from an agent (in my case, Hermes Agent), instruction compliance and parallel throughput become more important.

And honestly, this breakdown varies greatly by use case and individual. In my case, serious coding goes to Claude Code or Codex CLI — I don't intend for the local LLM to handle everything. On the agent side, I use LLM routing: light tasks go local, medium-and-above go to cloud through OpenRouter across multiple models. The role of local is as a receiver for light templated processing, pre-processing, and data that I don't want to send externally. I hope the numbers in this article can serve as material for deciding what to place in that "local slot."

With that said, my personal top candidate for everyday use is Qwen. When you have one partner machine, you end up wanting to rely on it for everything, so its well-rounded nature — 29 tok/s with MTP, image input support, and the intelligence density of AIME 92.7 — becomes compelling. I plan to start running a configuration that combines this with Nano for the persistent speed slot and Gemma for templated tasks requiring instruction compliance.

Summary

Using Qwen3.6-27B-NVFP4 as a starting point, I measured local LLMs for everyday use on a single DGX Spark side by side. What I found was that generation speed is largely determined by the smallness of active parameters, and that Qwen's reputation for being "fast" refers to its state with bundled MTP enabled. The difference between 12 tok/s raw and 29 tok/s with MTP is something you'd misread if you didn't know. MTP isn't a privilege exclusive to Qwen/Super with bundled checkpoints — Gemma can also go from 30 to 62 tok/s by adding the officially distributed external drafter. For an additional 832MB, that boost is too good to pass up.

The Qwen3.6-27B-NVFP4 that prompted all this runs smoothly on the vanilla stable image, and its well-rounded combination of intelligence, speed, and image input support makes it the axis of my intended partner configuration. Meanwhile, in terms of speed, raw Nano and Gemma with drafter are nearly tied and a clear step ahead, making these two the strongest choices for persistent use. Mixing and matching by use case seems like the way to go. Around NVFP4, vLLM build version affects both speed and behavior. For now, the recommendation is to keep the baseline split by model — "stable for bundled MTP, nightly for Gemma" — and whenever you change builds, verify not just the tok/s numbers but also the actual content of the output.

Reference Links

• nvidia/Qwen3.6-27B-NVFP4
• nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
• nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
• deepreinforce-ai/Ornith-1.0-9B
• nvidia/Gemma-4-26B-A4B-NVFP4
• google/gemma-4-26B-A4B-it-assistant — Official MTP drafter for Gemma 4 26B-A4B
• nvidia/Gemma-4-31B-IT-NVFP4
• MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies — Parallel benchmarks of Gemma 4 26B-A4B running on DGX Spark
• vLLM on the DGX Spark
• NVIDIA TensorRT Model Optimizer