I thought about an LLM configuration that partners with one DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA has released nvidia/Qwen3.6-27B-NVFP4, a NVFP4 quantization of Qwen3.6-27B. A benchmark circulating on X claiming it's "40% faster in a single session than the Unsloth version on DGX Spark" has caught many people's attention.

https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4

I use one DGX Spark as a support machine, accessed over the network from a laptop. I don't intend to dedicate it solely to LLMs—I want to run a vector DB and embedding model for RAG, and occasionally image generation with ComfyUI, all on the same machine. In other words, rather than just looking for the fastest model in a one-off benchmark, I want to decide "what model configuration makes sense if I'm going to operate this one machine as a partner." I had tried fast models, smart models, and Japanese-capable models separately, but had never measured them side by side under the same conditions. The release of Qwen3.6-27B-NVFP4 gave me a good opportunity to measure all the candidates together and think it through.

To state the conclusion upfront: generation speed on a single DGX Spark is largely determined by "the amount of weights loaded to produce one token." MoE with small active parameters has the advantage, and in raw speed, Nemotron Nano 30B-A3B pulled ahead at single 61 tok/s. On the other hand, the more dense-leaning Qwen3.6-27B only reaches 12 tok/s raw. However, Qwen includes an MTP module for speculative decoding, and enabling it boosts speed 2.4x to 29 tok/s. The "fast" in the benchmark mentioned at the top was that MTP-enabled number.

I wrote an article on 2026-06-28 comparing models by Japanese performance on a single DGX Spark. This time I've added Qwen3.6-27B-NVFP4 to that comparison and re-measured with a focus on speed.

https://dev.classmethod.jp/articles/ornith-1-0-dgx-spark-japanese-benchmark/

In this article, using Qwen3.6-27B-NVFP4 as a starting point, I'll think through the model configuration for operating a single DGX Spark as a partner, along the axes of speed, Japanese quality, and how well it can coexist with other uses. I hope this resonates with those who want to run a local LLM persistently but aren't sure which model to choose.

What Kind of Model Is Qwen3.6-27B-NVFP4?

Let me first confirm the background of the main subject. The base is Alibaba's Qwen3.6-27B, which NVIDIA quantized to NVFP4 using their in-house quantization tool ModelOpt (nvidia-modelopt 0.45.0). NVFP4 is a 4-bit floating-point format; the compression from 16-bit to 4-bit reduces disk and GPU memory usage to roughly 2.5x smaller. The downloaded checkpoint consisted of 3 files totaling 20.4GB.

Looking inside, there were several notable characteristics.

First, it's not a pure dense model but a hybrid architecture. Out of all 64 layers, every 4th layer uses standard full attention, while the rest use a linear attention called GDN (Gated Delta Net). Second, it's a reasoning model. When you ask a question, it first outputs its thinking process in English before arriving at an answer. And third, it includes one layer of MTP (Multi-Token Prediction) module for speculative decoding. This is the same mechanism I saw in the DGX Spark version of DeepSeek V4 Flash, and it comes into play later.

The quantization details turned out to be MIXED_PRECISION rather than uniform NVFP4. The linear attention layers use FP8, other linear layers use NVFP4, and the KV cache is also FP8. Context support extends up to 262K. The license is Apache 2.0, making it fairly easy to work with.

I'll also include the benchmark scores published officially. NVIDIA's claim is that it maintains accuracy nearly equivalent to the pre-quantization FP8 version.

A score of 92.7 on AIME 2025 means it can solve a considerable portion of challenging math problems. As befitting a dense-leaning model, it seems to pack a high density of intelligence per token.

Running It on a Single DGX Spark

Now for the main topic. The execution environment is based on the standard vllm/vllm-openai:v0.24.0-aarch64 (stable). No special community-built image is needed; the startup command is just this:

vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 --port 8000 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching --trust-remote-code

I only set a few environment variables. On DGX Spark, the FlashInfer sampler JIT build can sometimes fail, so passing VLLM_USE_FLASHINFER_SAMPLER=0 is recommended. Looking at the startup logs, the hybrid linear attention layers selected Triton/FLA kernels, and the FP8 layers selected FlashInfer kernels respectively, running without issues.

Japanese responses come back normally. Calculations like "I bought 3 apples for 160 yen and 5 tangerines for 200 yen. What's the total and the price per tangerine?" and Python code generation all arrived at correct answers after going through the reasoning process. Operation itself is stable.

How Much Does MTP Change Raw Speed?

Now for the critical speed question. I generated 256 tokens forcibly with the same prompt and measured pure decode speed excluding TTFT. First, here's the raw state without MTP.

Single at 12.13 tok/s. The benchmark at the top showed 26 tok/s, so I'm only getting half of that. This is where MTP comes to mind. This DGX Spark-optimized checkpoint includes an MTP module, so I added --speculative-config '{"method":"mtp","num_speculative_tokens":3}' and re-measured.

Single jumped from 12.13 to 29.38 tok/s, a 2.4x increase. This closely matches the 26 tok/s from the benchmark at the top. What I confirmed on actual hardware is that the "fast" number came from MTP-enabled state, not raw speed.

Why is a more dense-leaning model slow without MTP? Decode speed on DGX Spark is roughly determined by "memory bandwidth 273 GB/s ÷ amount of weights read per token." Dense models read all parameters every token, so reading 27B worth hits a bandwidth ceiling. MTP works by prefetching multiple tokens in a single forward pass and betting on them, so the actual effective speed increases by however many tokens it correctly predicts.

Side-by-Side Measurements with Other Candidates

Measuring Qwen alone makes it hard to judge whether it's fast or slow. I measured other candidates that can run on a single DGX Spark using the same harness and lined them up. The comparison includes two NVIDIA Nemotron variants (Nano for speed, Super for accuracy), Ornith which is also strong in Japanese, and the versatile Gemma. Single shows decode speed when running only 1 request, and 4-parallel shows total throughput when running 4 simultaneously; for models where MTP is available, the 4-parallel values show MTP-enabled figures. Only Gemma was measured on a nightly build due to the compatibility situation described later.

Dark blue shows raw speed, light blue shows the MTP-enabled boost. Gemma 62.4 with external drafter and raw Nano 61.2 are nearly tied, pulling ahead by a clear margin.

The fastest in raw speed was Nemotron Nano. Despite being a large 30B, it's MoE with only 3B active parameters per token, achieving 61 tok/s raw. What differentiates speed is not total parameter count but the amount of weights actually read per token. Looking at raw speed, the MoE models Nano and Gemma with small active parameters come out on top, followed by Super with active 12B, then dense Ornith and Qwen. The reason Ornith doesn't stretch far despite being 9B is that it remains in bf16 without quantization, meaning bytes per parameter are 4x higher than NVFP4.

Incidentally, Ornith also has a MoE 35B-FP8 variant (Active 3B), which in previous article measurements ran at 37.5 tok/s—3x faster than the 9B. However, the Japanese free-form evaluation rated the 9B higher, so I'm listing the 9B as the quality candidate in this article. The principle that speed is determined by "active parameters × quantization" applies equally to variant selection within the same family.

MTP is only an add-on for models that support it. Among the checkpoints, only Qwen and Super include bundled MTP modules: Qwen improved from 12.1 to 29.4 (2.4x), and Super from 15.8 to 24.0 (1.5x). Super with 120B reaching 24 tok/s with MTP is sufficiently practical as a quality-focused option.

And then there's Gemma. While the checkpoint doesn't include MTP, Google officially distributes a separate drafter (a lightweight model serving as a draft generator) in a separate repository, which enables MTP when specified. The additional download is only 832MB. This boosted raw speed from 30.19 to 62.36 (2.1x), nearly tying with Nano for the top spot. Two speculative tokens is sufficient; increasing to 3 doesn't improve speed (deeper lookahead has lower acceptance rates that cancel out). Nano and Ornith have no MTP mechanism, so their raw numbers represent their actual capability.

On the quality side, all models returned correct answers for code generation and numerical calculations. My impression is that Ornith stands out in Japanese free-form writing quality, which was also confirmed by ELYZA-tasks scores in the previous article. However, Ornith always outputs its reasoning process and this cannot be turned off, which tends to increase latency, making it more of a quality candidate than a speed candidate.

Gemma 4 Shining in Instruction Following and External Drafter

Gemma occupied an interesting position. Raw single at 30 tok/s, with TTFT also at its fastest at 0.05 seconds. What particularly stood out was instruction following: following a constraint like "explain in 50 characters or less" was achieved only by Gemma among the 5 models. Others output lengthy reasoning processes, far exceeding the character limit. For use cases requiring short, precise responses, Gemma stood a clear head above the rest.

Enabling MTP just requires adding one line with the drafter repository to the startup command:

--speculative-config '{"method":"mtp","num_speculative_tokens":2,"model":"google/gemma-4-26B-A4B-it-assistant"}'

This drafter is the same one I used in a previous verification article on Gemma 4 MTP, which at the time (vLLM in May) achieved 47.9 tok/s. The same combination now reaches 62.36 tok/s on the current nightly, with acceptance rate also rising from 55% to 68%.

https://dev.classmethod.jp/articles/dgx-spark-gemma4-mtp-multi-token-prediction-bench/

Enabling MTP doesn't remove image input capability. Only the drafter side switches to text-only; the main model continues to operate as VL. I actually tested by sending an image with shapes drawn on it, and it correctly returned "a red circle and a blue square" with speculative decoding still active.

One prerequisite: the NVFP4 version of Gemma 4 requires a relatively recent vLLM. With stable v0.24.0, the modelopt quantization path doesn't support Gemma's weight sharing (tie_word_embeddings) and fails to start; the nightly at time of writing works. That's why only the Gemma measurements in this article were done on nightly.

Incidentally, Gemma also has a dense 31B-IT version, which also started with the same vLLM. However, since it reads the full dense weights every token, it's slow at single 6.9 tok/s, and for regular use on DGX Spark, the MoE 26B-A4B is the more straightforward choice. The principle of choosing MoE with fewer active parameters within the same family applies here as well.

Build Matters More Than Config Tuning—But Not Always in a Good Way

There was another discovery worth remembering on the speed side. NVFP4 on DGX Spark behaves quite differently depending on the vLLM build. And it's not a simple story of "newer is faster."

First, I tried squeezing out more performance from Qwen's MTP through startup flags. I increased max_num_batched_tokens and added -O3, --async-scheduling, and --enable-chunked-prefill. Results were essentially unchanged. These flags improve parallelism and prefill efficiency, but don't affect single-stream decode.

Next, builds. Gemma's external drafter received the benefit of newer builds straightforwardly: the combination that was 47.9 tok/s on the May build reached 62.36 tok/s on the current nightly, with acceptance rate rising from 55% to 68%.

On the other hand, there was a pitfall with bundled MTP. Running Qwen and Super's bundled MTP on the latest nightly at time of writing produces higher apparent tok/s (Qwen 37.7 / Super 46.0), but the output is broken text that repeats the same tokens. Looking at speculative decoding acceptance rates, they're stuck at either 0% or 100%, clearly different from the healthy distribution (55–92%, stepping down from higher rates for earlier predicted tokens to lower for later ones). Broken repeated output is perfectly predictable by the drafter, so it appears "fast" at 100% acceptance rate—that's the trick. Throughput benchmarks force-generate output without examining the content, so chasing numbers alone would have fooled me.

With stable v0.24.0, the bundled MTP runs correctly under the same settings. Output is coherent, acceptance rates are step-shaped, quality checks all pass, and speed consistently comes in around 2x raw as shown in the table above. So this article uses v0.24.0 as the baseline (with Gemma only on nightly since v0.24.0 doesn't support it). Two lessons: for bundled MTP, stable is the safe choice at this time; when using nightly, always verify that output is coherent. And tok/s numbers are dangerous to look at in isolation—you need to check acceptance rate distribution and output content together.

Footprint Matters When Sharing One Machine with RAG and ComfyUI

So far I've talked about speed, but for regular use there's another axis: available memory.

My setup doesn't dedicate this DGX Spark to LLMs—I want it to coexist with a vector DB and embedding model for RAG, and occasional image generation with ComfyUI. The rough idea is to divide 128GB of unified memory roughly as follows:

What matters here is the --gpu-memory-utilization setting used in my benchmarks. I had it set to 0.85 for benchmarking, but at this setting vLLM holds onto over 100GB combining weights and KV cache. In fact, when starting Nano at 0.85, the logs showed KV cache alone reserving 81.6GiB. This leaves no room for RAG or ComfyUI. For use as a shared machine, it's more practical to bring this down to 0.3–0.4 and leave the rest for other uses.

Fortunately, reducing this value doesn't affect single-stream speed. Only the number of concurrent sessions and the maximum context length that can fit are reduced. In practice, I measured Gemma 26B-A4B at 0.70 and 31B-IT at 0.50, and single speeds for both were as expected.

From this perspective, here's a comparison of "how low can gpu-util go for each model when using it as a shared machine." The amount vLLM reserves is roughly this coefficient times the approximately 120GB visible to the GPU. As long as the weights fit and there's enough KV for requests from a single laptop, it's sufficient, so smaller models can push this quite low. Note that this table is a rough estimate back-calculated from measured weight sizes—I didn't actually restart all models at these values and verify.

Blue shows vLLM's reserved portion, green shows what's available for other uses. The four smaller models leave nearly 80GB free, but Super alone reverses this with only 36GB remaining.

The four smaller models fit within gpu-util 0.3–0.35, leaving nearly 80GB free. A practical setup would be keeping a vector DB and embedding model resident for RAG, and spinning up ComfyUI with Flux only when needed. Super alone uses 70GB just for weights, so the KV budget is tight without pushing gpu-util to 0.7, leaving under 40GB free. Running image generation simultaneously becomes just barely feasible with SDXL, and difficult with Flux.

Of course, running LLM and ComfyUI simultaneously means they compete for GPU compute, slowing both down. Embedding calls add their own overhead too. But given the premise of an individual occasionally accessing it from a laptop as a secondary machine, simultaneous usage isn't that frequent. The practical split would be keeping a small model resident at conservative gpu-util, always responsive for LLM requests, while using the free space for image generation or index updates. The idea is using this single DGX Spark as a jack-of-all-trades standing in for a laptop's internal GPU.

The Optimal Solution Varies by Use Case

Let me organize the results by use case. The "optimal model" for a single DGX Spark doesn't narrow to one choice—it splits depending on what you want to do.

For keeping an agent always on or for snappy back-and-forth conversation, Nemotron Nano 30B-A3B. At single 61 tok/s plus scaling to 254 tok/s at 8-parallel, it handles multiple simultaneous requests well. Being the only one of the five models to achieve this speed raw, without any drafter or extra flags, is a distinct advantage.

For short, precise responses, Gemma 4 26B-A4B. For scenarios where you want it to follow instructions without preamble—tool calls, templated responses—it performed most reliably among the 5 models. Adding the external drafter brings speed to 62 tok/s matching Nano, achieving both instruction following and speed. Though as mentioned, this requires a relatively recent vLLM.

For packing dense intelligence into one machine, Qwen3.6-27B-NVFP4 + MTP. The dense-model intelligence reflected in AIME 2025's 92.7 score can be brought to practical 29 tok/s speed with MTP. This seems well-suited for single-user heavy inference workloads.

For 120B-level accuracy, Super. With MTP active it reaches 24 tok/s, practically sufficient as an accuracy-focused option. Though as seen earlier, it nearly monopolizes one machine's memory. For prioritizing Japanese text quality above all, Ornith. These two are choices when the goal is clearly defined.

Beyond speed and quality, there are other selection axes. For sending images, among the current candidates Qwen and Gemma support image input with their VL architecture. Fortunately, enabling MTP doesn't remove image input capability. I confirmed both models respond correctly to image requests with speculative decoding active under MTP-enabled settings (speed measurements were done with text). Nano and Super are text-only, so multimodal requirements change the options. Use case matters too. For chat UI interaction like OpenWebUI, single speed and text quality are key; for tool calling from an agent (in my case, Hermes Agent), instruction following and parallel throughput become dominant.

And honestly, this division of use varies significantly by use case and person. In my case, I rely on Claude Code and Codex CLI for serious coding and don't intend local LLMs to carry everything. On the agent side, I also have LLM routing where lighter tasks go local and mid-size and above go to cloud-based multiple models via OpenRouter. The local machine's role is a receptacle for light templated processing, preprocessing, and handling data I don't want to send externally. I'd be glad if the numbers in this article serve as material for deciding what to put in that "local slot."

That said, my leading candidate for personal use is Qwen. When you have one partner machine, you inevitably end up wanting it to do all sorts of things, and the balanced combination of MTP delivering 29 tok/s, image input support, and the intelligence of AIME 92.7 proves compelling. I plan to start operating with a configuration that adds Nano as the speed-focused always-on slot and Gemma for instruction-critical templated processing alongside it.

Summary

Starting from Qwen3.6-27B-NVFP4, I measured local LLMs for regular use on a single DGX Spark side by side. What I learned is that generation speed is mostly determined by the smallness of active parameters, and that Qwen's reputation for being "fast" referred to its state with bundled MTP enabled. The difference between 12 tok/s raw and 29 tok/s with MTP is something you'll misread benchmarks without knowing. MTP isn't exclusive to Qwen / Super with bundled modules—Gemma also jumps from 30 → 62 tok/s by adding the officially distributed external drafter. For just 832MB of additional download, this boost is well worth trying.

Qwen3.6-27B-NVFP4 itself ran smoothly on the standard stable image, and given its balance of intelligence, speed, and image input support, I plan to make it the core of my partner configuration. On the other hand, in terms of speed, raw Nano and Gemma with drafter are nearly tied at the top, making these two strong choices for always-on use. Using different models for different purposes seems the right approach. NVFP4 behavior changes with vLLM builds in both speed and behavior. For now, it's recommended to split standards by model—stable for bundled MTP, nightly for Gemma—and whenever you change builds, verify not just the tok/s numbers but the content of the output as well.

Reference Links

• nvidia/Qwen3.6-27B-NVFP4
• nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
• nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
• deepreinforce-ai/Ornith-1.0-9B
• nvidia/Gemma-4-26B-A4B-NVFP4
• google/gemma-4-26B-A4B-it-assistant — Official MTP drafter for Gemma 4 26B-A4B
• nvidia/Gemma-4-31B-IT-NVFP4
• MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies — Parallel benchmark of Gemma 4 26B-A4B on DGX Spark
• vLLM on the DGX Spark
• NVIDIA TensorRT Model Optimizer