I also compared two quantized versions of Gemma 4 NVFP4 on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

I just recently wrote about a showdown between quantized versions of Qwen3.6-35B-A3B, and now Unsloth has released Dynamic NVFP4 quants for the entire Gemma 4 series (from E2B to 31B), which is generating a lot of buzz. In the Qwen case, the verdict was that NVIDIA won every benchmark against Unsloth's claim of "2.5x faster." So will Gemma 4 follow the same pattern? I suspect many of you are curious.

https://unsloth.ai/docs/basics/nvfp4

I had another personal reason for wanting to run this comparison. When I benchmarked Gemma 4 26B-A4B in my early-July article, my environment yielded 30 tok/s, while an overseas verification blog reported 52 tok/s — and this 1.7x discrepancy remained unexplained as unfinished homework. By measuring both quantized versions under the same conditions, I figured I could finally settle this mystery.

To give the conclusion upfront: this time, Unsloth won across the board. For 26B-A4B, NVIDIA's version achieved 31.6 tok/s while Unsloth's hit 49.1 tok/s — 55% faster — and both versions passed all quality checks. The "30 vs 52" discrepancy turned out to be explained almost entirely by differences in quantization format, not backend or build. Furthermore, speculative decoding with an external drafter, which had been broken across all builds since early July, was revived using a workaround found in an issue thread. The Unsloth 26B with drafter now reaches 68.8 tok/s, and the 31B — previously treated as a paperweight — has climbed to 20.8 tok/s. I also confirmed that the constraint "Gemma 4 NVFP4 requires nightly builds" is no longer an issue with the new stable vLLM v0.25.1.

The two prerequisite articles are below (published 2026-07-06 / 2026-07-13). The measurement conditions are aligned with both.

https://dev.classmethod.jp/articles/dgx-spark-single-node-best-llm-qwen3-6-27b-nvfp4/

https://dev.classmethod.jp/articles/dgx-spark-qwen3-6-35b-a3b-nvfp4-new-champion/

This article presents results from measuring both NVIDIA and Unsloth NVFP4 versions of Gemma 4's 26B-A4B and 31B on a standalone DGX Spark, covering speed, quality, image input, and the revival of speculative decoding. I hope it's useful for people running Gemma 4 on DGX Spark and those unsure which quantized version to choose.

Gemma 4 NVFP4 Also Comes in Two Flavors

Just like with Qwen, there are two sources for Gemma 4 NVFP4, and the quantization methods themselves differ.

Interestingly, while the Unsloth version had a larger checkpoint for Qwen 35B-A3B, it's actually smaller than the NVIDIA version for Gemma 4. Looking at config.json after downloading, the quant_method is compressed-tensors mixed-precision — the same structure as with Qwen.

Unsloth's published figures claim 1.41x for 26B-A4B and 1.45x for 31B. However, these numbers represent BF16-relative throughput with 128 parallel sessions on a single B200, so there's no guarantee they translate directly to single-session performance on DGX Spark. With Qwen, they didn't. I'll measure the raw performance using the same harness this time.

vLLM v0.25.1

Since vLLM automatically recognizes compressed-tensors for the Unsloth version, the only change needed from the NVIDIA version is removing --quantization modelopt.

vllm serve unsloth/gemma-4-26B-A4B-it-NVFP4 \
  --host 0.0.0.0 --port 8000 \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.85 \
  --reasoning-parser gemma4 \
  --enable-prefix-caching --trust-remote-code

Backend Choice Made No Difference

Now let me address the "30 vs 52" issue I couldn't resolve earlier. My first suspect was differences in the MoE backend. The benchmark reporting 52 tok/s specified --moe-backend marlin, while my 30 tok/s log was using flashinfer_cutlass. I tested four different backends on the NVIDIA 26B-A4B version and measured single-session decode speed.

The difference among the top three is 1.4% — within margin of error. So the backend hypothesis was wrong. Looking at the startup logs, auto internally selects FLASHINFER_CUTLASS, and it seems all choices end up on the same playing field.

Unsloth's documentation states that for DGX Spark you must use --moe-backend flashinfer_b12x or it'll be 2x slower — but specifying it prevents startup.

ValueError: NvFp4 MoE backend 'FLASHINFER_B12X' does not support the deployment
configuration since kernel does not support MoEActivation.GELU_TANH activation.

This is because Gemma 4's MoE uses GELU-tanh as its activation function, which the b12x kernel doesn't support. With Qwen 35B, the effect was "even if it works, only 1.5% difference," but with Gemma 4 it can't even get on the playing field. I also tested the recommended environment variable CUTE_DSL_ARCH=sm_121a with before/after comparison on the winning configuration, but got 47.98 tok/s — within margin of error. At least when using the official vLLM image, it seems neither of Unsloth's DGX Spark-recommended flags need attention.

The 26B-A4B Showdown: Unsloth Wins Completely

If the backend isn't the culprit, the remaining suspect is the quantization format itself. I measured both versions using the same vLLM v0.25.1, the same auto backend, and the same harness. The measurement methodology is identical to the previous two articles: force-generate 256 tokens, measure decode speed excluding TTFT, take the median of 3 runs for single, and measure total throughput for parallel requests.

Blue is the NVIDIA version (modelopt), green is the Unsloth version (compressed-tensors). The outcome is reversed compared to the Qwen 35B-A3B showdown.

Unsloth wins across the board: 55% ahead for single, 27% ahead even at 8 parallel. Since NVIDIA won by 12–17% across all configurations for Qwen 35B-A3B, the advantage between quantized versions has completely reversed depending on the model.

Now for the "30 vs 52" answer. NVIDIA's 31.6 aligns with my early-July measurement of 30.19, and Unsloth's 49.1 is fairly close to the reported 52 tok/s. In my environment at least, swapping through four different backends kept a ceiling of 31, while simply changing the quantization format got us to 49. The source of the 1.7x discrepancy appears to be the difference between the modelopt world and the compressed-tensors world.

Why does Gemma 4 reverse the outcome? Nothing definitive can be said, but circumstantial evidence points to the activation function. Just as the b12x kernel was rejected for not supporting GELU-tanh, Gemma 4's MoE has a different computation pattern from Qwen-family models, leaving fewer optimal kernel options for the modelopt version. Meanwhile, the compressed-tensors version's approach of preserving important layers in FP8/BF16 happened to be a good fit for this model's shape. The lesson emerging from these two showdowns is that quantization format isn't about "which is better" but "which fits each model."

The 31B Showdown Also Goes to Unsloth, But on a Different Stage

I verified the same pattern for the dense 31B. Single: NVIDIA 7.05 vs Unsloth 8.97 — 27% faster; and Unsloth also wins the 8-parallel total at 71.3 vs 55.1. The outcome alone mirrors the 26B-A4B result.

However, the absolute values tell a different story. A dense model that reads 30.7B worth of weights per token is naturally stuck in the single-digit tok/s range on DGX Spark's 273 GB/s memory bandwidth, and even switching quantized versions barely gets you to double digits. This aligns with the hypothesis I established in the previous article: "decode speed is determined by the number of bytes read per token." That said, since this 31B changes character with speculative decoding up next, it's too early to give up on it.

The Broken MTP Is Revived with a Workaround

One of the exciting things about Gemma 4 is the external drafter (gemma-4-XXB-it-assistant) that doubles decode speed via speculative decoding. In my early-July verification, adding the 832MB drafter to 26B-A4B brought it from 30.19 to 62.36 tok/s — a 2.07x improvement. However, attaching the drafter this time fails to start on both v0.25.1 and the latest nightly, crashing during torch compile with:

RuntimeError: a and b must have same reduction dim, but got [s47, 3840] X [5632, 1024].

Gemma 4's drafter is designed to receive 5632-dimensional input — formed by borrowing the target's embedding and concatenating it with the hidden state. However, a fix merged in early July for a different model added a guard that "prohibits sharing when target and drafter have different embedding widths." This caused the drafter to use its own embedding, leaving the post-concatenation input at 3840 dimensions — mismatched with the projection layer's expected 5632. Looking at the drafter weights, pre_projection is [1024, 5632], which matches the error numbers exactly.

Investigating further, I found two existing issues on vLLM with the same symptom (#47794 and #48848 — the latter reported by another DGX Spark user). Root cause identification and a fix PR were in progress. And in the comments of #48848, there was a saving suggestion: setting the environment variable VLLM_USE_V2_MODEL_RUNNER=1 to switch to the new execution system avoids this crash.

Trying it, it actually worked. The drafter starts up on stable v0.25.1, and the speculative decoding counter runs healthily. The V2 runner's own impact was -3.9% in a before/after comparison on bare 26B — nearly within margin of error. With that, I reran the full MTP showdown on top of the workaround:

Three things stand out. First, Unsloth's overall lead holds even with the drafter. The 26B-A4B at 68.8 tok/s surpasses Nemotron Nano in the standalone ranking and takes second place behind the reigning champion Qwen 35B-A3B.

Second, the speculation multiplier diminishes as the base gets faster. The NVIDIA version gains 1.91x while the Unsloth version only reaches 1.40x. Even with identical drafter prediction quality, the faster the target's single step, the higher the relative overhead of verification — an inherent property of speculative decoding. "The faster the quantized version, the less benefit from MTP" is worth keeping in mind.

Third is the dramatic turnaround for the 31B. The dense 31B, previously dismissed as a paperweight at single-digit tok/s, climbs into the 20s tok/s range with the drafter. Moreover, spec=4 — ineffective for 26B — clearly works for 31B. This confirms that Google's recommended num_speculative_tokens 4–8 for the 31B drafter holds true on DGX Spark at least up to 4. A great example of "the slower the model, the more speculative decoding helps."

One caveat: V2 runner is a switch to an in-development execution system, not a proper fix. It should be treated as a stopgap until the fix PR is merged. The issue thread is seeing confirming reports from other hardware, with consistent reports that performance is within margin of error of V1. There's also a report of warmup behavior where the first parallel request after startup has TTFT delayed by a few hundred ms, so keep that in mind for latency-sensitive use cases.

Checking Quality and Image Input

Since it would be counterproductive if the speed difference came from quantization sloppiness, I ran the same four Japanese quality check questions from the previous two articles (instruction adherence within 50 characters, free-form response, code generation, numerical calculation) across all four configurations.

All configurations, including those with the drafter, passed all questions. NVIDIA 26B contained "about the capital of Japan, within 50 characters including punctuation" in 37 characters and Unsloth 26B in 45 characters, code generation was verified to run correctly, and both versions got the apple and orange calculation right. Gemma 4 is inherently good at this kind of instruction adherence since it doesn't run lengthy thinking chains, and that character didn't break when changing quantized versions or execution systems.

Image input was also verified. Sending a test image with a red circle and blue square while speculative decoding was enabled produced correct responses with spec decoding active in both versions, with the Unsloth version showing 71.7–87.7 tok/s for images compared to text's 69.0 tok/s. The behavior confirmed in early July — where the drafter is restricted to text-only while the main model's VL capability is preserved — was maintained on top of the workaround.

DGX Spark standalone ranking after the Gemma 4 showdown. The three Gemma bars (★this article): base uses v0.25.1, MTP additions include the workaround (V2 runner). Others are from the previous two articles.

Summary

The Gemma 4 NVFP4 quantized version showdown yielded the opposite result from Qwen. The Unsloth version wins for 26B-A4B at 49.1 tok/s single — 55% faster — and also wins for 31B. Both versions pass all quality checks, so if you're using Gemma 4 26B-A4B on DGX Spark, there's currently no reason not to choose the Unsloth version. The "30 vs 52" discrepancy that had been unresolved homework since early July is now settled in my mind as a difference in quantization format.

At the same time, these two consecutive battles have solidified the somewhat unsatisfying conclusion: "Don't choose NVFP4 by distributor — measure each model and choose based on results." Qwen 35B-A3B favors NVIDIA, Gemma 4 favors Unsloth. The published multipliers from both are about B200 parallel throughput, and they don't speak to the feel of single-session performance on DGX Spark. The single-machine partner slot remains held by Qwen3.6-35B-A3B at 108 tok/s, but Unsloth's Gemma 26B-A4B has risen to second place at 68.8 tok/s with drafter. A configuration that pairs it with the champion — leveraging its strengths in instruction adherence and image input — is becoming a practical reality.

Reference Links

• Run Unsloth Dynamic NVFP4 Guide | Unsloth Documentation
• unsloth/gemma-4-26B-A4B-it-NVFP4
• unsloth/gemma-4-31B-it-NVFP4
• nvidia/Gemma-4-26B-A4B-NVFP4
• nvidia/Gemma-4-31B-IT-NVFP4
• google/gemma-4-26B-A4B-it-assistant — External drafter (cannot be launched with current builds)
• vLLM Releases — v0.25.0 includes Gemma4 tied embeddings fix
• vLLM Issue #47794 — Root cause of drafter regression (embedding sharing guard)
• vLLM Issue #48848 — Same symptom reported on v0.25.1 (DGX Spark). Workaround originated from this thread
• MiaAI-Lab/Gemma-4-31B-IT-NVFP4-Recipe — Long-context recipe for running 31B with full 262K context. Recommends spec=4 for drafter, consistent with this article's measurements (note: the workaround in this article is required to start the drafter)
• Gemma 4 26B-A4B on DGX Spark: 52 tok/s with NVFP4 — ai-muninn (The "52" side of the "30 vs 52" report)
• Accelerating Gemma 4: faster inference with multi-token prediction drafters — Google