I tried running Gemma 4 MTP on DGX Spark and actually measured the speedup in Japanese text generation

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

This is a follow-up to the Gemma 4 benchmark article from the other day. I tried running Gemma 4 MTP (Multi-Token Prediction), announced by Google on 2026-05-05, on the DGX Spark.

https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/

Simply put, MTP works by preparing a lightweight "draft predictor" that predicts ahead to the next tokens separately from the main model, which the main model then verifies all at once to speed up generation. This type of technology is called speculative decoding, and the key point with Gemma 4 is that Google officially distributes the paired model consisting of the main model (target) and the draft predictor (drafter) under the name *-it-assistant.

Google's official blog touts "up to 3x speedup," but what personally caught my attention was three questions: "Will it run on DGX Spark?", "Will it be effective for Japanese language tasks?", and "How will the memory bandwidth difference affect the results?"

Actually, at the time of the MTP release on 5/6, vLLM nightly did not support speculative decoding for Gemma 4's multimodal models, and it wouldn't even start. I put it on hold once, but PR #41745, merged the following day, officially added Gemma 4 MTP support, and when I pulled the latest nightly on the morning of 5/7, it started working right away.

In this article, I'll cover everything in one go: confirming MTP operation for all 4 models (E2B / E4B / 26B-A4B / 31B) with vLLM nightly, measuring latency for both JCQ (short text) and long-form generation, and actually verifying on real hardware the hypothesis that "acceptance rates might drop for Japanese language tasks." To give you the conclusion upfront: long-form generation became up to 2.1x faster with zero quality degradation. On the other hand, it had no effect on short text, and in some cases with the 26B MoE it actually became slower.

How Gemma 4 MTP Works

Speculative decoding itself is not a new technology — DeepSeek had already released a similar implementation in vLLM first. The distinguishing feature of Gemma 4 MTP is that Google officially distributes the drafter, which pairs with the main model (target), as a separate file under the name *-it-assistant.

Let me add two supplementary notes.

The first is about the drafter's size. The drafter shares the same tokenizer (the rules for splitting text into tokens) as the target and also borrows the main model's embedding layer directly. What the drafter holds on its own is just 4 layers and a narrowed-down prediction layer that reduces output candidates to keep things lightweight. The drafter for E2B, gemma-4-E2B-it-assistant, is 182 MB, which amounts to less than 2% additional cost compared to the target's 9.6 GB.

The second is the ingenuity applied to image-capable models. All Gemma 4 models support image input (multimodal), and vLLM's speculative decoding originally rejected multimodal models. PR #41745 works around this constraint by treating only the drafter side as a text-only model — stripping away image/audio input — while keeping the target running as multimodal. This means MTP works even for requests that include images. The following is actually output in the logs:

WARNING [llm_base_proposer.py:1375] Draft model does not support multimodal inputs, falling back to text-only mode
INFO    [llm_base_proposer.py:1487] Detected MTP model. Sharing target model embedding weights with the draft model.
INFO    [gemma4_mtp.py:536] Gemma4 MTP: centroids masking enabled (num_centroids=2048, top_k=32, active_tokens=4096/262144)

The centroids masking on the third line is a technique that narrows down the candidates when the drafter predicts the next token, from the original 260,000 tokens to approximately 4,000 tokens. By excluding tokens that are almost certainly not going to appear from the computation targets, the drafter's operations are kept lightweight. Gemma 4 MTP has thorough optimization down to these fine details. It feels like the kind of subtle tweak that quietly makes a real difference.

Test Environment

vLLM nightly is installed with the following steps. The only thing to watch out for is that without the --torch-backend=cu130 option for uv, dependency resolution gets stuck.

uv pip install --pre --upgrade vllm \
  --extra-index-url https://wheels.vllm.ai/nightly \
  --torch-backend=cu130

The test targets are all 4 Gemma 4 models and their respective drafters.

For this test, I used the NVFP4 quantized versions for 26B-A4B and 31B. Since the drafter uses the same tokenizer as the target, the additional cost is only 200 MB to under 1 GB for any model.

MTP Setup with vLLM

The actual startup command is as follows. This is an example enabling MTP with E2B + drafter.

vllm serve google/gemma-4-E2B-it \
  --host 0.0.0.0 --port 8001 \
  --served-model-name gemma4-e2b \
  --max-model-len 8192 --max-num-seqs 4 \
  --gpu-memory-utilization 0.5 \
  --enforce-eager \
  --max-num-batched-tokens 4096 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2,"model":"google/gemma-4-E2B-it-assistant"}'

The drafter is distributed with a model_type called gemma4_assistant, which is automatically rewritten to gemma4_mtp internally in vLLM. The model you specify is the drafter's repository, and the target is passed as the normal gemma-4-E2B-it.

Result 1: Speedup Ratio by Model for Long-Form Text

The benchmark was run by executing a prompt to "generate an explanation of about 200 characters" 12 times repeatedly, and evaluated by the average from the 2nd through 12th runs, excluding the first run (which tends to be slow while the model warms up). The conditions were temperature=0 (a setting to eliminate variation in output), max_tokens=256 (maximum number of tokens to generate), and a single client (no concurrent requests).

A speedup of 1.7x to 2.1x (the ratio of speedup compared to baseline) was confirmed across all models. E4B showed the greatest improvement (2.10x), which I think is because the size balance between the target and drafter happened to fall right in the sweet spot for effective bandwidth utilization. The 31B also achieved 1.91x, and while the absolute value of 12.5 tok/s seems modest, it gives a solid impression that it's working well even through the heavy compute path of NVFP4 marlin.

The acceptance rate (the proportion of tokens predicted by the drafter that the target adopted as-is) rises proportionally with model size, from 38% to 55%. It's personally interesting to me that this falls in line with the intuitive relationship: the larger and stronger the target, the more likely the drafter's predictions are to be correct.

Result 2: MTP Has No Effect on Short Text (JCQ)

Actually, I also measured under the same conditions using the Japanese commonsense reasoning benchmark JCommonsenseQA (200 questions, 3-shot format with examples provided, max_tokens=8). This is a task requiring extremely short responses, where only a single character label (A through E) needs to be returned.

For short text, all models are nearly equivalent in speed, and for 26B-A4B in particular, it slows down by about 19%. This is because the cost of running the drafter every time cannot be recovered when the number of generated tokens is small. The official MTP documentation also includes a note that "MoE (Mixture of Experts, a method that dynamically switches between specialized models) does not benefit well from speedup at batch_size=1," and that is exactly what was reproduced right before my eyes.

Flipping this around, it becomes clear that MTP is an optimization that only works for tasks that generate a large number of tokens. It produces a significant effect for use cases like chat responses, code generation, and summarization that generate hundreds to thousands of tokens, but enabling MTP for classification tasks that return only a single choice label is actually counterproductive. For production use, a design that "turns MTP on/off based on task length" seems to be necessary.

The fact that 26B-A4B flips from 0.81x for short text to 1.71x for long-form is the most visually striking result.

Result 3: Zero Quality Degradation

Since it would be counterproductive if accuracy dropped in exchange for speed, quality was verified using the correct answer rate on 200 JCQ questions.

The difference is within ±0.5pt, which amounts to a range of only 1 question difference out of 200. MTP is theoretically a mechanism that guarantees quality — even if the drafter predicts an incorrect token, the target side rejects it, so the final text output is the same as without MTP.

Result 4: Acceptance Rates Are Nearly the Same for Japanese and English

I tested the hypothesis that "the unique token splitting for Japanese (the SentencePiece method used in Gemma 4 creates many patterns, such as treating a single kanji character as its own token or combining two characters, which makes prediction harder for the drafter, and thus reduces the acceptance rate)" by running 12 trials under the same conditions with English prompts.

For E2B, the acceptance rate was +3.2pt higher for English, while for E4B it was conversely -2.6pt lower. There was no consistent direction, and the honest conclusion is that no significant difference by language was found. The tok/s values were also nearly equivalent, suggesting that MTP is not struggling with Japanese language tasks.

What is interesting is the number of output tokens: English yielded approximately 41% more tokens even with the same "200 characters / 150 words" specification. The way tokens are formed differs between Japanese and English — roughly speaking, "1 English word ≈ 1–2 tokens, 1 Japanese character ≈ 1–2 tokens" is common. Even with the same character count, the number of tokens differs, so when comparing the amount of generated text, it is fairer to align by token count rather than character count or word count.

In that case, it becomes natural to consider that the ceiling on the speedup ratio is determined by other factors.

Result 5: Memory Bandwidth Is the True Bottleneck

The logic behind why speculative decoding is faster lies in the fact that "the target verifies multiple tokens all at once in a single processing pass, making efficient use of memory bandwidth." The bottleneck in generating 1 token in LLM inference is the work of reading the model's weights (tens of GB) from memory every single time. Whether processing 1 token or N tokens, this read cost is nearly the same. Therefore, the more tokens that can be returned in a single processing pass, the faster the apparent generation speed becomes.

However, the DGX Spark's memory bandwidth is structurally low, and that is what determines the ceiling of the speedup ratio.

With only 1/12 the bandwidth of an H100, it is structurally reasonable that the speedup ceiling is pinned at around 2x. Google blog's "up to 3x" is naturally understood as the value achieved when running on an H100, and achieving 1.7x to 2.1x on the DGX Spark could be considered quite respectable.

In other words, the speedup ratio of MTP on the DGX Spark is broadly determined by the following relationship.

Final speedup ≈ min(
    1 + acceptance rate × number of speculative tokens,   # Theoretical MTP upper limit
    memory bandwidth ratio,                               # Ceiling from bandwidth
    compute-side efficiency                               # NVFP4 marlin, etc.
)

With an acceptance rate of 40–55% × 2 speculative tokens, the theoretical upper limit is 1.8–2.1x, and the BF16 versions (E2B / E4B) are operating nearly at that limit, while the NVFP4 versions (26B-A4B / 31B) are experiencing a small amount of loss on the compute side. The fact that the measured data fits neatly into this formula suggests that the behavior of speculative decoding is being reproduced faithfully on the DGX Spark as well.

Summary of Results So Far

All models started up, and speedup works effectively for long-form tasks across all models. Since MTP has no effect on short text or tasks with few generated tokens — and can even slow things down with MoE models — in actual production use, it is realistic to either switch MTP on/off depending on the type of task, or to prepare it as a separate endpoint.

Conclusion

I tested Gemma 4 MTP on the DGX Spark and was able to confirm operation and evaluate performance for all 4 models while the merge of PR #41745 was still fresh — the very next day. The key takeaways are as follows:

• Long-form generation achieves 1.7x to 2.1x speedup (max_tokens=256, after warm-up). E4B achieves the maximum of 2.10x
• No effect on short text (JCQ, max_tokens=8), and with the 26B MoE it slows down by about 19%. MTP's effectiveness varies with the number of tokens generated
• Zero quality degradation (within ±0.5pt in JCQ accuracy on 200 questions). This is expected since MTP is a mechanism that theoretically guarantees quality, but being able to confirm it with numbers is reassuring
• The difference in acceptance rates between Japanese and English is about ±3pt, which is limited. There is no structure where Japanese puts MTP at a disadvantage
• The DGX Spark's memory bandwidth of 273 GB/s (1/12 of H100) determines the ceiling of the speedup ratio. Google blog's "up to 3x" is the value achieved on H100, and achieving around 2x on DGX Spark is a structural ceiling

For those using Gemma 4 on DGX Spark, I would recommend enabling MTP for long-form tasks. Just having E4B BF16 go from 18 to 39 tok/s makes a big difference in how it feels. Separating it from short-form tasks for operation seems to be the right approach at this point.

The absolute values for NVFP4 models are modest due to going through Marlin, but even so, with 26B-A4B reaching 47.9 tok/s and 31B reaching 12.5 tok/s in long-form tasks, and combined with the DGX Spark's memory headroom (128 GB UMA), my impression is that it has reached a practical level where it can be used confidently for local long-form generation tasks. The fact that Google merged the PR within a day and by the following day it was already running on the DGX Spark is a result that makes you feel just how fast the ecosystem is moving.

Reference Links

• Google: Multi-Token Prediction for Gemma 4
• Google AI Docs: Gemma 4 MTP
• vLLM PR #41745: Add Gemma4 MTP speculative decoding support
• vLLM Recipes: Gemma 4 26B-A4B
• HuggingFace: Gemma 4 collection
• NVIDIA: Gemma-4-31B-IT-NVFP4
• JCommonsenseQA v1.1
• Verification scripts (GitHub)