I compared three types of speculative decoding (DSpark / DFlash / MTP) of NVIDIA Nemotron 3.5 Lightning on DGX Spark

I am Okuri, who deeply loves whiskey, cigars, and pipes. I recently joined the Manufacturing Business Technology Department.

On August 11, 2026, NVIDIA announced a new open model, Nemotron 3.5 Lightning. This model comes bundled with three implementations of speculative decoding, and the official documentation even provides hardware-specific recommendations, stating "DSpark is recommended for DGX Spark." I ran a hands-on comparison of all three methods on my DGX Spark.

• NVIDIA Nemotron 3.5 Lightning Delivers Fast, Accurate Specialized Task Execution for Long-Running Agents
• NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI
• nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · Hugging Face
• Nemotron/usage-cookbook/Nemotron-3.5-Lightning/

What is NVIDIA Nemotron 3.5 Lightning

Nemotron 3.5 Lightning is an open model designed to serve as the "execution layer" for always-on AI agents.

Long-running agents spend the majority of their execution time on high-frequency, routine operations such as tool calls, result validation, and delegation to sub-agents. Using a frontier reasoning model for each of these steps drives up both cost and latency. The intended division of labor is therefore to have frontier models like Nemotron 3 Ultra handle planning and complex decisions, while a smaller model like Lightning handles the high-frequency execution work.

The model specifications are as follows.

One thing worth noting is that the "Single-GPU Deployment" field of the model card reads "1× DGX Spark (GB10) or 1× H100" — the datacenter-grade H100 and the desktop mini PC DGX Spark are listed side by side. The Quick Start section also puts DGX Spark first, with H100 and GB200 following afterward. This is clearly a model where DGX Spark is treated as the primary target.

My colleague Morishige has written a blog post covering the details of Nemotron 3.5 Lightning 30B-A3B-NVFP4, so please refer to that for more information.

https://dev.classmethod.jp/articles/dgx-spark-nemotron-3-5-lightning-first-touch/

Why speculative decoding is effective on DGX Spark

Before diving into the actual measurements, let me explain why speculative decoding is effective here. This is the key point of this comparison.

The DGX Spark's GB10 Grace Blackwell Superchip uses a unified memory architecture where 128GB of LPDDR5x is shared between the CPU and GPU. While 128GB is a large amount of memory, being LPDDR5x means memory bandwidth is capped at 273 GB/s. Compared to datacenter-grade GPUs equipped with HBM, this becomes a bottleneck.

LLM decoding is inherently memory-bandwidth-bound because the model weights must be read from memory for every token generated. If you were to run a dense 30B model quantized to NVFP4, the weights would be roughly 15GB. Reading 15GB per token at 273 GB/s gives a theoretical maximum of around 18 tokens/second — not particularly fast for a lightweight model.

Nemotron 3.5 Lightning is designed to work around this ceiling through several mechanisms.

The third mechanism — speculative decoding — is particularly important. Speculative decoding works by having a lightweight draft model predict several tokens ahead, which the main model then verifies in a single batch. The benefit is greatest in low-concurrency environments with limited bandwidth.

DGX Spark fits this profile exactly: narrow bandwidth and low concurrency. DSpark is a method published by a team from Peking University and DeepSeek, and its name happens to look a lot like "DGX Spark." The official recommendation of DSpark for DGX Spark is therefore not just a name coincidence.

The three types of speculative decoding

Nemotron 3.5 Lightning comes bundled with three methods: MTP, DFlash, and DSpark. Here is an overview of all three.

The following Lightning-related checkpoints are published on Hugging Face.

MTP (Multi-Token Prediction)

MTP is built directly into the main model during pre-training. In Nemotron 3.5 Lightning, a continued pre-training phase is used to train the MTP layers after standard pre-training, followed by a dedicated reinforcement phase to further improve MTP accuracy. The same technique is used in Nemotron 3 Super and Nemotron 3 Ultra.

The biggest advantage is that no separate draft model is needed. The official blog describes it as optimal for medium to high concurrency, and notes that the optimal draft length decreases as concurrency increases.

DFlash

DFlash is a speculative decoding method developed at UC San Diego's Z-lab. NVIDIA's official blog published an entry showing up to 15x inference performance improvement on NVIDIA Blackwell using DFlash speculative decoding. Since DGX Spark is also a Blackwell family device, solid performance gains can reasonably be expected.

The official blog describes it as potentially delivering the best performance for certain workloads compared to other models.

DSpark

DSpark is a method published on arXiv in July 2026. The paper is titled "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation."

DSpark is an extension of DFlash. As a real-world production result, when integrated into DeepSeek-V4's serving environment, it reportedly improved per-user generation speed by 60–85% compared to the production baseline of MTP-1 at the same throughput level.

In the Nemotron 3.5 Lightning model card, DSpark is positioned as recommended for DGX Spark and low-concurrency datacenter workloads, with the note that DSpark is currently recommended for all cases.

Let's try it

Prerequisites

Setting up Docker

vLLM runs inside a container. For the DGX Spark setup, I followed the official vLLM Playbook, and for launching the container, I followed the Nemotron 3.5 Lightning vLLM cookbook.

The container image used is vllm/vllm-openai:v0.27.1, as specified in the Nemotron 3.5 Lightning model card.

$ docker pull vllm/vllm-openai:v0.27.1

Next, start the container. Following the vLLM cookbook, I use --entrypoint /bin/bash to enter the shell and then run vllm serve from inside the container.

$ docker run --rm -it --gpus all --ipc=host --network=host \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --entrypoint /bin/bash \
    vllm/vllm-openai:v0.27.1

Unless otherwise noted, all subsequent commands are run inside this container. First, set the models to use as environment variables.

# Inside container
$ export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
$ export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
$ export DFLASH_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash

Basing on the model card's DGX Spark recipe

The vllm serve options are based on the "1x DGX Spark (GB10)" recipe from the model card.

Speculative Decoding Strategies

The original looks like this.

# Model card "1x DGX Spark (GB10)" / Specdec method - DSpark
$ vllm serve --model $MODEL_CKPT \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 1048576 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --speculative_config.model $DSPARK_CKPT \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

I swap out the --speculative_config.* portion of this recipe to create four configurations and measure token throughput.

--speculative_config.method does not appear in the original model card text — the draft model specification alone is sufficient to make it work. However, since MTP has no draft model, omitting method would make it indistinguishable. To keep the syntax consistent across all four configurations, I chose to explicitly specify method for DFlash and DSpark as well, following the cookbook.

Changes made

I changed --max-model-len from the baseline. While the model card specifies 1M tokens (1048576), I lowered it to 65536 across all configurations.

This provides more KV cache headroom during the concurrency-8 measurements and ensures consistent comparison conditions across all four configurations. The model card itself notes the following for the H100 and GB200 recipes, indicating that lowering the value to match the workload is expected:

If you're memory-constrained — or want more KV-cache headroom at high concurrency — lower --max-model-len to match your workload.

Baseline (no speculative decoding)

First, start the plain configuration to serve as the comparison baseline.

# Inside container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Loading the 30B model takes a few minutes. Following the cookbook, wait until /v1/models responds. Run this in a separate terminal on the host side.

# Host side
$ until curl -sf http://localhost:8000/v1/models > /dev/null 2>&1; do
    echo "Waiting for server..."; sleep 5
  done
  echo "Server is ready"

Waiting for server...
Waiting for server...
Waiting for server...
Waiting for server...
Waiting for server...
Server is ready

The first run will take longer as the model needs to be downloaded.

MTP

Since MTP is built into the main model, no separate draft model is needed. It is activated by adding --speculative_config.method mtp, --speculative_config.num_speculative_tokens 3, and --speculative_config.moe_backend triton. Initially I omitted --speculative_config.moe_backend, which caused an error because --moe-backend marlin is not supported by the drafter. --speculative_config.moe_backend must be one of triton, batched_triton, flashinfer_trtllm, flashinfer_cutlass, or aiter.

# Inside container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.method mtp \
    --speculative_config.num_speculative_tokens 3 \
    --speculative_config.moe_backend triton \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

DFlash

DFlash requires specifying a separate draft model.

# Inside container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.method dflash \
    --speculative_config.model ${DFLASH_CKPT} \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

DSpark

DSpark similarly requires specifying a draft model.

# Inside container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.method dspark \
    --speculative_config.model ${DSPARK_CKPT} \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Note that the DGX Spark recipe in the model card does not include --speculative_config.method, and is written to work with only the draft model specified. Since the cookbook side explicitly specifies method, I aligned with the cookbook and made it explicit here as well.

After all measurements are complete, stop vLLM with Ctrl-C and then exit the container with exit. Since the container was started with --rm, it will be automatically deleted.

Running the benchmark

The client follows the cookbook's approach. The cookbook uses openai==2.38.0, with base_url set to http://127.0.0.1:8000/v1 and api_key set to null. Sampling settings are also fixed to the officially recommended Temperature 1.0 / Top_P 0.95.

$ pip install openai==2.38.0

I prepared a benchmarking script as bench.py. It sends code generation task prompts at a specified concurrency level and aggregates throughput.

import argparse
import time
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

PROMPTS = [
    "Write a Python function that parses an nginx access log file and returns the top 10 IP addresses by request count. Include type hints and docstrings.",
    "Implement a thread-safe LRU cache in Python with get and put in O(1). Include type hints and docstrings.",
    "Write a Python script that walks a directory tree and reports the 20 largest files, with a --min-size option. Include type hints and docstrings.",
    "Implement binary search over a rotated sorted array in Python, handling duplicates. Include type hints and docstrings.",
]

def parse_args():
    p = argparse.ArgumentParser()
    p.add_argument("--base-url", default="http://127.0.0.1:8000/v1")
    p.add_argument("--model", default="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4")
    p.add_argument("--concurrency", type=int, default=1)
    p.add_argument("--requests", type=int, default=8)
    p.add_argument("--max-tokens", type=int, default=2048)
    p.add_argument("--warmup", type=int, default=2)
    return p.parse_args()

def one_request(client, args, index):
    prompt = PROMPTS[index % len(PROMPTS)]
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=args.model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=args.max_tokens,
        temperature=1.0,
        top_p=0.95,
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    )
    elapsed = time.perf_counter() - start
    return response.usage.completion_tokens, elapsed

def main():
    args = parse_args()
    client = OpenAI(base_url=args.base_url, api_key="null")

    for i in range(args.warmup):
        one_request(client, args, i)

    wall_start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=args.concurrency) as executor:
        results = list(executor.map(lambda i: one_request(client, args, i), range(args.requests)))
    wall = time.perf_counter() - wall_start

    total_tokens = sum(tokens for tokens, _ in results)
    per_request_tps = [tokens / elapsed for tokens, elapsed in results]

    print(f"concurrency     : {args.concurrency}")
    print(f"requests        : {args.requests}")
    print(f"wall clock      : {wall:.2f} s")
    print(f"total tokens    : {total_tokens}")
    print(f"aggregate TPS   : {total_tokens / wall:.2f} tok/s")
    print(f"per-request TPS : {sum(per_request_tps) / len(per_request_tps):.2f} tok/s (avg)")
    print(f"tokens/request  : {total_tokens / len(results):.1f} (avg)")

if __name__ == "__main__":
    main()

ThreadPoolExecutor(max_workers=args.concurrency) limits the number of simultaneous requests, and the wall-clock time to complete all --requests is measured. Concurrency 1 and concurrency 8 are run as follows.

$ python3 bench.py --concurrency 1 --requests 10
concurrency     : 1
requests        : 10
wall clock      : 145.77 s
total tokens    : 11853
aggregate TPS   : 81.31 tok/s
per-request TPS : 81.27 tok/s (avg)
tokens/request  : 1185.3 (avg)

$ python3 bench.py --concurrency 8 --requests 80
concurrency     : 8
requests        : 80
wall clock      : 406.28 s
total tokens    : 98185
aggregate TPS   : 241.67 tok/s
per-request TPS : 30.97 tok/s (avg)
tokens/request  : 1227.3 (avg)

At concurrency 1, I use per-request TPS as the perceived speed for a single user; at concurrency 8, I use aggregate TPS as the overall server throughput.

Notes on measurement conditions

I made three adjustments when writing the script.

• enable_thinking is set to False.
  • Nemotron 3.5 Lightning has reasoning (thinking) enabled by default, but for the agent execution layer use case assumed here — high-frequency steps like tool calls and result formatting — disabling thinking results in higher speed.
  • It also stabilizes the output token count for a fair comparison of speculative decoding methods.
• Four different prompts are cycled through.
  • Sending identical prompts would cause prefill cache hits, so multiple different prompts are used to reduce the likelihood of caching.
• Two warmup requests are included.
  • The first few requests after startup are slower due to CUDA graph captures and similar initialization, so the --warmup requests are excluded from measurement.

Measurement results

| Method | Concurrency 1
per-request TPS | Concurrency 8
aggregate TPS | vs. Baseline
(concurrency 1) | vs. Baseline
(concurrency 8) |
|---|---|---|---|---|　
| No speculative decoding | 81.27 | 241.67 | 100.00% | 100.00% |
| MTP | 111.36 | 302.27 | 137.02% | 125.08% |
| DFlash | 95.48 | 268.61 | 117.48% | 111.15% |
| DSpark | 124.24 | 354.56 | 152.87% | 146.71% |

Speculative decoding clearly has a positive effect, with DSpark showing the highest gains at both concurrency 1 and concurrency 8. While the documentation suggests MTP becomes more effective at higher concurrency, the ranking did not change at concurrency 8 on DGX Spark. In these measurements, DFlash came in last among the speculative decoding methods, though this may vary depending on the workload.

Closing thoughts

Having three types of speculative decoding bundled with the same model, along with hardware-specific recommendations for each, is a configuration I haven't seen before. Rather than simply being a fast model, the fact that it specifies which method to use on which hardware and why reflects the character of Nemotron 3.5 Lightning as a model.

What I personally found interesting in this evaluation is that the 273 GB/s memory bandwidth constraint of the DGX Spark seems to connect directly to the design philosophy of Lightning. Because it targets always-on AI agents, local deployment is part of the picture, and the design makes it possible for a 30B-class model to run at practical speeds on bandwidth-limited hardware like the DGX Spark.

Getting a DGX Spark as an individual still has a high barrier, but having an always-on agent running on local hardware is becoming an increasingly realistic scenario. The requirement to run agents without sending data outside — especially prevalent among manufacturing customers — remains strong, and I expect progress in this direction to continue.

If you already have a DGX Spark, I recommend starting with the officially recommended DSpark configuration. The playbook hasn't fully caught up yet, so there is some manual setup involved, but the model card recipe works as-is, so using that is the safest approach.