I compared three types of speculative decoding (DSpark / DFlash / MTP) of NVIDIA Nemotron 3.5 Lightning on DGX Spark

I'm Okuri, a big fan of whiskey, cigars, and pipes. I recently joined the Manufacturing Business Technology Department.

On August 11, 2026, NVIDIA announced a new open model called Nemotron 3.5 Lightning. This model comes bundled with three different implementations of speculative decoding, and the official documentation even provides hardware-specific recommendations, stating "DSpark is recommended for DGX Spark." I went ahead and compared all three methods on my own DGX Spark.

• NVIDIA Nemotron 3.5 Lightning Delivers Fast, Accurate Specialized Task Execution for Long-Running Agents
• NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI
• nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · Hugging Face
• Nemotron/usage-cookbook/Nemotron-3.5-Lightning/

What is NVIDIA Nemotron 3.5 Lightning

Nemotron 3.5 Lightning is an open model designed to serve as the "execution layer" in always-on AI agents.

Long-running agents spend the majority of their execution time on frequent, routine operations such as tool calls, result validation, and delegation to sub-agents. Using frontier inference models for each of these steps drives up both cost and latency. The intended division of labor is therefore to have frontier models like Nemotron 3 Ultra handle planning and complex reasoning, while lightweight models like Lightning handle high-frequency execution tasks.

The model specifications are as follows.

One thing worth noting is that the model card's "Single-GPU Deployment" section lists "1× DGX Spark (GB10) or 1× H100". The data-center-grade H100 and the desktop mini-PC DGX Spark are listed side by side. The Quick Start section also puts DGX Spark first, followed by H100 and GB200. This is clearly a model that treats DGX Spark as its primary target.

My colleague Morishige has written a blog post covering the details of Nemotron 3.5 Lightning 30B-A3B-NVFP4, so please refer to that for more information.

https://dev.classmethod.jp/articles/dgx-spark-nemotron-3-5-lightning-first-touch/

Why speculative decoding works well on DGX Spark

Before running the actual measurements, let me explain why it is expected to be effective. This is the key point of today's comparison.

The GB10 Grace Blackwell Superchip in the DGX Spark uses a unified memory architecture where 128 GB of LPDDR5x is shared between the CPU and GPU. While 128 GB is a generous amount of memory, LPDDR5x limits the memory bandwidth to 273 GB/s. Compared to data-center GPUs equipped with HBM, this becomes a bottleneck.

LLM decoding must read model weights from memory for every token generated, making it prone to being bottlenecked by memory bandwidth. If a dense 30B model quantized to NVFP4 were run with all parameters active on every step, the weights would be approximately 15 GB. Reading 15 GB per token at 273 GB/s yields a theoretical maximum of around 18 tokens per second — underwhelming for a lightweight model.

Nemotron 3.5 Lightning is designed to work around this ceiling through several mechanisms.

The third point — speculative decoding — is particularly important. Speculative decoding works by having a lightweight draft model predict several upcoming tokens, which the main model then verifies in a single batch. The benefit is greatest in low-concurrency environments where memory bandwidth is the limiting factor.

DGX Spark fits exactly this profile of "narrow bandwidth, low concurrency." DSpark was proposed by a team at Peking University and DeepSeek, and while the name happens to resemble DGX Spark, the official recommendation to use DSpark on DGX Spark is not merely a naming coincidence.

Three types of speculative decoding

Nemotron 3.5 Lightning ships with three methods: MTP, DFlash, and DSpark. Let me start with an overview.

The following Lightning-related checkpoints are available on Hugging Face.

MTP (Multi-Token Prediction)

MTP is embedded in the main model during pre-training. In Nemotron 3.5 Lightning, a continued pre-training phase trains the MTP layers after the standard pre-training, followed by a dedicated reinforcement phase to further improve MTP accuracy. This technique is also used in Nemotron 3 Super and Nemotron 3 Ultra.

Its biggest advantage is that no separate draft model is required. The official blog explains that it is optimal for medium-to-high concurrency, and that the ideal draft length decreases as concurrency increases.

DFlash

DFlash is a speculative decoding method developed at the Z-lab at UC San Diego. NVIDIA's official blog reports up to 15x inference performance improvements on NVIDIA Blackwell using DFlash speculative decoding. Since DGX Spark belongs to the Blackwell family, meaningful performance gains are fully expected here as well.

The official blog describes it as potentially delivering the best performance for a given workload compared to other methods.

DSpark

DSpark is a method published on arXiv in July 2026. The paper is titled "DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation."

DSpark extends DFlash. As a real-world result, when integrated into the DeepSeek-V4 serving environment, it reportedly improved per-user generation speed by 60–85% at the same throughput level compared to the production baseline of MTP-1.

In the Nemotron 3.5 Lightning model card, DSpark is positioned as the recommended method for DGX Spark and low-concurrency data-center workloads, and the card currently states that DSpark is recommended in all cases.

Let's try it

Prerequisites

Preparing Docker

vLLM is run in a container. The setup on the DGX Spark side follows the official vLLM Playbook, and the container launch procedure follows the Nemotron 3.5 Lightning vLLM cookbook.

The container image used is vllm/vllm-openai:v0.27.1, as specified in the Nemotron 3.5 Lightning model card.

$ docker pull vllm/vllm-openai:v0.27.1

Next, start the container. Following the vLLM cookbook instructions, use --entrypoint /bin/bash to enter a shell and then run vllm serve from inside the container.

$ docker run --rm -it --gpus all --ipc=host --network=host \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --entrypoint /bin/bash \
    vllm/vllm-openai:v0.27.1

Unless otherwise noted, all subsequent commands are run inside this container. First, set the model paths as environment variables.

# Inside the container
$ export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
$ export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
$ export DFLASH_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash

Starting from the model card's DGX Spark recipe

The vllm serve options are based on the "1x DGX Spark (GB10)" recipe from the model card.

Speculative Decoding Strategies

The original is as follows.

# Model card "1x DGX Spark (GB10)" / Specdec method - DSpark
$ vllm serve --model $MODEL_CKPT \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 1048576 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --speculative_config.model $DSPARK_CKPT \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

I will swap out the --speculative_config.* portion of this recipe to create four configurations and measure token throughput.

--speculative_config.method is not written in the original model card; it is designed to work with just the draft model specification. However, MTP has no draft model, so without specifying method it cannot be distinguished from the baseline. To keep the syntax consistent across all four configurations, I chose to explicitly specify method for DFlash and DSpark as well, following the cookbook.

Changes made

I changed --max-model-len from the baseline recipe. The model card specifies 1M tokens (1048576), but I lowered it to 65536 across all configurations.

This is to give the KV cache more headroom during the concurrency-8 measurements, and to keep the comparison conditions consistent across all four configurations. The model card itself notes the following for the H100 and GB200 recipes, indicating that reducing this value to match the workload is expected:

If you're memory-constrained — or want more KV-cache headroom at high concurrency — lower --max-model-len to match your workload.

Baseline (no speculative decoding)

First, launch the plain configuration as the comparison reference.

# Inside the container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Loading a 30B model takes a few minutes to start up. Following the cookbook, wait until /v1/models responds. Run this in a separate terminal on the host side.

# On the host
$ until curl -sf http://localhost:8000/v1/models > /dev/null 2>&1; do
    echo "Waiting for server..."; sleep 5
  done
  echo "Server is ready"

Waiting for server...
Waiting for server...
Waiting for server...
Waiting for server...
Waiting for server...
Server is ready

The first run takes longer because the model needs to be downloaded.

MTP

Since MTP is built into the main model, no draft model needs to be specified. Adding --speculative_config.method mtp, --speculative_config.num_speculative_tokens 3, and --speculative_config.moe_backend triton enables it. Initially I omitted --speculative_config.moe_backend, which caused an error because --moe-backend marlin is not supported in the drafter. --speculative_config.moe_backend must be one of: triton, batched_triton, flashinfer_trtllm, flashinfer_cutlass, or aiter.

# Inside the container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.method mtp \
    --speculative_config.num_speculative_tokens 3 \
    --speculative_config.moe_backend triton \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

DFlash

DFlash requires specifying a separate draft model.

# Inside the container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.method dflash \
    --speculative_config.model ${DFLASH_CKPT} \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

DSpark

DSpark likewise requires specifying a draft model.

# Inside the container
$ vllm serve --model ${MODEL_CKPT} \
    --moe-backend marlin \
    --kv-cache-dtype fp8 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --speculative_config.method dspark \
    --speculative_config.model ${DSPARK_CKPT} \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Note that the DGX Spark recipe in the model card does not include --speculative_config.method and is written to work with just the draft model specification. Since the cookbook side specifies method explicitly, I aligned with the cookbook here and specified it explicitly.

Once all measurements are complete, stop vLLM with Ctrl-C and then exit the container with exit. Since the container was launched with --rm, it will be automatically removed.

Running the benchmark

The client follows the cookbook's approach. The cookbook uses openai==2.38.0, sets base_url to http://127.0.0.1:8000/v1, and specifies api_key as null. The sampling settings are also fixed at the officially recommended Temperature 1.0 / Top_P 0.95.

$ pip install openai==2.38.0

I prepared a benchmark script as bench.py. It sends code generation task prompts at the specified concurrency level and aggregates throughput.

import argparse
import time
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

PROMPTS = [
    "Write a Python function that parses an nginx access log file and returns the top 10 IP addresses by request count. Include type hints and docstrings.",
    "Implement a thread-safe LRU cache in Python with get and put in O(1). Include type hints and docstrings.",
    "Write a Python script that walks a directory tree and reports the 20 largest files, with a --min-size option. Include type hints and docstrings.",
    "Implement binary search over a rotated sorted array in Python, handling duplicates. Include type hints and docstrings.",
]

def parse_args():
    p = argparse.ArgumentParser()
    p.add_argument("--base-url", default="http://127.0.0.1:8000/v1")
    p.add_argument("--model", default="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4")
    p.add_argument("--concurrency", type=int, default=1)
    p.add_argument("--requests", type=int, default=8)
    p.add_argument("--max-tokens", type=int, default=2048)
    p.add_argument("--warmup", type=int, default=2)
    return p.parse_args()

def one_request(client, args, index):
    prompt = PROMPTS[index % len(PROMPTS)]
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=args.model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=args.max_tokens,
        temperature=1.0,
        top_p=0.95,
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    )
    elapsed = time.perf_counter() - start
    return response.usage.completion_tokens, elapsed

def main():
    args = parse_args()
    client = OpenAI(base_url=args.base_url, api_key="null")

    for i in range(args.warmup):
        one_request(client, args, i)

    wall_start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=args.concurrency) as executor:
        results = list(executor.map(lambda i: one_request(client, args, i), range(args.requests)))
    wall = time.perf_counter() - wall_start

    total_tokens = sum(tokens for tokens, _ in results)
    per_request_tps = [tokens / elapsed for tokens, elapsed in results]

    print(f"concurrency     : {args.concurrency}")
    print(f"requests        : {args.requests}")
    print(f"wall clock      : {wall:.2f} s")
    print(f"total tokens    : {total_tokens}")
    print(f"aggregate TPS   : {total_tokens / wall:.2f} tok/s")
    print(f"per-request TPS : {sum(per_request_tps) / len(per_request_tps):.2f} tok/s (avg)")
    print(f"tokens/request  : {total_tokens / len(results):.1f} (avg)")

if __name__ == "__main__":
    main()

ThreadPoolExecutor(max_workers=args.concurrency) limits the number of simultaneous requests, and the wall clock time to complete all --requests is measured. Concurrency 1 and concurrency 8 are run as follows.

$ python3 bench.py --concurrency 1 --requests 10
concurrency     : 1
requests        : 10
wall clock      : 145.77 s
total tokens    : 11853
aggregate TPS   : 81.31 tok/s
per-request TPS : 81.27 tok/s (avg)
tokens/request  : 1185.3 (avg)

$ python3 bench.py --concurrency 8 --requests 80
concurrency     : 8
requests        : 80
wall clock      : 406.28 s
total tokens    : 98185
aggregate TPS   : 241.67 tok/s
per-request TPS : 30.97 tok/s (avg)
tokens/request  : 1227.3 (avg)

At concurrency 1, I focus on per-request TPS, which represents the perceived speed for a single user. At concurrency 8, I focus on aggregate TPS, which represents the total server throughput.

Notes on measurement conditions

I made three adjustments when writing the script.

• enable_thinking is set to False.
  • Nemotron 3.5 Lightning has thinking (reasoning) enabled by default, but for the agent execution layer use case envisioned here — high-frequency steps like tool calls and result formatting — disabling thinking yields faster results.
  • It also helps stabilize the output token count for a fair comparison of speculative decoding methods.
• Four different prompts are used in rotation.
  • Sending identical prompts repeatedly would result in prefill cache hits, so multiple distinct prompts are used to reduce the chance of caching.
• Two warmup requests are included.
  • The first few requests after startup are slower due to CUDA graph capture and similar initialization, so the --warmup requests are excluded from measurement.

Results

Speculative decoding clearly provides a benefit, with DSpark delivering the highest improvement at both concurrency 1 and concurrency 8. While the documentation suggests MTP becomes more competitive as concurrency increases, on DGX Spark at concurrency 8, the ranking did not change. DFlash came in last among the speculative decoding methods in these measurements, though this is likely workload-dependent.

Closing thoughts

Having three speculative decoding methods bundled with the same model, along with hardware-specific recommendations for each, is a configuration I don't think I've seen before. Rather than simply being a fast model, Nemotron 3.5 Lightning seems designed with a clear vision of how it should be run and in what environment — and I think that says a lot about the character of this model.

What I personally found interesting in this evaluation is that the DGX Spark's memory bandwidth constraint of 273 GB/s appears to connect directly to Lightning's design philosophy. Because the model targets always-on AI agents with local deployment in mind, it is designed to run 30B-class models at practical speeds on bandwidth-limited hardware like the DGX Spark.

Getting your hands on a DGX Spark personally is still a high bar, but the idea of running a resident agent on local hardware is becoming increasingly realistic. The requirement to run agents without sending data externally is a persistent need, especially among manufacturing customers, so I expect this direction to continue evolving.

If you have a DGX Spark, I'd recommend starting with the officially recommended DSpark configuration. The playbooks haven't fully caught up yet so there's a bit of manual work involved, but the model card recipe works as-is, so using that is the most reliable approach.