I ran the 2-bit quantized MoE "Nemotron 3.5 Bolt 74B-A9B" on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.

NVIDIA has announced three new models for Nemotron 3.5: the 120B Super VL, the Bolt 74B-A9B that fits in 24GB, and the Bolt E4B at 4GB arriving on October 23rd.

https://blogs.nvidia.com/blog/nemotron-3-5-super-vl/

The first one that caught my attention was the Bolt 74B-A9B. It fits a MoE with 74B total parameters and 9B active parameters (the weights used per token) into 24GB by trimming the expert weights down to 2 bits. This time, NVIDIA has released it with the 2-bit precision baked in from the training stage.

Since I measured the Nemotron 3.5 Lightning on its release day back in August, I'll line them up using the same tasks and the same conditions (this article is from 2026-08-12).

https://dev.classmethod.jp/articles/dgx-spark-nemotron-3-5-lightning-first-touch/

To give the conclusion upfront: with 2-bit weights, it scored 8/9 on Japanese tasks and passed all 49 tests across 5 code generation problems — matching or partially exceeding Lightning. The llama.cpp available on launch day already supported it, and following the model card's DGX Spark recipe brought it up in 22 seconds. Single-request speed with speculative decoding was 61.61 tok/s, and resident memory was 31 GiB. That's more than enough headroom for a 128GB DGX Spark, which actually makes "why did they release it at this size?" the more interesting question.

This article covers the results of running Nemotron 3.5 Bolt 74B-A9B 24GB on DGX Spark's llama.cpp the day after its release. I measured speed, Japanese, code generation, tool calling, and image handling under the same conditions as Lightning and GLM's 2-bit, and wrap up with a take on what the 24GB and 4GB sizes actually mean.

What is Nemotron 3.5 Bolt 74B-A9B?

Let me first pin down what it is using primary sources. The numbers come from the model card, config.json, and GGUF metadata.

With Lightning, attention accounted for 6 out of 52 layers, but Bolt has 88 layers with 8 attention layers. It has the same number of layers as the parent Super VL, and looking at llm_config in config.json, the model name is NemotronHPuzzleForCausalLM.

True to the name Puzzle, the MoE shape differs per layer. The number of experts selected per input ranges from 240 to 496 per layer, and intermediate widths span 12 combinations from 1,536 to 2,560. Lightning had uniform 128 experts with top-6 across all layers, so this looks visibly different from the start. According to the model card, a router given a budget of "under 24GB" chose the number and width of experts per layer, then distilled them out from the parent model. The traces of carving a child from the parent remain as the layer-by-layer irregularity.

I also opened the GGUF to confirm the 2-bit internals.

16.3 GiB of the total 20.5 GiB are 2-bit experts. NVIDIA states that after quantization, they used QAD (quantization-aware distillation) to recover accuracy. This is the difference from a GGUF quantized externally after the fact — it was retrained from the parent with the assumption of running at 2 bits.

The comparison table in the official benchmarks pits it against Lightning and Qwen3.6-35B-A3B and Gemma4 26B-A4B.

Against Lightning's BF16, Bolt at 2-bit doubles Terminal-Bench. It looks like the increase from 3B to 9B active parameters shows up directly. I'll see whether this headline result holds on my local tasks.

Getting It Running

This time too, it ran without a hitch.

The llama.cpp master includes Bolt support dated October 15th, the day of release. My DGX Spark had a llama.cpp from August, so I pulled master and built it for GB10. The only build flag needed was -DCMAKE_CUDA_ARCHITECTURES=121.

The launch command can be used verbatim from the model card's "RTX Spark 128GB" 32GB recipe.

llama-server \
  --model "$MODEL" \
  --mmproj "$MMPROJ" \
  --batch-size 4096 \
  --ubatch-size 2048 \
  --ctx-size 262144 \
  --parallel 1 \
  -ngl 999 \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 \
  --spec-draft-model "$SPEC_DRAFT" \
  --spec-draft-ngl 999 \
  --temp 1.0 \
  --top-p 0.95 \
  --flash-attn on \
  --jinja

Speculative decoding uses the DFlash method, with a draft from a separate 0.93 GiB GGUF called LiLiCorr. In llama.cpp, DFlash is specified with --spec-type draft-dflash and --spec-draft-model. The startup log shows the draft model loading and n_max=7, block_size=8, so you can confirm speculative decoding is active there.

Measured values from download to startup:

llama.cpp reads weights via mmap, so there's no compilation or pre-allocation of KV — it just starts once the GGUF is open. The 31.4 GiB resident is consistent with the model card's "32GB recipe." The reason it's higher than the design target of "under 24GB at batch 1, 128K" is that this recipe stretches the context to 262,144.

The model card also includes a vLLM recipe. It's a configuration that adds a b12x kernel for 2-bit on top of the nightly image. It also started with the DGX Spark-targeted nightly-aarch64 (pushed October 15th). Single-request without speculative decoding was 31.46 tok/s, about 20% slower than llama.cpp under the same conditions (39.47 tok/s). On top of that, llama.cpp gains another 1.5x with DFlash. The numbers in this article use llama.cpp throughout, since it starts faster and runs faster.

Measurement Conditions

In the speed section, I split measurements into two: verbatim recipe --parallel 1, and --ctx-size 65536 --parallel 8 to match other models. All tasks from Japanese onward use the latter, without speculative decoding. llama.cpp divides --ctx-size by the number of concurrent slots, so with 8 slots each slot gets a maximum of 8,192 tokens. Sampling used the recommended values of temperature 1.0 / top_p 0.95. The budget (output token limit) was 8,192 as the baseline; for tasks that hit the limit, I added a separate row using a 2-slot launch (32,768 per slot) with a budget of 16,384.

I compared against two targets: GLM-5.3-Flash (Unsloth UD-Q2_K_XL, 320B-A18B) running on the same llama.cpp, and Nemotron 3.5 Lightning 30B-A3B (NVFP4, vLLM). GLM ran on a separate DGX Spark and Lightning on vLLM, so I only compare quality judgments, not speed. GLM's thinking strength was set to the highest level (max), while Lightning and Bolt were on high — they are not aligned. I note this below each table.

Speed

First, verbatim recipe --parallel 1 to observe speculative decoding effectiveness. Median of 3 runs with forced output of 256 tokens at temperature 0.

1.56x. The acceptance rate (proportion of lookahead candidates accepted by the main model) is readable from llama-server's stats. For the fixed-text speed test it was 35% (181 out of 511 drafts, average accepted length 3.45), for a 3-sentence Japanese explanation it was 39%, and for short tool-call responses it was 56%. That means roughly 3–5 of the 7 lookahead steps were hitting, which is nearly the same level as the accepted length of 3.27 seen with Lightning on DSpark.

For concurrency, I measured --ctx-size 65536 --parallel 8 without speculative decoding, to match other models.

At 8 concurrent requests, total throughput was 102.45 tok/s, with per-request speed dropping to 13.26 tok/s — one third of the single-request speed of 39.30. GLM-5.3-Flash, also a 2-bit MoE on llama.cpp, measured 17.72 tok/s single-request and 63.97 tok/s total at 8 concurrent on another unit. Bolt with 9B active parameters running 2.2x faster than GLM with 18B active is a direct reflection of the difference in active weight volume.

Lightning measured 79.59 tok/s single-request and 331.97 tok/s total at 8 concurrent on vLLM. I won't put these in the same table since the engines differ. The gap includes both the 3B vs. 9B active difference and the question of how far llama.cpp's 2-bit MoE kernels scale at 8-way parallelism. The poor parallel scaling also aligns with the model card designing for "1 sequence" when targeting 24GB. This Bolt is fast for single-user use; for a team sharing one machine, Lightning is more suitable.

Japanese

I ran 10 Japanese tasks. Budget was 8,192, thinking was set to high. Scoring was automated, checking mechanically verifiable conditions such as character count, presence of required words, and whether JSON parses correctly.

The open-ended writing task is excluded from the total since it can't be scored automatically. The GLM column used thinking strength max with the same budget of 8,192.

The Japanese quality was not broken by 2-bit. Bolt passed the 50-character constraint and vocabulary constraint that GLM's 2-bit failed, matching Lightning's automated score of 8/9. On the vocabulary constraint task, it produced a usable 207-character internal announcement that included all 3 required terms and avoided both prohibited words.

The one failure was the task requiring summarization while preserving proper nouns — after 12,303 characters of thinking, it exhausted the 8,192-token budget and produced no body text. Lightning and GLM failed the same way, as all models burn through the budget with thinking at 8,192. So I reran Bolt alone with a separate launch at 32,768 tokens per slot with a budget of 16,384.

The result was the same. It spent the full 16,384 tokens on thinking, thought for 25,525 characters, hit the limit again, and produced no body text. All other 9 tasks returned the same judgments at 16,384. Lightning, when extended to 16,000, eventually produced body text after 20,678 characters of thinking but dropped some proper nouns. For Bolt, the honest read is that thinking simply doesn't stop for this particular task. As I'll show later, with the medium setting it answered the same task with 1,105 characters of thinking and retained 4 of 6 terms, so how this task looks depends heavily on the thinking strength setting.

Looking at thinking length, Bolt thinks noticeably shorter than Lightning. For the 50-character answer task, Lightning used 7,245 characters while Bolt used 652; for the honorific rephrasing task, 6,839 vs. 558. Lightning with 3B active wanders and thinks long; Bolt with 9B decides quickly and answers. Both scored 8/9, but the path to the answer was 10 times shorter. Even though Lightning generates twice as fast, the total time for 10 tasks — 356 seconds for Bolt vs. 307 seconds for Lightning — was nearly the same, thanks to Bolt's shorter thinking.

I also measured rows with different thinking strength settings. Bolt's thinking has 3 levels: the default high, medium (which appends "reasoning effort: efficient" to the last utterance), and off (which disables thinking entirely).

Medium is interesting: even with tokens consumed down to one quarter and time down to one quarter, the automated score stayed at 8/9. For the proper-noun summary, high exhausted the budget and produced no body text, while medium answered in 183 characters and retained 4 of 6 terms. It did fail, but it produced an answer. Turning off thinking retained all 6 terms but expanded to 257 characters, breaking the character limit, and also dropped one required term in the vocabulary constraint task. Just like Lightning, thinking appears to be used for following instructions. For personal conversational use, medium looks like the best-balanced setting.

Code Generation

Two parts: writing 5 problems from scratch from Japanese specifications, and fixing a mini repository with planted bugs. Scoring was done by running pytest locally — no self-reported results from the model.

Writing from Scratch

Full marks — all 49 tests passed. The differentiator was parse_duration (the problem of converting strings like "1h30m" to seconds). Lightning exhausted the 8,192 budget on thinking for this one problem and couldn't produce code; it only solved it after extending to 16,000. GLM's 2-bit failed the same way. Lightning thought for 26,056 characters and GLM for 27,064 before running out of budget. Bolt solved it with 8,696 characters of thinking and 2,792 tokens — the same problem solved with one-third the thinking under the same budget.

Fixing Existing Bugs

A task where a mini repository with 5 failing tests out of 12 is handed over, and the model is asked to fix it using only file read/write tools.

All three brought all 12 tests to passing. Bolt finished in 5 turns. Lightning reached the same result in 13 turns, meaning the amount of progress per turn differs. Malformed tool calls (unparseable output) were 0.

Can It Be Used as a Tool?

Checking whether it can serve as an agent component. The tool-calling format is the same XML as Lightning. llama.cpp reads this format from the template with just --jinja, converting it to OpenAI-compatible JSON for output. There's no need to specify a parser name as with vLLM.

<tool_call>
<function=get_weather>
<parameter=city>
Tokyo
</parameter>
</function>
</tool_call>

All 5 basic call tests passed. A task giving it a pseudo filesystem to find bugs also completed in 4 turns with 6 tool calls, with 0 malformed calls. Both tool calls and thinking come through on chunks in streaming mode.

The required Loop Issue Did Not Appear

The most notable issue with Lightning was that specifying tool_choice: required — "you must call a tool" — caused it to return the same call 159 times in a loop. Bolt does not exhibit this.

With Lightning, I suspected an interaction with vLLM's qwen3_coder parser. This time the same XML format is read through llama.cpp's template, and it stops at 1. Whether this is a model generation difference or an engine-side difference can't be isolated from this single observation, but when using Bolt on llama.cpp, this concern appears to be moot.

I also checked how it handles multiple tool calls. For a prompt like "Tell me the weather in Tokyo and Osaka" requiring queries to two locations, Lightning called them one at a time over 2 turns, while Bolt returned both in a single message. This matches the parent Super VL's calling style. I won't call this a fixed trait based on one observation, but it's favorable in environments where tool roundtrips are costly.

Suitability as an Agent Component

Checking fitness for harnesses like opencode or Hermes Agent via 6 probes. I note upfront that I'm measuring capabilities these harnesses depend on, not the results of running the harnesses themselves.

The Lightning column was measured at budget 4,096; consistent format improved to 10/15 when extended to 16,000. Bolt's 3 columns used 8,192.

The tool selection probe sends 6 requests with 40 tool definitions loaded and checks whether the expected tool is chosen. Bolt missed the same 2 questions across all 3 conditions. Not choosing the cron-creation tool for "please schedule this to run every morning at 9am" is the same miss as Lightning; Bolt additionally chooses a read tool for "please append to the file." Since the same 2 questions are missed regardless of thinking strength, it reads more like a selection tendency than a thinking-depth issue.

The format reproduction probe checks whether a daily report template written 5 times consistently produces the same structure each time. Lightning produced 0 characters of body text in all 5 attempts at budget 4,096, but Bolt scored 4/5 at high and all 5 attempts had consistent structure at medium. The 1 failure at high had matching heading order but wrote one section with 2 lines instead of 3. At off, responses come back in 2–3 seconds but spread into 3 different formats across 5 attempts, and the "place a summary line at the end" rule from the long system prompt was dropped — matching Lightning's pattern of degradation when thinking is disabled.

Medium, on the other hand, dropped the summary-line rule from the long system prompt. The thinking strength that balances format consistency vs. long policy compliance is different for different use cases.

Per-harness judgment was "challenging" for opencode, Hermes Agent, and OpenClaw under all 3 conditions. The sticking points are the 2 missed tool selections plus either consistent format or the long system prompt depending on the condition. Since execution-layer work (call the tool, verify the result, fix the bug) passed 5/5, the failures — as with Lightning — are on the paperwork and selection-tendency side. As noted before, these are capability probes, not results from running the harnesses themselves.

Images

Since Bolt can read images, I ran 3 probes with the mmproj bundled in the GGUF.

All 3 passed. Hallucinations (listing items not in the image) were 0. The GGUF + mmproj path handles only still images; video input mentioned in the model card was not tested. Lightning was text-only, so image reading is an advantage Bolt has over it within the Nemotron 3.5 family.

Why Release at 24GB and 4GB?

From here I'll step away from measurements and into interpreting the sizes.

The model card states the design goal in one line: "at batch size 1 with 128K context, keep resident memory under 24GB at inference time." That budget includes the quantized weights and the state held during inference (2-bit expert activations and 16-bit Mamba state). The number is clearly targeting the RTX Spark laptop released the same day and the 32GB GeForce RTX 5090. The announcement blog also says "fits in 24GB, runs alongside other apps." It's sized to live next to a game or video editor, which is why there's over 90GB left over when you run it on a 128GB DGX Spark.

Another angle is the construction method. Bolt is a child carved from Super VL 120B-A12B using a technique called Star Elastic. They distilled to recover the parent's knowledge — 72B tokens at 8K context and 48B tokens at 128K — then trimmed to 2 bits and distilled again. Lightning uses a different skeleton (52 layers, 128 experts); Bolt is the direct descendant of Super VL. Grow one parent, and you can carve children of different sizes just by changing the budget. Whether E4B 4GB uses the same technique hasn't been disclosed, but the cost of adding steps to the size ladder has clearly come down.

Arranging the ladder against local hardware:

As "a base for domain-specific fine-tuning" — which is what I first hoped for — Bolt 74B sits in a slightly different place. The BF16 weights that serve as the fine-tuning starting point are 149.8GB on Hugging Face, which doesn't fit in a 128GB DGX Spark. The 2-bit GGUF is a distribution format for a trained model, not a fine-tuning starting point. For local specialization training, Lightning's 65.8GB BF16 is more realistic, and once E4B at 4B scale arrives, it will likely be the primary candidate including on Jetson. Bolt 74B reads as a model for deploying a capable executor in 24GB, not for fine-tuning.

Looking back at my local numbers through that lens, everything fits. Single-request with speculative decoding is 61.61 tok/s. It reaches the same judgment as Lightning with shorter thinking, and even problems requiring ~10,000 characters of thinking like parse_duration are solved within an 8,192 budget. It looks tuned for one person running a coding agent solo. Conversely, at 8 concurrent it drops to 13 tok/s per request, so shared team use is better left to Lightning. Even within the same Nemotron 3.5, the design differentiates by deployment context.

Summary

Nemotron 3.5 Bolt 74B-A9B 24GB scored 8/9 on Japanese tasks, passed all 49 tests across 5 code generation problems, and scored 5/5 on tool calling — all with 2-bit weights. 2-bit was supposed to be a tradeoff you accept for size at the cost of accuracy, but that accuracy degradation didn't show up on my local tasks. The effectiveness of QAD, which bakes in 2-bit from the training stage, was something I was able to verify locally.

Running it was easy. The llama.cpp master on launch day already supported it; the model card recipe brought it up in 22 seconds with 31 GiB resident memory. Single-request with DFlash speculative decoding was 61.61 tok/s. At the same time, 8 concurrent requests topped out at 102 tok/s combined, and the design's single-sequence assumption shows up in the numbers.

24GB is not a size for DGX Spark — it's a budget for coexisting with other apps on an RTX laptop or a 32GB GeForce. The BF16 fine-tuning starting point is 149.8GB and won't fit locally, so fine-tuning candidates fall to Lightning or the E4B arriving October 23rd. Bolt 74B is, in my reading, a model for packing a capable executor into 24GB for deployment.

What I want to try next is the Bolt E4B 4GB arriving October 23rd. I have a Jetson Orin Nano Super on hand, so I plan to measure whether the 4GB truly runs on edge hardware and whether it can serve as a fine-tuning base — using the same tasks as this article.

Reference Links

• NVIDIA announcement blog (Super VL and Bolt) (2026-10-15)
• nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB — Model card. Design goals, Star Elastic, QAD explanations and official benchmarks are here
• nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF — The GGUF we ran this time. RTX Spark-oriented 32GB / 28GB recipes are also here
• nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF — Draft for DFlash speculative decoding
• nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 — The base model before quantization. Start here for fine-tuning
• llama.cpp PR #30520 — Nemotron Super 3.5 VL / Bolt 74B A9B support (merged 2026-10-15)
• Star Elastic (arXiv 2605.07182) — A method for extracting budget-specific child models from a parent model
• QAD: Quantization-Aware Distillation (arXiv 2601.20088)
• RTX Spark laptop launch blog (launched 2026-10-16, home of Bolt)
• Unsloth's Nemotron 3.5 Bolt guide — Unsloth GGUF (IQ2_XXS 22GB, 1-bit also available) and RAM estimates for 24–32GB. Context listed as 128K, which differs from the official 256K
• OpenMDW-1.1 License
• Article measuring Nemotron 3.5 Lightning on DGX Spark — The Lightning figures in the comparison table were measured here
• Article measuring GLM-5.3-Flash 2-bit GGUF on DGX Spark — Comparison with externally applied 2-bit quantization