I tried running the 4GB conversational model "Nemotron 3.5 Bolt E4B" on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA released Nemotron 3.5 Bolt E4B 4GB on October 23. This is the smallest model in the Bolt series, which was announced on October 15 with a preview stating it would be released on October 23.

https://research.nvidia.com/labs/nemotron/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB/

At the end of my previous article on Bolt 74B-A9B 24GB (written as of 2026-10-16), I wrote that I would "measure E4B with the same tasks once it's released." Bolt 74B was a 2-bit "distributing side" model carved out from Super VL 120B. I also wrote that the receiving end for specialized local learning would go to either Lightning or E4B. The E4B itself is based on Google's Gemma 4 E4B-it, with NVIDIA adding only reinforcement learning on top. The target use cases are voice assistants, game NPCs, and robot conversation interfaces, with a focus on calling tools at the right moment during conversation and respecting instruction priorities.

https://dev.classmethod.jp/articles/dgx-spark-nemotron-3-5-bolt-74b-a9b-first-touch/

NVIDIA has published speed figures for the RTX 5070 and Jetson Orin Nano, but there are no numbers for the DGX Spark. I measured using the usual tasks on my local DGX Spark and Jetson Orin Nano Super. The parent model Gemma 4 E4B-it and the same 4B-class Qwen3.5 4B are also lined up using the same llama.cpp and the same Q4_K_M.

To state the conclusion upfront: with llama.cpp on DGX Spark, single-request speed was 128.79 tok/s (with DFlash). The total for 8 parallel requests was 355.81 tok/s. Japanese 10 tasks scored 8/9, code generation scored 5/5, and tool calling scored 5/5, placing the 4B model in the same table as Bolt 74B. Compared to the parent Gemma 4 E4B-it, speed and Japanese were the same, with differences showing up in code accuracy and adherence to long instructions.

This article covers the characteristics of Bolt E4B 4GB, getting it running on DGX Spark, actual measurements of speed, Japanese, code, and tools, and a comparison with the parent Gemma 4 E4B. I hope this is useful for those who have a DGX Spark or Jetson and want to run a small conversational model as a resident process.

Characteristics of Nemotron 3.5 Bolt E4B 4GB

Here is a table summarizing the relevant information from the model card and technical report.

The "4GB" in the name does not refer to file size. It is an estimate that memory usage fits within approximately 4GB for the intended workload — single-turn conversations with 2.5K to 16K token inputs returning fewer than 500 tokens. In NVIDIA's measurements, this was 3.5 GB on an RTX 5070 and 4.2–4.9 GB on a Jetson Orin Nano. The GGUF file itself is 6.18 GB.

The official benchmarks cover 5 groups: conversation, multi-turn tool calling, instruction priority, long context, and grounded responses. Comparisons are made against the same 4B-class Gemma 4 E4B-it and Qwen3.5 4B, as well as the previous generation Nemotron 3 Nano 4B. It achieves the highest score on 15 out of 21 benchmarks, but there are clear areas where it falls short.

All values are BF16, measured by NVIDIA using NeMo Gym. It improves on instruction priority and tool call timing, but does not match Qwen3.5 4B on long context or Tau2. The "up to 80% reduction in output tokens" mentioned on the research page is also from comparisons within this same table.

What I found interesting in the technical report was the reason for not including SFT. Multi-turn tool calling did improve with SFT, but Gemma 4's conversational naturalness and conciseness regressed, so they settled on reinforcement learning alone. Japanese sits outside this English-only reinforcement learning, and the technical report notes that MMLU-ProX Japanese dropped by 1.9 points from Gemma 4. My local 10 Japanese tasks serve as a way to check whether this side effect is showing up.

Getting It Running

The model card instructions simply pass the GGUF and draft to the official llama.cpp container.

docker run --rm --gpus all -p 8000:8000 -v "$MODEL_DIR:/models:ro" \
  ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459 \
  -m /models/Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf \
  --spec-draft-model /models/Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-ngl 999 \
  --alias nemotron-bolt --host 0.0.0.0 --port 8000 -c 65536 -ngl 999 --jinja

The draft model card requires llama.cpp b11808, which is two weeks newer than the container. I built that b11808 for DGX Spark with sm_121 and used it for measurements. The build takes just under 4 minutes.

I also ran the model card container b11459 on DGX Spark. DFlash works in this version too, and tool calls are returned in structured format. The difference is in slot count — the card instructions don't pass --parallel, so it starts with the default of 4 slots. In this state, single-request speed with DFlash was 117.79 tok/s and without DFlash was 55.61 tok/s. As a quick-start path, the container instructions work as-is. For consistency, the tables in this article use values from my own b11808 build with --parallel 1 explicitly set.

cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON
cmake --build build-cuda -j"$(nproc)" --target llama-server
./build-cuda/bin/llama-server -m Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf \
  --spec-type draft-dflash --spec-draft-model Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
  --spec-draft-n-max 7 --spec-draft-ngl 999 -c 65536 --parallel 1 -ngl 999 --jinja --port 8089

Startup takes 3.8 seconds, and the log shows DFlash loading. The draft receives intermediate states from 5 layers of the main model and proposes 7 tokens per computation. The log's n_extract=5 and n_max=7 match the technical report's description exactly.

common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=4, n_extract=5, sample_from_anchor=true

Running as a resident process with the full 64K context allocated, memory usage was 8.7 GiB. This is the weight's 5.8 GiB plus the draft's 0.7 GiB, plus KV cache for 64K tokens. To stay within the "4GB" target, you would shrink -c to match your expected input length.

One thing to note on the request side is the thinking default. The chat template's enable_thinking defaults to OFF, and all examples in the model card explicitly set it via chat_template_kwargs. The tables in this article also explicitly specify thinking ON and OFF per request. Tool calling returns structured tool_calls with just --jinja. llama.cpp has a dedicated parser for the Gemma 4 format, and it works as-is with NVIDIA's modified template.

Speed

Speed comparisons use the same DGX Spark, same llama.cpp b11808, and same Q4_K_M for both Bolt E4B and the parent Gemma 4 E4B-it. Single-request uses --parallel 1, 8-parallel uses --parallel 8 (8192 per slot), output is forced to 256 tokens, and the median of 3 runs is reported.

Raw speed without a draft is nearly identical between Bolt and Gemma 4. Since they share the same architecture and quantization, this is expected. The 8-parallel totals of 356 vs. 360 are also aligned. Qwen3.5 4B is about 10% faster in single-request, but about 10% slower in 8-parallel total. Qwen has no distributed draft available, so there is no speculative decoding row.

The difference appeared in the draft. Bolt's DFlash boosted single-request by 2.07×, while Gemma 4's MTP boosted it by 2.19×. As a result, Gemma 4 is 8% faster in single-request. The technical report states "DFlash Bolt is 1.05× faster than MTP Gemma 4 on RTX Spark," but the result was reversed on DGX Spark with llama.cpp. Looking at acceptance rates, DFlash passes 3.1–3.8 tokens out of 7 proposed, while MTP passes 2.7–2.9 tokens out of 4. DFlash proposes more per step, but the heavier per-step computation offsets this. The research page's "up to 21%" refers to the RTX 5070, and even in NVIDIA's own table for Spark systems the figure is 5%, so this is a difference that can reverse depending on the hardware.

NVIDIA's published RTX Spark figure of 67.0 tok/s is close to Bolt's draft-free 62.16 tok/s. The conditions differ — their measurement uses 3K token input returning 50 tokens, versus the forced 256 tokens in this article — but the order of magnitude matches.

Japanese

The 10 Japanese tasks involve reading free-form responses manually, with 9 tasks also evaluated by automated scoring. The budget (maximum output tokens, including both thinking and response) is 8,192, aligned with the Bolt 74B article.

The budget is 16,384 only for Qwen3.5 4B with thinking ON; the reason is explained below. Sampling followed each model's recommended settings.

❌* indicates a case where the output token budget was used up by thinking and no response was generated, making evaluation impossible.

All four configurations of Bolt and Gemma 4 scored 8/9, failing the same 1 task: summarizing to 150 characters while retaining 6 proper nouns. Bolt E4B retained 5 of the 6, which is better than Bolt 74B's 4. Since Bolt 74B and Lightning also fail this task, I don't attribute it to model size.

Qwen3.5 4B scored 7/9 with thinking OFF and 8/9 with thinking ON. The reason for extending the budget to 16,384 tokens for thinking ON is that some tasks consumed the entire 8,192 token budget just for thinking. The total across 10 tasks was 46,557 tokens in 742.5 seconds — 14 times longer than Bolt E4B with thinking ON.

One takeaway from this table is that the pass/fail results didn't change between thinking ON and OFF. With thinking OFF, total time was 15.1 seconds, less than a quarter of ON, and output tokens were 841 vs. 3,241. If you're using this as a conversation front-end that gives short answers, you can safely leave thinking off. As for whether English-only reinforcement learning degraded Japanese, the scores matched parent Gemma 4 E4B-it exactly, so no difference was visible at this task granularity.

The free-form task asked for two advantages and two disadvantages of remote work, evaluated by reading the output. Bolt E4B with thinking ON wrote a 253-token response organized into 4 items with headings and numbered lists, citing reduced commute time, location flexibility, blurred work-life boundaries, and weaker communication. Gemma 4 E4B-it wrote the same structure in 738 tokens. Both were reasonable in content; the only difference was length. The direction NVIDIA describes as "up to 80% reduction in output tokens" showed up as a 66% reduction in this one task.

Code Generation

5 tasks involve writing code from scratch based on Japanese specifications, plus a task of fixing 3 bugs in a mini repository within 30 turns. Scoring is done by running pytest locally.

Bolt E4B completed all 5 tasks and passed all 49 tests. This matches Bolt 74B's results and surpasses Lightning's 4/5. Gemma 4 E4B-it failed 2 tasks due to an import error: from typing import list. Bolt E4B did not exhibit this pattern in any of the 5 tasks. Since the reinforcement learning target wasn't code, it's unclear whether this is coincidence or a side effect — 5 tasks aren't enough to decide.

Qwen3.5 4B also passed all 5 tasks with an output token budget of 16,384. However, its output for parse_duration was 12,817 tokens — 7 times the 1,739 tokens used by Bolt E4B. The ability to reach the correct answer is the same, but the token usage differs.

Bug fixing scored 5/5 for all three models. Bolt E4B took 11 turns, Gemma 4 took 12, and Qwen3.5 4B took 8. Bolt 74B finished in 5 turns, confirming that reading and fixing multiple files in one step is still better handled by larger models.

Tool Calling

Tool calling is evaluated across 4 scenarios: single call, parallel call format, adherence to tool_choice=required, and a pseudo filesystem.

There is no difference between Bolt and Gemma 4. Since llama.cpp natively parses the Gemma 4 format, both return tool_calls properly. Qwen3.5 4B also matches on single calls and pseudo filesystem, but when tool_choice=required is specified, it returns without error but with 0 calls. Clients designed to always receive a tool call need to account for this behavior.

Agent suitability was assessed using 6 probes targeting capabilities that agents depend on, not by running the harness itself.

The 1 task failed in multi-tool selection was a recurring scheduling request where both Bolt and Gemma 4 failed to call the cron tool. Lightning and Bolt 74B failed the same task. Bolt E4B with thinking ON outperformed the parent in 3 areas: it followed the rules in a long system prompt, produced the same structured daily report format 4 out of 5 times, and did not call unnecessary tools when a simple explanation sufficed. Gemma 4 with thinking ON called tools roughly once every two prompts that only needed an explanation. This aligns with NVIDIA's explanation of training for "instruction priority" and "tool call timing."

Qwen3.5 4B with thinking ON produced the same structured daily report all 5 times, making it the only model in this table with a ✅ for template stability. However, it failed to follow long system prompt rules and called tools for prompts that only needed explanations.

That said, both opencode and Hermes Agent would rate Bolt E4B as "challenging" as well. Tool selection with many tools loaded and template output stability are the sticking points. This was the same verdict for Bolt 74B and Lightning — a 4B model as the reasoning core of a coding agent is not a realistic setup.

I also tested thinking budget limits. Using --reasoning-budget 1024 in llama-server to cap thinking at 1,024 tokens, the Japanese pass/fail stayed at 8/9, but total time increased to 67.5 seconds compared to 55.2 seconds without a limit. In the suitability probes, long system prompt adherence, over-calling, and template stability all dropped. This was a single run, but if you want to cut latency by limiting thinking, simply turning it OFF is cleaner than setting a budget.

What Changed from Gemma 4 E4B

Here I reorganize the tables above to show only the differences from the parent.

Adding Qwen3.5 4B gives a three-dimensional picture. Raw speed is 10% faster, and single-call and pseudo filesystem tool results match. Scores reach the same level as Bolt E4B with an output token budget of 16,384, but the tokens and time used along the way are 7–14× higher. The "up to 80% reduction in output tokens" NVIDIA claims is what this difference looks like in local numbers. tool_choice=required returns 0 calls, and template output stability is the best of the three models. Long context and Tau2, where Qwen outperforms in NVIDIA's table, are not covered by this task set.

It's also worth thinking about where a 4GB model fits in. This article's setup allocated the full 64K context as a resident process at 8.7 GiB. On a 128 GB DGX Spark, placing it alongside a larger model like Bolt 74B or Lightning is negligible. A setup where E4B handles conversation and tool calling as a front-end while routing heavy reasoning to a larger model in the back can be tested on the same machine.

I also tested on an 8 GB Jetson Orin Nano Super, building the same b11808 for sm_87. Unsloth's UD-Q4_K_XL (4.5 GB) starts up with an 8K context and achieves 14.68 tok/s single-request, with 0.393 seconds to first token. This aligns with NVIDIA's published figure of 15.5 tok/s for Orin Nano. Tool calls also return in structured format just like on Spark, so using it as a front-end that handles one tool call per conversation is feasible on 8 GB. On the other hand, NVIDIA's Q4_K_M (5.9 GB) fails to allocate CUDA memory on this machine with 6.3 GB free and won't start. There's no room for the DFlash draft either, with only 400 MB free after startup. On an 8 GB device, you need to choose a more heavily quantized version and not run any other models alongside it.

Let me also revisit my hypothesis from the previous article that "Lightning or E4B will be the target for specialized training." E4B's BF16 is 15.9 GB, and the reinforcement learning recipe, environment, and training datasets are publicly available via NeMo RL and NeMo Gym. Bolt 74B's BF16 at 149.8 GB wouldn't fit on Spark, but E4B is a feasible size. I haven't actually tried it, so I'll leave this as something to verify next time.

Summary

Nemotron 3.5 Bolt E4B 4GB ran at 128.79 tok/s single-request with llama.cpp on DGX Spark. The total for 8 parallel requests was 355.81 tok/s. Japanese 10 tasks scored 8/9, code generation 5/5, and tool calling 5/5, placing it in the same table as the 2-bit Bolt 74B. Turning thinking OFF did not change the Japanese 10-task pass/fail from the same 8/9 as ON, and total time for the 10 tasks dropped from 55.2 seconds to 15.1 seconds.

Running side-by-side with the parent Gemma 4 E4B-it under identical conditions, speed and Japanese were the same. Differences showed up in passing all 5 code generation tasks, following long system prompt rules, and not calling tools unnecessarily for explanatory questions. NVIDIA's explanation of reinforcement learning tuned toward "instruction priority" and "tool call timing" matches the observed differences locally. On the other hand, it remains a challenging fit as the reasoning core of a coding agent with many tools loaded.

DFlash was slower than Gemma 4's MTP in single-request on DGX Spark. Since this is a 5–8% difference that can reverse depending on hardware, the better approach is to choose based on which model you want as a front-end rather than on the draft method. On the 8 GB Jetson Orin Nano, Unsloth's 4.5 GB version ran at 14.68 tok/s, while NVIDIA's Q4_K_M version failed to load.

Next, I'd like to run the publicly available reinforcement learning recipe on DGX Spark to verify whether E4B can serve as a target for specialized training.

Reference Links

• NVIDIA Nemotron 3.5 Bolt E4B 4GB (Research Page)
• Nemotron 3.5 Bolt E4B 4GB: Technical Report (PDF)
• nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB — Q4_K_M GGUF and llama.cpp instructions
• nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash — DFlash draft GGUF
• nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 — BF16 used as the training base
• unsloth/gemma-4-E4B-it-GGUF — Q4_K_M and MTP draft for Gemma 4 E4B-it used in comparison
• unsloth/Qwen3.5-4B-GGUF — Q4_K_M for Qwen3.5 4B used in comparison
• llama.cpp release b11808 — Version required by the DFlash draft model card
• Running the 2-bit quantized MoE "Nemotron 3.5 Bolt 74B-A9B" on DGX Spark (2026-10-16, previous article)