I tried running GLM-5.3-Flash 320B on 2 DGX Spark units and measured whether using 2 units is worth it

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Last time, I ran GLM-5.3-Flash at 320B on a single DGX Spark using a 2-bit GGUF, achieving 17.7 tok/s. Just seeing a 320B MoE return responses at a readable speed on a single machine sitting on a desk was already an interesting sight. However, the quantization was trimmed down to 2-bit, and running inference with thinking set to the default maximum takes 15 minutes for 10 Japanese tasks. This time, the question is: what happens if we connect a second unit to the same model?

https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4

With two units, the combined unified memory is 242 GiB, which can directly accommodate the community-published NVFP4 checkpoint (184.3 GiB). The trimming is far less aggressive than 2-bit, and the engine is vLLM tensor parallelism. We've covered running large models on two connected DGX Sparks twice before with DeepSeek, but this is the first time we can place a single-unit measurement side by side with a two-unit measurement using the same harness and the same tasks. The question of whether a second unit is worth buying is being answered here with GLM-5.3-Flash. Since it's obvious that speed will improve, what we want to see is not just whether it works, but whether it reaches a usable level on the same tasks as the single-unit piece.

To state the conclusion upfront: single-request speed went from 17.7 → 26.5 tok/s, a 1.5x improvement, and 10 Japanese tasks went from 927 seconds to 413 seconds. It's not just functional — it's at a level suitable as a working tool. Code generation produced complete answers on all 5 questions, fixing a buggy repository completed in 38 seconds, tool calls honored contracts including tool_choice=required, and recall of premises in 10-turn brainstorming was perfect on all questions. However, the real value of two units wasn't in the speed multiplier — that's the impression this time around. Speculative decoding stayed on, 8 slots fit within the 262K context, the combined throughput of 8 parallel requests reached 76.2 tok/s, and even a 200K token prompt returned the passphrase. Since you can choose configurations based on use case — writing code alone, reading long documents, sharing among a few people, or sharing among many — a personal verification machine becomes a team server. The Japanese character corruption encountered in the original was eliminated simply by switching to the RedHat version checkpoint. The cause and the story of how switching affected speculative decoding performance are also covered.

Since the single-unit piece (as of the 2026-08-30 article) already covered the model characteristics, reasoning_effort, quantization, and images, this article focuses on what changes with two units.

https://dev.classmethod.jp/articles/dgx-spark-glm-5-3-flash-first-touch/

The wiring for two units is unchanged from the DeepSeek V4 Flash-0731 two-unit piece (as of the 2026-08-10 article) — still QSFP direct connection with MTU 9000 RDMA.

This article covers running the GLM-5.3-Flash NVFP4 checkpoint on 2 DGX Sparks with vLLM, the process of varying speculative decoding and parallelism to determine configurations by use case, comparisons using the same harness as the single-unit piece, and the cause of Japanese character corruption and actual measurements after switching checkpoints. I hope this resonates with people who are on the fence about buying a second unit.

The Verification Environment Is Two DGX Sparks Connected via QSFP Direct and vLLM Tensor Parallelism

The configuration is vLLM TP=2 with one unit as head and the other as worker. The head accepts OpenAI-compatible API requests, and the weights are split across both units. Here is a summary of the environment.

The configuration in diagram form is as follows.

The compose files, env, startup and shutdown scripts, and patches are all bundled and published. For readers who want to get it running today, the README steps there are the shortest path, and this article covers the actual measurements and obstacles behind it.

https://github.com/himorishige/glm53-flash-2x-dgx-spark-recipe

Let me write out the fixed values that won't be touched. --block-size 2304 is the page size required by fp8 paged MQA; removing it causes output corruption without any error. --language-model-only is for skipping the multimodal front-end load, and without it you consume an extra 15.7 GiB. The tool call parser is glm47; specifying glm silently drops tool calls. The startup order is to bring up the worker first, wait 25 seconds, then start the head. When stopping, bring down both ranks together. Stopping only one leaves the remaining GPU pegged at 100% and frozen. On the host side, it's enough to set vm.swappiness=0 and drop_caches before startup.

The KV cache pool size is not fixed manually; it's measured by vLLM's profiler. With this checkpoint, 1,151,844 tokens fit in a 7.48 GiB pool, which works out to 4.4 simultaneous requests that each fill up the 262K context.

The Official Image Would Not Start; We Got It Running with a Community Image and Patch

vLLM released a dedicated image vllm/vllm-openai:glm53-flash-arm64-cu130 to coincide with the GLM-5.3-Flash release, but this model could not be started in a two-unit GB10 configuration. Weight loading and NCCL connection went through, but the following assert fires at warmup.

RuntimeError: concat_and_cache_mla, /workspace/csrc/libtorch_stable/cache_kernels.cu:866, pe_dim must be 64 for fp8_ds_mla

Changing --kv-cache-dtype to fp8, fp8_e4m3, or auto made no difference. GLM-5.3-Flash is a NoPE model with rope dimension 0, and the cache path used by the DeepSeek Sparse Attention indexer is not accepted by the stock kernel. Every report of it running on two Sparks used a patched image or plugin; no case of the stock image working alone was found.

So the main approach is to use tonyd2wild's v11 image with the SM121-compatible top-k patch applied via compose bind mount, distributed by the same author. The patch is a 2-location diff from stock that avoids persistent_topk, which doesn't fit in GB10's shared memory. Without it, the engine reportedly crashes on long prompts — and indeed, with this configuration, prompts up to 200K went through. Startup takes 542–608 seconds and passes three gates: the model list, one math question, and a tool call.

Speed Is Determined by Speculative Decoding, and MTP k=3 Was Fastest for This Checkpoint

Once it started, we measured by changing one factor at a time from the day-1 configuration. Greedy with forced length of 256 tokens; C=1 is the median of 3 runs; code generation is the effective tok/s when actually solving 5 problems with effort=low. All rows use the v11 image + top-k patch + fp8_e4m3 KV, 262K context, 2 slots.

The no-speculation baseline of 14.60 tok/s matches community reports (14.3–14.6), and this is the denominator. Just using the model's built-in MTP head brings it to 22.63–26.47 tok/s, with k=3 being the fastest for single requests. This matches the trend seen on single-unit llama.cpp where draft 3 was most effective.

In the original article's LibertAI checkpoint, the lineup in this table was different. DFlash2 — a dedicated drafter for GLM-5.3-Flash that loads a separate 2.2 GB model — was fastest at 30.84 tok/s single-request, followed by MTP k=3 at 27.78. With the RedHat version, the no-speculation baseline matched exactly, but DFlash2 alone dropped to 25.57, and the gap with MTP disappeared. Since the drafter is a separate model trained to match the raw weights, a change in checkpoint quantization likely reduces draft acceptance rate — though since we didn't measure acceptance rate directly, this is interpretation. Interestingly, for code generation effective speed, DFlash2 remains at the top with 27.74 tok/s. Code has many boilerplate patterns that drafts tend to hit well. Since this drafter is CC BY-NC-ND 4.0, commercial use would mean using the MTP rows.

Let's also look at KV cache type and context length. With KV set to bf16, the 3.62 GiB needed for a single full 262K request doesn't fit in the pool, making it impossible to start at all. With fp8 KV, 4.4 requests fit in the same space, so there's no reason to choose bf16 KV in this configuration. For context length, dropping from 262K to 65K makes no difference: single-request is 25.26 vs. 25.26, and 8-parallel total is 76.2 vs. 75.3. There's no need to reduce context to gain speed. Rows within 3% of each other are treated as equivalent.

MoE backend flashinfer_cutlass, CUDA graph, batched 8192, autotune off, b12x, and util 0.90 were not measured this time.

8 Slots Fit at 262K, and Two Configurations Cover the Use Cases

Next, parallelism. In the single-unit piece, 8 slots at 65K context was the limit, but with two units, 8 slots fit at 262K. Of GLM-5.3-Flash's 45 layers, 34 are linear attention (KDA) with no KV cache, and the profiler allocated a 7.48 GiB KV pool holding 1,151,844 tokens. Here are parallel totals by configuration.

Single-request speed doesn't drop even with 8 slots, reaching 76.2 tok/s total at 8-parallel with MTP k=4. Single-unit llama.cpp was 63.97 tok/s at 8-parallel, so two units comes out ahead in total throughput as well. All 8 slots can be set to 262K context, and the actual pool capacity is about 4+ simultaneous requests at full 262K. For typical use cases like brainstorming or document reading, 8 people hammering it won't break it.

From these measurements, two configurations suffice. For solo use: 262K with 2 slots and MTP k=3. For sharing among several to many people: 262K with 8 slots and MTP k=4. Those who spend a lot of time writing code can switch solo speculation to DFlash2 to push code generation effective speed to 27.7 tok/s. There's no need to drop context length to the single-unit piece's 65K; speed is unchanged at 262K. Use case judgments are summarized in the "Which Configuration to Choose by Use Case" section after the quality results.

The published recipe defaults to solo use; shared mode is switched by setting OVERRIDES at startup.

# Shared use: MTP k=4 with 262K × 8 slots
OVERRIDES="SPEC_METHOD=mtp MTP_NUM_TOKENS=4 MAX_NUM_SEQS=8" scripts/start-worker.sh

With one unit it was "a box for my own use," but once it holds 8 concurrent 262K contexts at a combined 76 tok/s, it works as a server for several people in a department. The biggest change from going to two units isn't the speed number — it's this expansion in how you can use it.

A 200K Token Prompt Still Returned the Passphrase

Long-context tests were run on the 2-slot configuration with DFlash2. A unique haystack was prepared for each request with a passphrase embedded at the 50% depth, measuring TTFT, prefill, and decode at forced length 512, then verifying recall by asking for the passphrase in a separate request.

Prefill is flat at 1,369–1,480 tok/s through 200K, and TTFT scales proportionally with length. For a 200K token prompt, 139.4 seconds until the first character returns, and the passphrase came back correctly for all 5 sizes. The reason decode speeds up with longer prompts is that speculation acceptance rate increases — when the output is echoing back the haystack content, the drafter tends to hit well. Since speculation-enabled decode depends on what's being generated, it's safer to look at TTFT and prefill as metrics for long-context performance.

We also ran two simultaneous requests. Sending two 200K requests concurrently gave TTFT of 209.7 seconds, per-request prefill of 1,070 tok/s, combined decode of 3.5 tok/s, and both completed in 292 seconds. That's holding a combined 400K tokens of context simultaneously. The context limit is set at 262K, so 1M was not tested this time.

What Changes When Measured with the Same Harness as the Single-Unit Piece

From here, the results from running the same tasks as the single-unit piece on a serve with 8 slots and DFlash2 loaded. Context length was aligned to the single-unit's 65K, but as noted in the previous section, this has no effect on speed. The single-unit column uses figures from the previous article running a 2-bit GGUF on llama.cpp as-is.

KV cache type is fp8_e4m3 for 2-unit and f16 for 1-unit — this alone could not be aligned. For the effort=max row with an 8192 token budget, one task on both machines — summarizing while preserving proper nouns — ran out of budget during thinking and returned with the body empty. This task was remeasured separately with a 16,384 budget, where 2-unit scored 8/9 at 577.5 seconds. Only the proper nouns task failed, preserving 5 of 6 terms but dropping one department name.

Time is shorter across all items. Japanese 10 tasks effort=max went from 927 to 413 seconds, brainstorming A from 433 to 386 seconds, and code fixing from 72 to 38 seconds. Even running with the default max, responses come back in just under 7 minutes, softening the impression from the single-unit piece that "max doesn't match local speed."

On pass/fail, the failing tasks are the same cast — 50-character constraint, specified words, proper nouns — as with the single unit. The code generation high run dropped 2 out of 11 tests on slugify, but since the same problem passed completely on low, max, and 16k budget runs, this looks more like sampling variance than a capability gap. In the single-unit piece, running low 3 times gave 7/9, 7/9, 6/9, so a 1-point difference in a single measurement can't be called a capability difference. Mechanical matching for all 3 brainstorming scenarios was perfect, same as before.

To be honest about parallelism: DFlash2 at 8-parallel total is 59.42 tok/s, which is below the single-unit's 63.97. DFlash2 is disadvantaged at higher concurrency, but per-request at 13.4 tok/s is faster than the single-unit's 8.42, and for shared use, MTP k=4 reaches 76.2. Don't read the total alone and conclude that two units is slower.

The agent suitability table shows probes targeting individual capabilities that the harness depends on — not the harness itself running. With two units, tool selection from 40+ tools failed at all 3 levels, and max failed one language-fixation probe as well. These are also single measurements, so no definitive attribution to quantization differences. The judgment is the same as in the single-unit piece: works for Hermes Agent equivalents, challenging for opencode-level tool counts.

The Japanese Character Corruption Was Due to the Checkpoint

The original verification was done with the LibertAI version of the NVFP4 checkpoint, which was the first one published by the community. During the main brainstorming runs, I noticed unfamiliar characters mixed into the output: U+FFFD replacement characters appearing 7 times in 27,757 characters, with patterns like "許容\uFFFD囲" and "\uFFFD実的" where a single kanji was missing. The same probes run on the single-unit 2-bit GGUF over 28,000+ characters had zero occurrences.

I eliminated conditions one by one on the LibertAI version. The U+FFFD column is the count of replacement characters in the total output.

It appears regardless of speculation type or whether speculation is off. Trimming the tail with top_k 40 or min_p 0.05 to match llama.cpp defaults doesn't eliminate it, and neither does switching KV to bf16. In other words, this isn't a sampler tail issue or KV precision issue — the model is selecting invalid tokens as primary candidates. Even decoding the token IDs received via return_token_ids offline using the checkpoint's bundled tokenizer.json shows corruption at the same positions, confirming this is a problem with the token sequence itself, not the display layer.

The Corruption Mechanism Is in the Split Between 2-Byte Fragments and 1-Byte Continuations in the Tokenizer

The tokenizer's md5 matches zai-org's original, so it's not a modification in the checkpoint. Examining the vocabulary with the tokenizers library reveals that GLM-5.3's byte-level BPE doesn't hold most Japanese simplified-form kanji as single tokens — they're spelled out as 2-token sequences of "2-byte fragment + 1-byte continuation." For example: 測 as e6b8 + ac, 範 as e7af + 84, 陥 as e999 + a5 — and so on, where 1,095 of the 154,820 vocabulary tokens are not valid UTF-8 on their own. When the model outputs the 2-byte fragment and then skips the 1-byte continuation to move to the next character, a replacement character is born at that point.

The statistics of corrupted words align with this split pattern. 「現実」 was corrupted 11 out of 66 times, 「効果測定」 5 times, and similarly for 許容範囲, 陥る, 桁, 継ぎ, 併用, 拡大, 毀損 — all containing characters that follow the fragment path. Meanwhile, 「現場」, which can be spelled with one token per character, appeared 95 times without a single corruption.

Switching to the RedHat Version Brought It to 0

While in the middle of isolating which layer was causing the continuation decision to break, I found a report from the v11 image author who had observed the same symptom with Hangul and reported it disappearing when switching to the RedHat checkpoint. The LibertAI version is a checkpoint where routed experts were NVFP4-quantized using ModelOpt weight-only; the RedHat version used llm-compressor to quantize the same layers to NVFP4. The model and tokenizer are identical. Since it was just a matter of swapping the model path without changing any flags, I tried it as-is.

With the same Japanese probes, same number of responses, same Japanese ratio, the previously corruption-prone 「測」「範」「現実」 appeared 40+ times without a single corruption. The full output of this article's main harness and 30 brainstorming turns also had zero replacement characters. The conclusion is that this is caused by the quantization construction of a specific checkpoint — not the weights, tokenizer, kernel, speculation, or KV. I wasn't able to isolate exactly what's wrong inside the ModelOpt version, but as a user, knowing this much was enough.

In the original article, I addressed this symptom by writing a UTF-8 guard logits processor that masked corrupting tokens at every step. It was a workaround that dropped speed to 14.6 tok/s due to incompatibility with speculative decoding, but it's unnecessary with the RedHat version. The guard remains in the recipe for those who continue using the LibertAI version.

Which Configuration to Choose by Use Case

Let me draw a line between "works" and "usable." Here, the criteria are: not failing harness tasks, latency appropriate for the use case, honoring contracts like tool_choice, and no character corruption. Meeting all four is "usable"; having a condition on any one is "conditional."

All four top rows in the table had no failing tasks. Looking at speed alone, 17.7 tok/s on a single unit was already readable, but the difference between code fixing taking 72 vs. 38 seconds, and default effort=max Japanese 10 tasks taking 15 minutes vs. just under 7, is quite tangible in terms of how it feels to wait. Personally, writing code solo and sharing it among a few people for brainstorming were the use cases I found most compelling from this run.

What stuck with me was a 320B model becoming a team tool with two units. It fixes a Python repository with failing tests in 5 turns and 38 seconds, answers 8 questions about a 12,242-character document in 58 seconds, and returns passphrases even from a 200K token load. It sustains a combined 76 tok/s with 8 people hitting it simultaneously — running on two boxes sitting on a desk, not in the cloud.

The conditional row deserves honest treatment too. Resident agent use: probes for selecting the right tool from 40+ tools failed at all 3 levels, so clients with many tools remain challenging, same as with one unit. This didn't change by adding a second unit or switching checkpoints.

For reference, I also sent the same prompts from my local Mac to Fireworks' GLM-5.3-Flash (glm-5p3-flash). Single-request median was 46.6 tok/s (10 runs, 34.0–51.8) — 1.8x the two-unit's 26.5; at 8-parallel, ~60 tok/s per request, 397 tok/s total, or 5x the two-unit. The gap is large for long inputs: 200K token TTFT was 7.49 seconds there vs. 139.4 seconds locally. These are single measurements from one time of day, with shared tenancy adding run-to-run variance — but the safe read is that there are almost no scenarios where the two-unit setup beats the cloud on speed. If there's still a reason to choose two units, it's: data stays local, you hold 262K contexts in your own box, and there are no token charges. The "usable as a team tool" framing in this article is premised on those conditions.

Summary

I ran GLM-5.3-Flash NVFP4 with vLLM across 2 DGX Spark units, varying speculative decoding settings and parallelism counts, then compared the results against the same tasks used in the single-unit article.

As for whether it's practically usable — it is. The four use cases of writing and fixing code solo, reading long documents, bouncing ideas with a small group, and sharing among many users all passed the tasks without dropping results, and ran faster than a single unit. Configuration is chosen based on use case. For solo use, 262K with 2 slots and MTP k=3 gives 26.5 tok/s; for code-heavy workloads, DFlash2 gives an effective 27.7 tok/s; for sharing, 262K with 8 slots and MTP k=4 gives a combined 76.2 tok/s. The value of 2 units lies less in the 1.5x boost for single requests and more in this range of flexibility.

Another finding was empirical evidence that the behavior of NVFP4 checkpoints changes depending on how they were created. The ModelOpt-based checkpoint I used initially corrupted roughly 2 Japanese kanji characters per 10,000, and switching to the RedHat version based on llm-compressor reduced this to 0. The same switch also changed the effectiveness of speculative decoding — the external DFlash2 drafter slowed down while the built-in MTP became the fastest option. When selecting a checkpoint, it's safer to run your own language and use-case probes at least once, rather than relying solely on benchmark scores.

I'll also note the limitations. The official stock image failed to launch; what worked was a community image with patches applied. The DFlash2 drafter is licensed CC BY-NC-ND 4.0, so for commercial use, MTP is the upper bound. The root cause of continuation token drops with the ModelOpt version was not fully isolated. The cutlass MoE backend, CUDA graph, and batched 8192 were not tested. In the language-fixed agent evaluation, the model dropped at most 1 question at peak, and whether that's within single-run variance has not been verified through repetition.

Next, I'd like to measure endurance with multiple users simultaneously on 262K contexts and turn the conditions for successful team operation into concrete numbers. Since the SGLang GB10 path has also emerged, a side-by-side comparison using the same tasks would also be interesting.

Reference Links

• RedHatAI/GLM-5.3-Flash-NVFP4 - Hugging Face — Checkpoint used in this article (llm-compressor / compressed-tensors)
• LibertAIDAI/GLM-5.3-Flash-NVFP4 - Hugging Face — Original checkpoint (ModelOpt; see the section on Japanese character corruption)
• zai-org/GLM-5.3-Flash - Hugging Face
• incoai/GLM-5.3-Flash-DFlash2 - Hugging Face — DFlash2 drafter (CC BY-NC-ND 4.0)
• tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark — v11 image and SM121 top-k patch, U+FFFD comparison by checkpoint
• Libertai/glm53-flash-vllm-gb10 — stock image assert and cutlass util floor
• himorishige/glm53-flash-2x-dgx-spark-recipe — compose files, scripts, and patches used in this article
• GLM 5.3 Flash API & Playground | Fireworks AI — API used for reference comparison
• 320B の GLM-5.3-Flash を DGX Spark 1 台で動かして実用の分かれ目を測ってみた (Published 2026-08-30)
• 284B の DeepSeek V4 Flash-0731 を DGX Spark 2 台で動かしてみた (Published 2026-08-10)
• DGX Spark 2 台をクラスタケーブルでつないでみた (Published 2026-02)