Tried running DeepSeek V4 Flash-0731 at 284B on two DGX Spark units

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

When you hear "running a 284B model locally," what speed do you imagine? When I wrote an article about running DeepSeek V4 Flash-0731 on a single DGX Spark at the beginning of August, it was around 20 tok/s after dropping to 2-bit quantization. That's readable, but you still feel like you're waiting.

https://dev.classmethod.jp/articles/dgx-spark-deepseek-v4-flash-0731-llama-cpp/

This article is about what happens when you expand that to two units. I ran the 4-bit GGUF that wouldn't fit on a single unit and the official DeepSeek FP8 checkpoint on separate stacks.

Actually, I set up this 2-unit configuration once before at the end of June. Back then, it was a DSpark preview build, and with thinking disabled for code generation, it reached 55 tok/s, while long texts dropped to 2.4 tok/s by the time 247K tokens were read. And that 55 tok/s was measured when hitting it directly with curl—when accessed from an OpenAI-compatible client, the response appeared empty.

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-dspark/

In less than two months, both the model and the way it runs have completely changed. The model is now the official 0731 release, it starts without any custom patches, and nodes are connected via RDMA with MTU 9000. Measuring again with the same two machines yielded completely different numbers.

To state the conclusion upfront: 284B has become practically usable locally. Running the official FP8 with vLLM delivers a decode speed of 76 tok/s. That's well above silent reading speed, so there's almost no sensation of waiting for generation. A 900K token prompt also works, and even with 6 people hitting it simultaneously, it maintains a combined 210 tok/s. Tool calls and code fix tasks all passed in my local harness.

This is running on two boxes sitting on a desk, not a cloud API.

This article compares two configurations for running DeepSeek V4 Flash-0731 on two DGX Sparks with real measurements and organizes which to choose based on use case. I hope it resonates with people curious about how far local LLMs can be pushed toward practical use.

How Your Options Expand with Two Units

DeepSeek V4 Flash-0731 is a MoE model with 284B total parameters, 13B active. It comes in two forms: Unsloth's GGUF and DeepSeek's official safetensors.

https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

GGUF has a rich variety of quantization options. Here's how the sizes and single-unit compatibility look. I'll list both GB and GiB to avoid misjudgments from mixing them. The DGX Spark's "128GB" appears as 121 GiB from the OS.

This time, the single-unit limit was UD-IQ3_S; the next one up, UD-Q3_K_XL, is 119.4 GiB, leaving no room for the KV cache. In other words, the practical upper limit for a single-unit configuration is 3-bit.

On the other hand, the official safetensors come in only one variant.

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731

This is 166.9 GB, or 155.43 GiB, in 48 shards. It's a format mixing FP8 with FP4 for the MoE portions, and there's no choice in quantization. It obviously doesn't fit on one unit, so running it is itself the reason for a 2-unit configuration.

To summarize, what becomes accessible with two units is "4-bit GGUF" and "official FP8." The latter in particular is the official distribution without quantization reduction, making it the ceiling if you're running locally.

Splitting 4-bit Across Two Units with llama.cpp

llama.cpp has an RPC backend that lets you offload layers to a ggml-rpc-server running on another machine. Only the client side needs the GGUF file; tensors are sent over the network. This time, only the client side needed to download the model. The vLLM approach coming up later requires 167 GB on both nodes, so this difference is quietly significant.

Startup involves launching the client and worker separately. The worker waits on the second unit, and the client is run from the first.

ggml-rpc-server --host 192.168.200.14 -p 50052 -c

llama-server -m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \
  --rpc 192.168.200.14:50052,127.0.0.1:50052 \
  -ngl 999 -sm layer --no-mmap --cache-ram 0 \
  -c 32768 -np 1 --host 127.0.0.1 --port 8080

I also started one worker on the first unit so that its own GPU is used via RPC as well. This way both nodes are treated identically, and layers split cleanly.

The nodes are directly connected via QSFP cables, with a 200 Gb/s link, MTU 9000, and 0.7 ms round-trip time. When llama.cpp finds libibverbs at build time, it incorporates RDMA, so it automatically connects via RoCEv2 on startup.

RDMA probed: dev=rocep1s0f1 gid=3 RoCEv2 qpn=595 inline=316
RDMA activated: qpn=595->595 mtu=4096 rx_depth=24

GID index selection previously required manual specification, but now it resolves automatically.

The 144.44 GiB model loaded in 401 seconds, settling with 101 GB on the first unit. Since it's layer splitting rather than strict halving, it doesn't come out exactly equal.

One thing worth knowing before setting this up: as of August 2026, connecting two or more workers in llama.cpp itself causes a crash the moment inference starts. Applying the unmerged PR #26500 fixes it. I lost about half a day to this detour myself, so I've folded the details below.

ggml/src/ggml-rpc/ggml-rpc.cpp:519: Remote RPC server crashed or returned malformed response

[create_node] invalid data ptr[graph_compute] failed to create graph node 3085

Running Official FP8 with 1M Context in vLLM

The other path is vLLM. It distributes the official FP8 checkpoint across two units with tensor parallelism. Setting it up from scratch is quite involved, so I used a published recipe.

https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark

It's a complete setup for two DGX Sparks with Docker Compose and startup scripts included. The image used internally is Anemll's vLLM port for the GB10.

https://github.com/Anemll/dspark-vllm-gx10

I used the recipe's defaults as-is. max_model_len is 1,048,576, max_num_seqs is 6, KV cache is nvfp4_ds_mla, and DSpark speculative decoding is enabled at 5 tokens. The only change I made was explicitly disabling thinking.

The startup log reveals how much KV was allocated.

Available KV cache memory: 12.34 GiB (TP0) / 12.05 GiB (TP1)
GPU KV cache size: 1,789,449 tokens
Maximum concurrency for 1,048,576 tokens per request: 1.71x

KV for 1.79 million tokens was loaded. For a 1M request, there's room for 1.71 of them. Memory was nearly equal at 108 GB on the first unit and 107 GB on the second. Since it's tensor parallel, unlike the layer-splitting llama.cpp, it divides cleanly in half.

Setup cost is heavier than llama.cpp. The image is 9.79 GB, and the model is 167 GB, which needs to be placed on both nodes, and there's a specific convention for Hugging Face cache placement. When I downloaded with hf download --revision <sha>, refs/main wasn't created, and the snapshot couldn't be found in offline mode. If you download with a fixed revision, writing refs/main yourself lets it work smoothly.

That said, compared to doing the same thing at the end of June, it's much easier now. Here's a comparison of what changed between then and now:

The June figures are from that article and weren't re-measured this time. Since the model changed from a preview build to the official version, it's not a strict comparison, but the difference of 28x faster despite a prompt 3.6x longer seems well beyond what measurement error can explain. The combination of the KV cache format changing to nvfp4_ds_mla, vLLM upgrading to the 0.25 series, and the GB10 port maturing all likely contributed. I haven't isolated how much each factor contributed.

Comparing on Four Axes

Here's the main part. I'll go through speed, long context, concurrent access, and quality in order.

Speed Varies 3.8x Depending on Stack Choice

First, with llama.cpp, I measured the same quantization on one unit versus two. I reused the harness from the single-unit article: context 2,048 tokens, 128 tokens generated, median of 3 runs.

The striking figure is the decode ratio. Splitting across two units only costs 0.97x to 0.99x in decode. You could also say the cost of distribution is almost nothing. Each layer sends only tens of KB between nodes, which gets lost in the noise on a 0.7 ms link. Prefill moves MB-scale data, so it drops to 0.85x–0.90x, but even that gap narrows as context grows.

In the article connecting two DGX Sparks in February, Qwen3-235B saw about a 6% drop. The model is different here, but the degradation has narrowed to 2%. The addition of RDMA may be contributing.

https://dev.classmethod.jp/articles/dgx-spark-two-node-clustering/

Beyond that, let's look at the cost of raising quantization. Comparing within the same 2-unit configuration:

The gap between the 3-bit that was the 1-unit limit (18.02) and the 4-bit that only works on 2 units (16.54) is 8.2%. That's a trade of 8% speed for one step higher quantization.

But this changes when speculative decoding is added. DSpark speculative decoding was merged into llama.cpp main on August 2nd—the same day I published the single-unit article. That article said "it's not in main yet," so it was already outdated at the time of publication.

Enabling it with --spec-type draft-dspark gave these results:

4-bit with speculation reaches 20.03 tok/s, nearly matching the 20.13 of the fastest single-unit 2-bit configuration. You could view it as getting 4-bit at the same speed as 2-bit.

To be honest though, the benefit of speculation varies considerably by content. Pulling acceptance rates from logs showed a range from 0.34 to 0.95, with average accepted lengths moving between 2.68 and 5.77 tokens. Without multiple runs and taking the median, it's easy to cherry-pick favorable numbers.

vLLM, on the other hand, was in a different league.

Decode at 76 tok/s, prefill around 1,900 tok/s. That's 3.8x the 20.03 tok/s of 4-bit llama.cpp with speculation.

To put 76 tok/s in perspective: it's clearly faster than silent reading speed. There's almost no sensation of waiting for generation to finish. Achieving this speed locally with a 284B model was honestly beyond what I had imagined.

One note when comparing these numbers: the two stacks differ in weights and quantization method. The llama.cpp side is 4-bit K-quant at 144.44 GiB; the vLLM side is FP8 mixed with FP4 for MoE at 155.43 GiB. Speculative decoding implementations also differ—vLLM has MTP enabled by default. I think it's more accurate to read this as "this much difference comes from the choice of stack" rather than declaring one engine superior.

Long Context: Only TTFT Grows

Since vLLM claims 1M context, I tried pushing it to see how far it'd go—a 900K token prompt.

Even with the prompt 440x longer, decode barely drops. From 75.6 at 2,048, it only falls to 69.04 at 900K—a drop of just 8.7%. What grows is TTFT; at 900K tokens, it took 14.5 minutes to generate the first character.

Top shows decode, nearly flat from 75.6 at 2K to 69.0 at 900K. Bottom shows TTFT, nearly linear on the log-log graph—meaning wait time grows proportionally to prompt length.

The constraint for long-context use isn't speed, it's latency. For use cases like feeding an entire repository and asking questions, it becomes: submit, make tea, come back. Once it starts generating, it's fast—so the only rough part is the initial wait.

For reference, the recipe's stated values were 1,028 seconds TTFT and 875 tok/s prefill for 900K tokens. My measurements were 863 seconds at 1,043 tok/s, slightly faster than stated.

I only measured llama.cpp up to 32,768 context. It also showed that decode degradation with longer contexts was gentler on 2 units (maintaining 0.98x), but I haven't tested 1M-scale, so my conclusion is: for long context, use vLLM.

Concurrent Access Doesn't Hit a Ceiling Even at 12 Users

Thinking about the use case of a team sharing one setup, I tested increasing concurrent requests. With llama.cpp 4-bit and 12 slots configured:

All requests succeeded up to 12 concurrent, and total throughput was still growing. In the single-unit article, the ceiling was 57.93 tok/s at 8 concurrent, dropping to 56.89 at 12—so this is a contrast. But the engine and quantization are both different, so don't subtract the raw numbers.

On a per-user basis, 12 users get down to 5.11 tok/s. That's about the speed where the next line appears while you're reading, so whether you can tolerate it depends on the use case.

I also measured vLLM up to 6 concurrent. With short prompts, it reaches a combined 210.8 tok/s.

It's strong for short queries from multiple people, but the benefit of parallelism disappears as context grows. At 32,768, the combined 27.6 tok/s for 6 concurrent is barely different from the single-request 18.0. Long context and concurrency compete for KV, so the design favors one or the other.

Both continue growing total throughput with more concurrency, with no sign of plateauing. vLLM line uses prompt length 2,048; llama.cpp uses 4-bit GGUF. Weights and engines differ, so the absolute levels can't be compared.

Quality Showed No Difference in Scores

To see whether raising quantization was worthwhile, I ran the same quality harness from the single-unit article on both. That's 5 machine-judged tasks, tool calls, and a task of fixing a small Python repository with 3 planted bugs until tests pass.

All scores were identical. Honestly, this harness failed to demonstrate "the value of higher quantization." The 3-bit in the single-unit article also scored perfect, so if both score perfect, there's no differentiation. It confirms nothing is broken, but nothing more. The agent behavior differed at 5 vs 4 turns, but each was only run once, so drawing any conclusions from that would be overreaching.

The difference showed up in elapsed time—the code fix task took 3.9x longer. This directly reflects decode speed, and in an agentic pattern with multiple back-and-forth turns, the wait time is cut to a quarter. Even if you reach the same conclusion, the experience would feel quite different.

Which to Choose by Use Case

Summarizing the measurements so far, here's how to choose:

Personally, I'd lean toward vLLM for regular use. It wins on speed, long context, and concurrency, and speculative decoding is effective from the start. However, setup is heavier, requiring 167 GB on both nodes and a Docker-based launch.

llama.cpp's strengths are its lightness and swappability. You can decide for yourself whether to drop quantization for speed or raise it for quality, and only the client side needs the model. It seems better suited for when you're still experimenting or want to compare various quantizations.

Whichever you choose, what stands out to me is that 284B is practically usable locally. Tool calls go 6/6, a Python repository with failing tests gets fixed in 5 turns, and you get a response even when feeding 900K tokens—equivalent to a full repository. The fact that this runs within sight of power cables and network cables, not somewhere beyond the network, should matter quite a bit depending on the data you're handling.

Conclusion

I ran DeepSeek V4 Flash-0731 on two DGX Sparks and compared two configurations: llama.cpp and vLLM.

A 284B model reached practical speed on two units sitting on a desk. Running the official FP8 with vLLM delivers 76 tok/s decode, handles 900K token prompts, and maintains a combined 210 tok/s even with 6 concurrent users. All quality harnesses passed. Where a single unit was limited to 3-bit, two units now reach 4-bit and the official FP8. The cost of distributing the same model across two units was about 2% in decode—smaller than I expected.

I'll also note what I didn't accomplish. I only tested llama.cpp context up to 32,768 and haven't verified 1M-scale. Quality-wise, all four harnesses scored perfect across the board, so I couldn't measure quantization differences. I'd need harder tasks to fill that axis. Also, the tensor-parallel-over-RPC implementation in PR #26610 exists, but I couldn't get it working with my build this time. When that lands in main, the 2-unit speed story may change.

Reference Links

• unsloth/DeepSeek-V4-Flash-0731-GGUF
• deepseek-ai/DeepSeek-V4-Flash-0731
• MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark — vLLM recipe for two DGX Sparks
• Anemll/dspark-vllm-gx10 — vLLM port for GB10
• llama.cpp PR #26500 — Fix for buffer reference with multiple workers
• llama.cpp Issue #26820 — Report of the same symptom
• llama.cpp Issue #26529 — About --cache-ram 0
• Running DeepSeek V4 Flash-0731 on a Single DGX Spark with llama.cpp (Published 2026-08-02)
• Connecting Two DGX Sparks with a Cluster Cable (Published February 2026)