Tried running DeepSeek V4 Flash 284B on DGX Spark with DwarfStar 4

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

ds4.c is a small inference engine written by Redis author antirez, specifically for DeepSeek V4 Flash. It's not a general-purpose GGUF runner or a llama.cpp wrapper — it's a C-based engine designed solely to run "one specific model, locally, end-to-end in a complete form." Just recently, it was renamed to "DwarfStar 4" and the CUDA backend was officially merged. What had previously been Mac Metal-only can now be built with a single make on Linux + NVIDIA as well.

https://github.com/antirez/ds4

DeepSeek V4 Flash is a MoE model with 284B parameters. Of the total 284B, only around 13B are actually active at any given time (per DeepSeek's announcement). The context it can handle is 1 million tokens. DwarfStar 4 is designed to load this onto a 128GB-class machine using 2-bit quantization, and to persist the internal memory used during generation (KV cache) to disk. Since I have a DGX Spark (GB10, 128GB unified memory, 273 GB/s memory bandwidth) on hand, I ran it right after the CUDA support was merged.

The three things I was personally curious about were:

1. How does a 284B MoE actually fit on a 128GB DGX Spark? Are the weights also being streamed from disk?
2. Would the community report that "the DGX Spark generates at about 12 tokens/sec, slower than M3 Max (about 27 tokens/sec)" reproduce on real hardware?
3. Does the mechanism of offloading the KV cache to disk actually help in practice, for long-context or repeated prompts?

To give the conclusion upfront: the build went through with just make, and the 284B model fit in 128GB at about 81GiB. Generation runs at 8–15 tokens/sec, which is as slow as you'd expect from the memory bandwidth, but prefill (prompt ingestion) comes in fast at 80–250 tokens/sec incrementally. The disk KV cache reduced a ~32K-token prompt re-send from 115 seconds to 7.7 seconds. The result was very characteristic of the DGX Spark: "too slow for conversational chat, but realistic for batch processing of long documents."

How DwarfStar 4 and DeepSeek V4 Flash Work

The DeepSeek V4 Flash Model

DeepSeek V4 Flash is a MoE model that runs relatively lightly for its 284B scale, thanks to the small number of active parameters. According to antirez's analysis, even with long instructions, the thinking output tends to be shorter than other models (about 1/5 if you avoid maximum thinking).

The GGUF distributed by DwarfStar 4 is purpose-built — arbitrary DeepSeek/GGUF files won't work. The q2-imatrix used here is about 81GiB on disk. The quantization is asymmetric: only the expert networks, which make up the bulk of the model, are aggressively compressed down to 2-bit, while the core parts that matter for quality (attention layers, output layers, etc.) are kept at higher precision. A q4-imatrix (~153GB) is available for machines with 256GB or more, but on the 128GB DGX Spark, q2-imatrix is the only option.

The Idea of Putting the KV Cache on Disk

DeepSeek V4 Flash compresses its KV cache into a small representation using learned transformations during training, and varies the compression strength by layer (up to 1/128 for heavily compressed layers). On top of that, instead of attending to every past token each time, it uses a mechanism that focuses only on the most relevant subset (top 512 per token), which makes the KV extremely compact.

Measured on the DGX Spark, this comes to about 14KB per token. Even extrapolated to 1 million tokens, that's only on the order of tens of GiB, and including the pre-allocated context buffers, it fits within about 26GB (per the README's estimate, about 22GB of that is for the index used by the "relevant token attention" mechanism).

The README contains the line: The KV cache is actually a first-class disk citizen. The idea is: with KV this compressed and modern SSDs being as fast as they are, there's no reason the KV cache has to live in RAM. In practice, ds4-server hashes the prefix of a prompt and saves it as a <sha1>.kv file, so when the same prefix arrives later, it loads that checkpoint instead of re-running prefill from token 0. The file is written with ordinary read/write calls, avoiding the need to add extra memory mappings to a process that already has 81GiB mapped.

The reason llama.cpp integration is said to be difficult is that the DeepSeek V4-specific KV structure and the design that assumes disk KV persistence are fundamentally incompatible with how a general-purpose GGUF runner is built. You could say this is only possible because of the deliberate choice to be a "single-model-only engine."

Prefill Is Compute-Bound, Generation Is Memory-Bandwidth-Bound

This is a pattern that comes up every time the DGX Spark is discussed, and it applies here too. Prefill, which reads the prompt, can be computed in parallel batches, so the GPU's compute power is the bottleneck — and the GB10's computational performance matters here. Generation (decode), which produces one token at a time, requires loading the parameters from memory at every step, making memory bandwidth the bottleneck. The DGX Spark's memory bandwidth is 273 GB/s, which is lower than M3 Max (~400 GB/s) or M3 Ultra (~819 GB/s), so Apple Silicon should have the edge in generation — that's the prediction.

Let's see how things actually played out.

Test Environment

Since the CUDA support had only been merged a few days prior, I'm noting the commit hash explicitly.

Building and Running on DGX Spark

Built with a Single make

The setup was surprisingly straightforward.

git clone https://github.com/antirez/ds4 ~/works/dwarfstar4/ds4
cd ~/works/dwarfstar4/ds4
export PATH=/usr/local/cuda/bin:$PATH
make

Looking at the make internals, on Linux it automatically builds the CUDA backend (ds4_cuda.cu) and uses CUDA_ARCH ?= native to have nvcc target the visible GPU. With the aarch64 + CUDA 13.0 combination, no patches or environment variable tweaks were needed — in about 16 seconds, three binaries were produced: ds4 (CLI), ds4-server (HTTP API), and ds4-bench (benchmarking). -arch=native correctly picked up GB10 (sm_121), and there were no issues with Mac-specific code or dependency libraries.

The model is fetched using the included script. I selected q2-imatrix for a 128GB machine.

./download_model.sh q2-imatrix   # Downloads ~81GiB from Hugging Face → linked as ./ds4flash.gguf

Startup and Memory Usage

Let's try sending a short prompt once.

./ds4 -p "In one short paragraph, what is Redis?" --cuda --nothink -n 64

ds4: context buffers 751.71 MiB (ctx=32768, backend=cuda, prefill_chunk=2048, raw_kv_rows=2304, compressed_kv_rows=8194)
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121)
ds4: CUDA host registration skipped: operation not supported
ds4: CUDA loading model tensors into device cache
ds4: CUDA loading model tensors 16.02 GiB cached
...
ds4: CUDA startup model cache prepared 80.76 GiB of tensor spans in 26.432s
ds4: CUDA q8 fp16 cache budget exhausted; using q8 kernels (request=8.00 MiB cached=0.00 GiB free=3.54 GiB reserve=6.08 GiB total=121.69 GiB)
**Redis** is an open-source, in-memory data structure store that is commonly used as a high-speed cache, message broker, and database. ...
ds4: prefill: 17.29 t/s, generation: 14.81 t/s

The output is coherent text. Two things to note.

First, loading 80.76 GiB of weights into the "device cache" takes about 26 seconds on the first run. Once the file is in the page cache, subsequent runs shrink to about 10 seconds. Since the GB10 has unified memory, this is physically a copy within the same RAM.

Second, the line CUDA q8 fp16 cache budget exhausted; using q8 kernels. With 81GiB of weights + ~6GiB of reserved space + context buffers, only 3.5–6GiB of the 122GiB system memory is free — not enough room for the 16-bit expanded cache that would speed up computation, so it falls back to computing in 8-bit. The "128GB limit" shows up in the log the moment the process starts. The CUDA host registration skipped: operation not supported message is specific to the GB10 unified memory environment and doesn't affect operation.

Result 1: Context Length vs. Throughput

Using the included ds4-bench, you can measure prefill and generation throughput as context grows incrementally. It places checkpoints at 2048, 4096, … and at each point measures "prefill of just the newly added portion" and "generate 128 tokens using greedy decoding (no early stopping)."

./ds4-bench -m ds4flash.gguf --prompt-file speed-bench/promessi_sposi.txt \
  --ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128 \
  --csv results/dgx-spark-q2-sweep.csv

Appended to this is a coarser sweep from 65536 → 262144 doubling each step (64 generated tokens), giving the following graph.

Generation (red) starts around 13 tokens/sec and gradually drops to 8 tokens/sec at 260K tokens. Prefill (blue) steps up from a cold start of ~65 tokens/sec and stabilizes around 200 tokens/sec.

Generation throughput is about 13 tokens/sec at short contexts, dropping to about 8 tokens/sec at 260K tokens. This is consistent with the community-reported "~12 tokens/sec on DGX Spark," and matches the memory-bandwidth-bound prediction. The official benchmark table in the README lists DGX Spark GB10 (128GB, q2, 7047 tokens) as prefill 343.81 tokens/sec and generation 13.75 tokens/sec — generation is nearly the same, while my incremental prefill measurement (~200 tokens/sec) is somewhat lower. The README measures a single-shot prefill at ~7K tokens, while mine walks incrementally to long contexts, so this difference is attributed to measurement methodology.

The low prefill numbers at the start are likely because the expert network pages within the 81GiB of weights warm up gradually as more tokens are processed — the step-wise climb visible in the graph's warm-up region extends to around 30K tokens.

Peak memory at 260K tokens was about 115GiB (~7GiB free). According to ds4-bench logs, the context buffers alone consumed about 4.5GiB. The README also states "100K–300K tokens is realistic for 128GB machines," and 260K tokens feels like the practical upper limit for the DGX Spark.

Result 2: The Compressed KV Cache Really Is Small

Now let's actually verify the idea that "the KV cache is fine to put on disk." Looking at the rightmost column of the table above, kvcache_bytes grows linearly at about 14KB per token (slope ~13.8KB). Extrapolating to 1 million tokens:

Measured points form an almost perfect line. Even linearly extrapolated, 1 million tokens only yields about 12.8 GiB — easily fitting in 128GB alongside the 81GiB of weights.

Even at 260K tokens, the compressed KV is only 3.6GB. In other words, within the context range manageable on a 128GB DGX Spark, the KV fits entirely in RAM. The biggest insight from hands-on experience was this: disk KV caching is not "an overflow mechanism for when KV doesn't fit in RAM" — it's "a mechanism to reuse the expensive prefill cost paid once." Where TurboQuant/RotorQuant takes the "compress KV to make it smaller" approach, this takes a different angle: "compression + persist prefix to disk."

https://dev.classmethod.jp/articles/dgx-spark-turboquant-kv-cache/

So how much does that "reuse" actually help? I started ds4-server with disk KV caching enabled, sent a ~100KB (~32K token) text with a question, evicted the session with a small intermediate request, then sent the same large prompt again.

./ds4-server --ctx 200000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 24576 --port 8000

Cold (no cache): 31,930-token prefill + 24-token generation = 115.2 seconds. Warm (same prompt re-sent): disk KV cache hit, 7.7 seconds. About 15× speedup.

Looking at the trace, the warm run shows cache_source: disk-text with cached_tokens: 30720 — 30,720 of the 31,930 tokens (rounded down to the nearest multiple of 2048 for storage) were loaded from the disk checkpoint, and only the remaining ~1,200 tokens plus generation were rerun. In /tmp/ds4-kv/, several hundred MB of .kv files appear, roughly matching the KV size for ~30K tokens (~475MB) observed earlier.

This is exactly the scenario described in the README: "Claude Code often sends a large initial prompt of around 25K tokens before really getting started. With --kv-disk-dir enabled, after the first expensive prefill, subsequent continuations and session resumes can reuse that saved prefix." In practice, 115 seconds became 7.7 seconds.

Result 3: Is It Practical for Agent Use Cases?

Next, let's look at whether it's practical for agent use cases. Since ds4-server speaks both OpenAI-compatible and Anthropic-compatible APIs, the README even includes a wrapper script to point Claude Code at ds4-server. I measured what it actually feels like for a 284B-class model running locally at 12–15 tokens/sec, using long-context tasks.

First, as a baseline generation throughput test, I sent the prompt "Output the numbers from 1 to 120 separated by spaces" — 24-token prompt + 239-token generation took 16.1 seconds. That's about 14.9 tokens/sec for generation. For conversational chat, that's indeed frustratingly slow.

Next, two long-context batch tasks. The first was a code review: I concatenated DwarfStar 4's own C source files (ds4.h, ds4_gpu.h, ds4_bench.c, ds4_cli.c, and others) and asked it to "identify 5 bugs or dangerous assumptions." The second was a summarization: I concatenated 9 of my own published DGX Spark articles and asked it to "summarize the whole thing in one paragraph in Japanese, then list 5 recurring themes."

The code review output included observations about: fseek(SEEK_END) + ftell causing byte count mismatches in Windows text mode; fread not accounting for embedded NUL bytes in buffers; APIs that return raw pointers lacking a mechanism to prevent deallocation while a view is alive (use-after-free risk); and the bench loop potentially using uninitialized data when snapshot saving fails. These were genuinely sensible observations for a C code review (with filenames and line numbers included, though since line numbers weren't provided, some are estimates). Reading 35,088 tokens and responding took about 2 minutes 45 seconds.

For the summary, feeding 56,662 tokens — 9 serialized articles — produced a response starting with the title "DGX Spark's Performance Limits and the Advantages of MoE Models — Local AI Workflows Enabled by 128GB Unified Memory," correctly capturing: how the 273 GB/s memory bandwidth caps long-context generation; the affinity between MoE models with few active parameters and the hardware; discussions of lightweight techniques like LoRA SFT, MTP, and NVFP4 quantization; and local autocompletion with Continue.dev + Ollama. Honestly, it was a pretty accurate Japanese-language summary. It's genuinely interesting that a 2-bit quantized 284B model can summarize Japanese technical writing this well. Total time: about 5 minutes.

In other words, it's too slow for back-and-forth conversation, but for use cases involving "feeding a large input once and getting a comprehensive answer back in one shot" — code review, long-document summarization, RAG final-answer generation — it falls squarely in the DGX Spark's sweet spot where fast prefill matters. And with --kv-disk-dir, you don't have to pay the initial prefill cost again on subsequent calls, making it a good fit for agent use cases where a system prompt and tool definitions are prepended every time.

https://dev.classmethod.jp/articles/dgx-spark-claude-code-local/

Go/No-Go Matrix

Summarizing everything in one table:

Summary

• DwarfStar 4, just days after the CUDA support merged (commit a97e7a3, 2026-05-12), built cleanly on DGX Spark with just make. No issues with aarch64 + CUDA 13.0.
• The 284B DeepSeek V4 Flash, using q2-imatrix (~81GiB) which aggressively compresses only the expert networks to 2-bit, fits in 128GB. It uses 110–115GiB during operation, and the "128GB limit" shows up in logs from the very start.
• Generation runs at ~8–15 tokens/sec, capped by the 273 GB/s memory bandwidth — as expected, Apple Silicon (M3 Max gets ~40%, M3 Ultra gets ~30% more) has the edge here. Prefill, on the other hand, runs at 80–250 tokens/sec incrementally, competitive on the compute side.
• The compressed KV cache is extremely compact at ~14KB per token, and for the context sizes manageable on this machine, it fits entirely in RAM. Disk KV caching is not an overflow mechanism — it's a mechanism to reuse expensive prefill, and it provided ~15× speedup on a ~32K-token re-send.
• Claude Code and agent use cases: too slow for conversational chat, but realistic for batch long-context tasks (code review, summarization, RAG final answers). With --kv-disk-dir, repeated prefill of fixed prompts can also be avoided.

There's something satisfying about a "single-model-only engine" working this cleanly. When DeepSeek releases an updated version of V4 Flash, I'd like to try it again.

References

• DwarfStar 4 (formerly ds4 / ds4.c) repository: https://github.com/antirez/ds4
• DeepSeek V4 Flash dedicated GGUF: https://huggingface.co/antirez/deepseek-v4-gguf
• Verification scripts and raw data: https://github.com/himorishige/dgx-spark-blog (dwarfstar4-deepseek-v4-flash-bench/)

The Mac (M3 Max / M3 Ultra) figures were not measured in this verification and are taken from the benchmark table in the DwarfStar 4 README.