I tried connecting two DGX Sparks together to test distributed inference of large-scale models

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Through the testing in my previous article, I confirmed that a single DGX Spark can run large-scale MoE models like Qwen3-235B, and I was also able to set up a code completion environment.

However, one thing I kept wondering about was: "If I connect two units, can I run even larger models? Will it be faster?" NVIDIA's official Connect Two Sparks Playbook also publishes instructions for clustering two units with a 200Gb/s QSFP direct connection.

To cut to the chase, connecting two units did not make things faster. However, since models of 143GB or 168GB that don't fit on a single unit run at practical speeds, being able to run larger models is the biggest benefit of a two-unit configuration.

In this article, I'll cover everything from the physical connection via QSFP cable to actual distributed inference measurement data, sharing what it's like to set up a two-unit configuration.

Test Environment

Hardware Configuration

This is a configuration where two DGX Sparks are directly connected with a QSFP112 DAC cable.

Combined, the two units have 256GB of memory. However, they are not physically unified; instead, llama.cpp's RPC protocol is used to distribute the model across the two units.

Software Stack

This time I'm using llama.cpp directly instead of Ollama. Since Ollama does not currently support RPC distributed inference, the configuration uses llama-bench to measure raw throughput.

Models Tested

The validation targets are 5 models including both Dense and MoE architectures, selected in stages ranging from those that run on a single unit to those that require two units.

All GGUF files used are from unsloth.

Environment Setup

Connecting the QSFP Cable

The back of the DGX Spark has a ConnectX-7 Smart NIC port, and the physical connection is complete simply by plugging in the QSFP112 DAC cable. Since it supports hot-plugging, I was able to do this without powering down.

Network Configuration

After connecting the cable, assign IP addresses using netplan. In my environment, the interface name was enp1s0f0np0.

network:
  version: 2
  ethernets:
    enp1s0f0np0:
      addresses:
        - 192.168.100.10/24  # Node A
      mtu: 1500

Set 192.168.100.11/24 on the Node B side.

# Apply the configuration
sudo netplan apply

# Verify connectivity
ping -c 3 192.168.100.11

Building llama.cpp

Build on both nodes with CUDA and RPC enabled.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build . --parallel $(nproc)

The build completed without any particular issues even in the ARM environment.

Starting RPC and Verifying Operation

The distributed inference flow is as follows. Start the RPC server on Node B, then from Node A, distribute and run the model across the two units via RPC.

# Node B: Start RPC server
CUDA_VISIBLE_DEVICES=0 ./build/bin/rpc-server \
  --host 192.168.100.11 -p 50052

# Node A: Start local RPC server
CUDA_VISIBLE_DEVICES=0 ./build/bin/rpc-server -p 50052 &

# Node A: Run benchmark (2-unit configuration)
./build/bin/llama-bench \
  -m ~/models/Devstral-2-123B-Q4_K_M/Q4_K_M/Devstral-2-123B-Instruct-2512-Q4_K_M-00001-of-00002.gguf \
  --rpc 192.168.100.11:50052,127.0.0.1:50052 \
  -ngl 99 -fa 1 \
  -p 128 -n 256 -r 5
# -ngl 99: Offload all layers to GPU
# -fa 1: Enable Flash Attention
# -p 128 -n 256: 128 prompt tokens, 256 generated tokens
# -r 5: Run 5 times and take the average

The model file only needs to be placed on Node A, and it will be streamed to the RPC server. Loading takes a little time, but once loaded, inference runs distributed automatically.

Network Bandwidth Measurement

The performance of distributed inference depends heavily on network bandwidth. I measured it with iperf3 before benchmarking.

These numbers are with MTU left at 1500. That's about 53% utilization against the theoretical 200Gb/s link, but since llama.cpp's RPC only sends and receives tensors on a per-layer basis, bandwidth is unlikely to be the bottleneck.

Benchmark Results

Now for the main topic. I used llama-bench to separately measure prefill (prompt processing) and decode (token generation). The tables below show the average and standard deviation of 5 runs each.

llama.cpp's RPC offers two model splitting methods. Layer split is pipeline parallelism that divides the model into front and back halves by layer, while row split is tensor parallelism that divides tensors along the row direction. In the tables below, a single unit is denoted S1, two units with layer split as D2-layer, and row split as D2-row.

Establishing a Baseline for Dense Models with Devstral 2 123B

Starting with the Dense model Devstral 2. At 75GB, it runs comfortably on a single unit. I measured it as a baseline to see whether two units would make it faster.

Unit is tok/s. pp = prefill, tg = decode (token generation).

Decode is exactly the same 2.64 tok/s for both 1 and 2 units. Token generation for Dense 123B is bottlenecked by memory bandwidth, and even when distributed to 2 units, the benefits of bandwidth appear to be cancelled out by the RPC communication overhead.

The only difference appeared in prefill for long prompts (pp2048), with a +25% improvement. This is a phase where increased computation allows the GPU's processing power to shine, so the two units' combined compute resources are taking effect.

However, 2.64 tok/s for decode is honestly quite slow, making it difficult to use interactively. The conclusion is that for running Dense 123B on DGX Spark, a single unit is sufficient in terms of speed.

Verifying the Distributed Effect of MoE Models with Qwen3-235B

Next is the MoE model Qwen3-235B. At 105GB, it barely runs on a single unit. This was personally the test I was most curious about — whether there would be a speed difference between 1 and 2 units.

Decode is slightly slower with 2 units.

MoE (Active 22B) achieves 15.5 tok/s even on a single unit, which is a sufficiently practical speed. With 2 units, the RPC latency adds a slight decrease. Prefill shows a +17% improvement for long prompts (pp2048), but the lack of benefit for decode was somewhat surprising.

With MoE, only the active parameters (22B) worth of memory is read during token generation, so the single unit's memory bandwidth is already sufficient. Distributing it doesn't change the bottleneck, and only the communication overhead increases.

Models That Only Run on 2 Units

This is where the two-unit configuration really comes into its own. Models that don't fit in 128GB simply cannot run on a single unit.

Llama 4 Maverick (143GB)

This is Meta's MoE model, notable for its 128 experts configuration.

An interesting finding emerged here. In decode, row split outperforms layer split by +19%. Maverick is a MoE with 128 experts, and with row split which divides tensors along the row direction, each expert's tensors are evenly distributed across the 2 units, seemingly improving efficiency in the decode phase.

With Qwen3-235B, layer split was better, so the optimal splitting method changes depending on the number of experts and model size.

Decode at 13.16 tok/s is at a level that can withstand interactive use, and I think it's a value unique to the two-unit configuration that a 143GB model runs at this speed.

Qwen3-Coder-480B (168GB)

This is a coding-specialized 480B MoE model.

Decode at 9.86 tok/s. It's slower than Maverick (17B active) due to the larger active parameters of 35B, but a 480B parameter model is running on two desktop units at nearly 10 tok/s.

When loading 168GB across the two units' 256GB, Node A's memory usage reached 116/119GB. It's probably best to consider the practical upper limit to be around 170GB.

DeepSeek-V3-0324 (204GB) — Abandoned Due to OOM

The most anticipated DeepSeek-V3-0324 (204GB) couldn't even be loaded. Even with KV cache quantization (q8_0), Node A froze. At this point, the practical upper limit appears to be around 170GB.

Let's verify how much of a constraint this upper limit is with the latest models. As of the time of writing, the much-talked-about GLM-5 is a 744B parameter, 40B active MoE, and with unsloth's Dynamic 1-bit quantization it's 176GB, or 241GB at 2-bit. Even at 1-bit, it's 8GB larger than Qwen3-Coder-480B (168GB), slightly exceeding the practical upper limit of a two-unit configuration.

https://unsloth.ai/docs/models/glm-5

The estimated decode speed based on Active 40B would be around 8–9 tok/s, which would be comparable to Qwen3-Coder even if it ran. While I'd love to try the latest large-scale MoE locally, the current DGX Spark 2-unit + llama.cpp RPC setup falls just short.

Comparing Decode Performance Across All Models

Let's line up the results so far by decode (tg256).

Decode speed is roughly proportional to the size of the active parameters. While Dense 123B Devstral 2 achieves 2.64 tok/s, MoE 22B active Qwen3-235B achieves 15.5 tok/s — about 6 times faster. On the DGX Spark, where the 273 GB/s memory bandwidth is the bottleneck, MoE models with small active parameters are clearly the better fit.

Choosing a Split Mode

Summarizing the results regarding layer split and row split mentioned at the beginning of the benchmark results, the following tendencies were observed.

The reason row won significantly for decode with Maverick is that by row-splitting the many expert tensors of the 128-expert configuration, the computation assigned to each node tends to be more evenly balanced. Since Qwen3-235B has only 16 experts, the benefit of tensor splitting is likely small.

There is no universally optimal splitting method, so I think the safest approach for large MoE models is to try both.

An Honest Assessment of $7,998

Two DGX Sparks at $7,998. Let me organize whether this investment is worthwhile by use case.

Cases Where Two Units Are Worth It

If you want to run models that don't fit on a single unit, this is the primary and essentially the only use case. Maverick (143GB) at decode 13 tok/s and Qwen3-Coder-480B (168GB) at decode 10 tok/s are at a level that can be used interactively, and being able to try large-scale models locally without using cloud APIs seems convenient for research and validation.

Cases Where Two Units Are Not Necessary

If the model is 105GB or less, a single unit is sufficient. Qwen3-235B achieves 15.5 tok/s on a single unit, and it won't be faster with 2 units. If you're going to invest an additional $3,999, it would be better value to put that toward cloud API usage fees instead.

The Upper Limit Wall

With the current state of llama.cpp RPC, the practical upper limit for model size is around 170GB. As demonstrated by the failure to load DeepSeek-V3 (204GB), "256GB combined across 2 units" doesn't mean all of it is usable. If you want to run DeepSeek-V3 class models, it's currently more realistic to consider an M3 Ultra Mac Studio (up to 512GB) or cloud A100/H100.

Summary

After testing 5 models with a two DGX Spark configuration, I think the deciding factor for buying a second unit comes down to "whether you absolutely need to run models that don't fit in 128GB locally." Maverick (143GB) running at decode 13 tok/s and Qwen3-Coder-480B (168GB) at decode 10 tok/s — at levels usable for interactive work — was a genuinely pleasing result. On the other hand, it's not suited for speeding up models that already run on a single unit. Otherwise, a hybrid configuration of 1 unit + cloud API would likely offer greater flexibility.

This time I tested with llama.cpp + RPC, but there's another option for DGX Spark: NVIDIA's own TensorRT-LLM + NVFP4 quantization. I'm curious how the two-unit results would change with NCCL-based tensor parallelism. Also, EXO Labs has reported a 2.8x improvement with a heterogeneous cluster combining DGX Spark + Apple Silicon, and I'd like to try a configuration combining DGX Spark's compute power with the Mac's bandwidth.

Reference Links

• DGX Spark Playbooks — Connect Two Sparks
• llama.cpp RPC documentation
• Intuition Labs DGX Spark review
• EXO Labs — NVIDIA DGX Spark + Apple Mac Studio = 4x Faster LLM Inference with EXO 1.0