I tried TurboQuant and RotorQuant on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

On March 24, 2026, Google Research announced TurboQuant (ICLR 2026). It is a method that compresses the KV cache (the memory area that retains past token information) accumulated during LLM inference to 3 bits, reducing memory usage by up to 6x. It doesn't reduce the VRAM of the model itself — it only applies to the cache portion during inference — but what stands out is that it requires no calibration, no fine-tuning, and can be retrofitted to any Transformer.

https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

In LLM inference, the longer the context, the more the KV cache strains memory. This is especially true when running large models, and having limited headroom for the KV cache is a common challenge across many environments. If this can be compressed to 1/6th, context length and the number of concurrent requests should improve considerably. I was eager to try it out on my own DGX Spark right away.

The day after the announcement, it became a major topic in the DGX Spark user forum, and just 2 days later, an alternative implementation called RotorQuant appeared. Using a mathematical theory from geometry called Clifford algebra, it achieves compression quality equivalent to TurboQuant with far fewer parameters.

This article presents the results of actually running TurboQuant and RotorQuant on the DGX Spark.

What is TurboQuant?

Why does the KV cache become a bottleneck?

In Transformer inference, the Keys and Values of past tokens are kept in a cache to avoid recomputation. Since the size of this cache grows proportionally with context length, the longer the text being processed, the more memory is consumed.

For example, processing a 32K token context with an 8B parameter model consumes several GB for the BF16 KV cache alone. Extending to 128K tokens means 4x that by simple calculation. Combined with the model's own memory footprint, even the 128GB on a DGX Spark quickly hits the limit.

An existing countermeasure is quantization to FP8, but this only halves the bit width, so the compression ratio is capped at 2x.

Two-stage compression: PolarQuant + QJL

TurboQuant combines two techniques. Simply put, the flow is "shuffle the data, then compress it, and correct the error with 1 bit."

First, PolarQuant (AISTATS 2026) randomly rotates Key vectors before quantizing them. The rotation removes information bias, enabling efficient encoding with fewer bits. Since there's no need to maintain scale values per block, there is virtually zero additional memory overhead.

Next, QJL (AAAI 2025) corrects the quantization error from PolarQuant using 1 bit. This is a mechanism to preserve the precision of inner products used in Attention computation, ensuring that the similarity between the original vectors can be correctly calculated even after quantization.

Regarding quality, even at 3.5-bit quantization, scores on LongBench and Needle-in-a-Haystack remain at a level that is statistically indistinguishable from FP32. The paper reports that distortion is within 2.7x of the information-theoretic compression limit.

Comparison with existing methods

What's great about this for DGX Spark?

Unified memory characteristics — "saves DGX twice"

The GB10 in the DGX Spark is a unified memory architecture where the CPU and GPU share LPDDR5x memory. Unlike discrete GPUs where VRAM and RAM are separate, compressing the KV cache frees up not only capacity but also memory bandwidth. This has been discussed on the NVIDIA Developer Forum with the expression "saves DGX twice."

https://forums.developer.nvidia.com/t/why-turboquant-saves-dgx-twice/364736/

The closer to the memory limit, the greater the benefit

One important note here is that TurboQuant does not apply to model weights. It does not enable models that don't fit in 128GB to be loaded. It only applies to the KV cache during inference.

Conversely, the closer a model is to the memory limit, the greater the benefit. Loading Nemotron 3 Super 120B-A12B with NVFP4 consumes approximately 87GB, leaving 41GB for the KV cache. Applying TurboQuant here means an effective KV cache capacity equivalent to 200GB can be secured in theory.

Smaller models already have plenty of headroom for the KV cache, so the relative benefit of compression is lower.

Affinity with MoE models

This situation becomes even more extreme with MoE (Mixture of Experts) models. MoE is an architecture that has many "expert" sub-networks, but uses only a subset of them per token during inference.

For example, Qwen3.5-35B-A3B has 35B total parameters, but only 3B active parameters are used for inference per token. With INT4 quantization, the weights fit in just a few GB, so most of the 128GB can be used for the KV cache. However, handling a 262K token context in FP16 means the KV cache alone can reach tens of GB, negating the memory efficiency advantage.

Compressing the KV cache 5x with TurboQuant / RotorQuant can theoretically resolve the imbalance in MoE models where "weights are light but the KV cache is heavy."

However, this is only a theoretical estimate. There is still little publicly available data on actually running the combination of MoE models + KV cache compression on DGX Spark, and I'd like to run actual measurements after full integration into vLLM.

Community activity

Within 3 days of the paper's publication, Feature Requests or PoCs have been submitted to all major inference frameworks: vLLM (Draft PR #38280), llama.cpp (Issue #20977), and MLX. In the DGX Spark community, active discussions continue on the forum, with reports of a +22% speed improvement over FP8 on GLM-4.7-Flash using a vLLM community fork.

On March 27, a Feature Request for RotorQuant (Issue #38291) was also filed with vLLM, and the multiquant branch of the same community fork already includes RotorQuant integration and a 2-bit version.

Amidst these developments, an alternative implementation called RotorQuant (scrya-com/rotorquant) has emerged. While TurboQuant uses a 128x128 rotation matrix (16,384 parameters), RotorQuant compresses this down to 356 parameters using Clifford algebra rotors while maintaining equivalent quality. Being implemented in Triton (a JIT compiler for GPUs), avoiding CUDA C++ build issues is a major practical advantage.

Running it on DGX Spark

Test environment

vLLM integration is still a work in progress

First, I attempted to build the vLLM community fork (turboquant branch) that was being discussed on the forum on the DGX Spark. Using the method of applying patches on top of eugr/spark-vllm-docker, vLLM itself started and inference ran, but KV cache compression was not actually engaged. A combination of factors — the custom Attention backend implementation still being under development, build system compatibility issues, and others — meant that as of March 27, full feature reproduction had not been achieved.

The fork developer reported on the forum a +22% speed improvement over FP8 and 2.3x KV cache compression on GLM-4.7-Flash, so the implementation itself is working. The Draft PR (#38280) to the upstream vLLM is also in progress, so waiting for official integration is the realistic approach.

RotorQuant runs with pip install

While struggling with the vLLM fork build, RotorQuant worked surprisingly easily.

git clone https://github.com/scrya-com/rotorquant
cd rotorquant
pip install -e .
pip install triton

That's all it takes inside the NGC PyTorch container. No CUDA C++ kernel build is required; Triton generates kernels with JIT.

Triton benchmarks — up to 188x speedup

First, I benchmarked how well RotorQuant's Triton kernels perform on the DGX Spark.

Triton's JIT compilation optimizes for the GB10 microarchitecture, delivering high performance while avoiding handwritten CUDA kernel compatibility issues.

TurboQuant vs RotorQuant — Compression quality comparison

Since the same repository contains implementations of both TurboQuant and RotorQuant, they can be directly compared on the GB10.

Compression quality is nearly equivalent, and both achieve a perfect score on Needle-in-a-Haystack. A notable feature of RotorQuant is its dramatically lower parameter count — 46x fewer (16,399 → 356).

TurboQuant CUDA kernel also confirmed working on GB10

The TurboQuant QJL CUDA kernel also successfully JIT-built inside the vllm-ng17e image (PyTorch 2.12, CUDA 13.0). It could not be built with NGC 25.03 (PyTorch 2.7) since SM121 was not supported, so a newer version of PyTorch is required.

Memory compression is an effective 7.5x, but the CUDA kernel speed is not much different from the PyTorch version. This is because the QJL kernel is not optimized for GB10's SM121, which is a known issue in the forum as well.

Running inference with actual models

Qwen2.5-3B — Needle-in-a-Haystack

I ran a Needle-in-a-Haystack test with Qwen2.5-3B-Instruct using RotorQuant's inference demo (poc_high_context.py).

The needle was correctly detected at all bit rates and all context lengths. Attention Fidelity (Cosine Similarity) at 3-bit is 0.991–0.995, which is almost indistinguishable from FP16.

I also tried Qwen3.5 and Gemma 3, but the internal KV cache implementations of these models were incompatible with RotorQuant's verification scripts, and correct results could not be obtained. The Qwen2.5 series with pure Transformer architecture works without issues.

Current limitations

Through the verification process, several limitations also became apparent.

First, TurboQuant / RotorQuant is specifically a technology for compressing the KV cache in Transformer Attention layers. For hybrid models that include Mamba layers, such as Nemotron 9B-v2-JP, it does not apply to the Mamba-side cache, and in this verification, compression effects could not be confirmed via either vLLM or RotorQuant. Pure Transformer models (such as the Qwen2.5 series and Gemma series) are the current targets.

In addition, full integration into vLLM and llama.cpp is still forthcoming. The official code is scheduled for Q2 2026 release (turboquant.net), and the vLLM Draft PR has just completed Phase 1 (algorithm validation). The community fork is working, but reproducing it at this stage takes some effort.

On the other hand, RotorQuant emerged just 2 days after TurboQuant was announced, and with voices in the forum asking "Why MultiQuant…", there seems to be a direction toward combining multiple quantization methods. The pace of development in this area is very fast, so I'd like to keep watching it.

Post-publication update (Added 2026-03-29)

Community activity has continued since the article was published, so I'll add a supplement including results I actually tested myself.

Verifying the RotorQuant aarch64 support PR

A PR #2 for native DGX Spark (aarch64) support was submitted to RotorQuant. Automatic GPU architecture detection was added to setup.py, and for the Blackwell generation (compute capability 12.x), it now builds with a PTX fallback.

I manually applied the changes from this PR along with a separately submitted PR #4 (bug fix after trivector storage removal) to main, and ran it in the NGC PyTorch 25.03 container.

=== verify_correctness ===
  Rotor sandwich:     Cosine sim 1.00000000  PASS
  Inverse sandwich:   Cosine sim 1.00000000  PASS
  Full fused pipeline: MSE 0.033747          PASS

=== benchmark_full_fused ===
  n_vecs    PyTorch (ms)     Triton (ms)     Speedup
   256        1.905            0.033          57.9x
  1024        1.986            0.031          63.3x
  4096        3.655            0.031         117.8x
 16384       36.996            0.197         187.6x
 65536      169.020            0.986         171.4x

The Triton benchmark shows nearly the same figures as at publication time (187.6x for Full Pipeline at 16K), confirming that compression quality is unaffected after the trivector removal.

Note that building the CUDA C++ kernel fails because _get_cuda_arch_flags in NGC 25.03 (PyTorch 2.7) does not recognize SM 12.1. This is the same constraint mentioned in the article body — "SM121 not supported in NGC 25.03" — and PyTorch 2.12 or later is required. Since the Triton path works fine, there is no practical impact on RotorQuant usage.

A DGX Spark-specific vLLM fork

I also found a vLLM fork called mitkox/vllm-turboquant. Reading the code, it is a TurboQuant integration fork that explicitly targets SM121 (DGX Spark), with processing in CacheConfig validation that raises an error if (major, minor) != (12, 1). It is literally DGX Spark-exclusive.

vllm serve <model> --kv-cache-dtype turboquant25 --enable-turboquant

It uses a hybrid Triton and FlashInfer configuration, taking a different approach from the flash7777/vllm introduced in the article. Two stages are available — 2.5-bit and 3.5-bit — with a design that controls per-layer quantization parameters via a metadata JSON. I wasn't able to confirm it working this time as a full vLLM build is required, but the overall design seems quite well thought out.

Progress on the upstream vLLM PR and llama.cpp

While active review is progressing on the upstream vLLM Draft PR #38280, bug reports are also coming in, such as errors with Qwen3.5-27B-FP8 and garbage output with Qwen2.5-7B. As throughput at Phase 1 is still around 0.6x compared to BF16, it will likely take a bit more time before it's ready for production use.

On the llama.cpp side, a CPU version tq3_0 implementation in ik_llama.cpp has been confirmed working. While this is a CPU rather than GPU implementation, technical discoveries have been made — such as the non-commutativity of RoPE and Hadamard transforms (20.8% RMSE deviation) — which are likely to lead to future improvements.

Summary

From testing TurboQuant and RotorQuant on DGX Spark, a few things became clear.

3-bit compression of the KV cache works not just in theory but in practice, passing rigorous tests like Needle-in-a-Haystack. RotorQuant's Triton kernels run on DGX Spark with just a pip install, demonstrating up to 188x speedup and more than 7.5x memory compression.

On the other hand, full integration into vLLM still seems to be some time away. Community forks are being actively developed, but reproducing them from scratch comes with a fair amount of overhead.

In DGX Spark's unified memory environment, KV cache compression applies to both capacity and bandwidth, making its impact feel even greater than on discrete GPUs. While waiting for official integration, it seems best to proceed with preliminary testing using tools like RotorQuant that can be tried with a simple pip install.

Reference Links

• TurboQuant Paper (arXiv:2504.19874)
• Google Research Blog: TurboQuant
• turboquant.net (Official project site)
• PolarQuant Paper (arXiv:2502.02617)
• QJL Paper (arXiv:2406.03482)
• QJL Official Code (GitHub)
• RotorQuant (GitHub)
• vLLM Issue #38171: TurboQuant Support
• NVIDIA Forum: Why TurboQuant saves DGX twice
• flash7777/vllm turboquant branch
• eugr/spark-vllm-docker
• llama.cpp Issue #20977: TurboQuant support
• vLLM Issue #38291: RotorQuant Support
• flash7777/vllm multiquant + RotorQuant release
• MLX TurboQuant (Blaizzy/mlx-vlm PR #858)