I tried building a team-shared RAG using NVIDIA RAG Blueprint × DGX Spark and connecting it with MCP

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.

Have you ever thought it would be great to have a shared knowledge base where you could consolidate your team's verification notes and published articles, and access them directly from Claude Code or opencode? NVIDIA's RAG Blueprint is a reference implementation that provides everything from document ingestion to search, generation, Web UI, and MCP server as a docker compose bundle — and it looks like exactly the right fit for this use case.

https://github.com/NVIDIA-AI-Blueprints/rag

My team is evaluating an opencode + NeMo Switchyard + open-weight model development environment, and as a candidate for our shared knowledge repository, we've been running this RAG Blueprint on our local DGX Spark machines to verify its suitability for team operation. Our policy is local self-containment. The generative LLM that produces answers is DeepSeek V4 Flash running on a DGX Spark, and the embedding and reranking that support search also run locally as NIM containers — a configuration that keeps documents and inference off external APIs.

To state the conclusion upfront: with a two-tier configuration of 3 DGX Sparks (1 for services + 2 for inference), everything works at a practical level — from document ingestion to MCP connections from Claude Code / opencode and Japanese queries. The combined resident memory for the RAG service and search NIMs is measured at approximately 22 GiB, and Japanese PDFs with tables can be correctly retrieved down to individual cell values. We're currently in the verification phase before opening it up to the team, but it handles simultaneous access from around 10 people without issues.

Note that I previously wrote a first-touch article about NeMo Switchyard, which we use for routing (published 2026-07-03).

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

This article won't go into detailed setup procedures or troubleshooting, but instead introduces the overall configuration, how it works, and how to actually use it. I hope it can serve as a blueprint for those who want to build a locally-oriented shared knowledge base for their team.

Overall Configuration of the Team Shared RAG

First, the big picture. We're running verification with a configuration that splits 3 DGX Sparks into two layers with different roles.

The service layer is a node that consolidates the entire RAG Blueprint suite (API server, ingestion pipeline, vector DB, Web UI) along with the embedding/rerank NIMs that support search, the extraction NIM group used during ingestion, the MCP server, and the Switchyard router. The inference layer consists of 2 machines running only vLLM, serving the generative LLM DeepSeek V4 Flash in parallel. The only thing that goes outside is the generation fallback (Fireworks AI) — document ingestion and search are entirely self-contained within the DGX Sparks.

There are two reasons for separating the layers. One is fault isolation: replacing or restarting vLLM models doesn't impact the RAG service or Web UI. The other is memory. Even with 128GB of unified memory, co-locating a 100B-class model's vLLM with the RAG stack leaves almost no free memory. By moving generation to dedicated nodes, the service layer becomes a lightweight node that runs on just CPU and I/O.

Access from team members is designed to remain within a private network such as Tailscale or Cloudflare Tunnel + WARP. With no inbound public ports exposed and both the Web UI and MCP placed inside the same boundary, you can operate fairly securely without building your own authentication infrastructure. For personal or small-scale verification, Tailscale is convenient, and for team deployment, Cloudflare Zero Trust (where cloudflared on the service layer establishes an outbound Tunnel, and members reach it via the WARP client) combined with corporate account authentication is a good fit. Whichever you choose, the RAG configuration itself doesn't change.

What Is the NVIDIA RAG Blueprint?

The NVIDIA RAG Blueprint is a reference implementation for enterprise-grade RAG pipelines. The version used this time is v2.6.0 (the latest tag at the time of writing), and its main components are as follows:

• rag-server: An API server that integrates search and generation (LLM can be swapped via OpenAI-compatible interface)
• ingestor-server + nv-ingest: Document ingestion pipeline (PDF layout analysis, table extraction, etc.)
• Milvus, Redis, SeaweedFS, etcd: Vector DB and its supporting infrastructure
• rag-frontend: Bundled Web UI
• examples/nvidia_rag_mcp: A thin adapter that exposes the RAG API as MCP tools

The strength of this stack is that all inference components — extraction, embedding, reranking, and generation — can be swapped via environment variables. You can run them locally as NIM containers, offload them to NVIDIA hosted APIs (build.nvidia.com endpoints), or point them to any OpenAI-compatible server. This wide flexibility in swap points became the foundation for re-configuring the setup for DGX Spark.

Agentic RAG is also bundled as a highlight of v2.6.0. Simply adding "agentic": true to a request switches to a plan-and-execute pipeline that decomposes questions into tasks, formulates a plan, repeats search and answer for each task, and then integrates the results. Locally, we confirmed that for compound questions like "what are the differences between these two tools and how can they be combined," it correctly pulls from each document and returns an integrated answer. That said, because it requires more LLM calls, responses take around 1 minute compared to a few seconds for standard mode. How we positioned this in our current configuration is covered in the MCP section.

Let me also mention DGX Spark-specific considerations. The core container images are officially distributed as amd64-only and don't run as-is on the arm64 DGX Spark, but all 4 core images built from source without any patches (and none of the dependency packages lack arm64 support either). The embedding/rerank and table-structure NIM containers have official arm64 builds and work as-is. The only exception is the OCR NIM — the current arm64 image includes x86_64 binaries and cannot start. Since the OCR engine itself (nemotron-ocr) is published on PyPI with arm64-compatible wheels, we're working around this by wrapping it in a thin NIM-compatible API server.

Trade-offs: What Runs Where

Even with 128GB of unified memory, directly porting the data center GPU configuration that Blueprint assumes is impractical. So we made deliberate trade-offs about where and when to run each function.

The logic is simple: only the heaviest component — generation — is separated onto the 2 inference layer machines, and everything else is consolidated onto a single service layer machine. Embedding and reranking are small (1B-class), and GPU activity per query is intermittent bursts of around 100ms, so keeping them always-on has almost no load beyond memory. The document extraction NIM group, on the other hand, has large images and consumes significant memory, so we run it only when needed via compose profiles. Since ingestion is a low-frequency event, this is perfectly manageable.

Search performance is backed by measured numbers. Embedding single-request p50 is 88ms, reranking p50 is 31ms. Unlike external APIs, there are no rate limits, so both ingestion and search can run at the speed the hardware allows.

Here's how memory works out. The RAG service suite uses approximately 12.7 GiB, and the embedding + rerank NIMs use approximately 9.4 GiB, for a combined resident total of approximately 22 GiB. This is just under 20% of a 128GB machine, and this figure barely moves under load. In tests that gradually applied 4 → 8 → 16 simultaneous MCP connections simulating 10-person team usage, all requests succeeded with zero errors, and host memory showed almost no change. It looks like there's little risk of the service layer hardware struggling first.

Generation Is Handled by Local DeepSeek V4 Flash

The wiring of the generative LLM is the key to this configuration. The LLM in the RAG Blueprint serves not just generation but four roles — query rewriting, filter expression generation, and summarization — each configurable via environment variables. Rather than connecting directly to individual servers, we inserted a single layer of the NeMo Switchyard router in between.

# Using the official env override pattern (last-source wins), routing all 4 LLM roles to the router
source deploy/compose/.env
source switchyard.env   # Set all SERVERURL values to Switchyard :4200

The Switchyard side has 2 routes. They are split into rag-main for generation and rag-light for lighter rewriting roles, allowing the destination for each to be tuned independently. The default destination is the inference layer's vLLM (local priority), and during inference layer maintenance, it falls back to Fireworks AI running the same V4 Flash. This is one of the benefits of open-weight models — since the same model runs both locally and in the cloud, model behavior doesn't change when falling back.

The intent of this wiring is portability. From the RAG side, the connection target is always just the Switchyard route name, so swapping the other side from vLLM to Fireworks or to a different model doesn't require changing a single character in the RAG configuration. In fact, adjustments like adding thinking token suppression settings to route defaults and applying them uniformly across all roles can be done entirely on the router side.

The V4 Flash on the inference layer runs on 2 DGX Sparks with vLLM in parallel (as documented in an article published 2026-06-30). The configuration achieves approximately 290 tok/s combined across 16 simultaneous requests, which is sufficient performance for the RAG generation role.

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-dspark/

Ingesting Documents

Moving on to how to use it. Ingestion has two paths — upload via Web UI and via API — and both go through the ingestor-server into the nv-ingest pipeline. PDFs are processed by combining text extraction (pdfium) with layout analysis and table structure NIMs.

The standout in ingestion quality is table handling. Tables in PDFs are structured as table elements including their layout, and when a query asks "list the specific values in the table," it correctly enumerates the actual cell values from both tables. Both a year-by-year data table and a package dependency version table answered correctly. Speed is also practical — 25 seconds for 2 PDFs with tables, and the feel is that something the size of one section of an internal wiki goes in within a few minutes. When higher-precision VLM-based extraction is needed, there's also a slot to plug in Nemotron Parse.

Japanese worked fine too. After ingesting 4 Japanese technical articles and a self-made PDF with 2 Japanese tables, both Japanese tables were extracted as table elements. Scanned PDFs can also be ingested via the aforementioned custom OCR serving. Testing with an image-only PDF with no text layer, the Japanese table answered correctly on cell value questions, and even from a heavily decorated A4 flyer, it correctly retrieved fraud tactics and a consultation hotline phone number. One caveat: documents without tables, like flyers, require enabling infographic extraction (APP_NVINGEST_EXTRACTINFOGRAPHICS=True). Without it, only table and chart regions are OCR'd, causing ingestion to fail with zero extracted elements.

Connecting from Claude Code and opencode via MCP

Here's the main topic. The Blueprint includes a thin adapter that exposes the RAG API as MCP tools, and setting it up with streamable_http transport allows remote connections from each team member's terminal. Since you can reach the service layer address directly through a private network like Tailscale or Cloudflare WARP, no authentication header configuration is needed on the MCP client side. Registration from Claude Code is done with a single command.

claude mcp add --transport http nvidia-rag http://<service layer IP>:8091/mcp

From opencode as well, you just add a single MCP server entry to the configuration file.

// opencode.jsonc
{
  "mcp": {
    "nvidia-rag": {
      "type": "remote",
      "url": "http://<service layer IP>:8091/mcp",
    },
  },
}

This gives the agent access to search (vector search + rerank) and generate (search + answer generation) as tools. The basic usage is search. Search returns cited chunks in 3–4 seconds, and the agent can process them in its own context. Even in tests with 16 simultaneous requests from all team members, all succeeded. The intended use case is to call generate only when you want the RAG side to compose the answer text.

We decided not to use Agentic RAG by default via this MCP path. Since Claude Code and opencode can run their own loops — decomposing questions themselves and calling search multiple times — having the RAG side also plan and decompose would duplicate the role and just add wait time. We see Agentic RAG as most valuable at single-question-single-answer entry points like Web UI or Slack.

Rather than exposing everything directly, we split the MCP server into 2 faces for the team. This is the key design decision in this setup.

Since the bundled adapter exposes all 12 tools by default — including destructive ones like collection deletion — the public face is limited to just 5 read-only tools (search, generate, summarization, collection list, document list). Another issue: since the default embedding is a VL model with image support, table chunk citations include base64 page images, which caused search responses to balloon to 507KB in some cases. This directly impacts the agent's context, so the public face includes a filter that replaces images with placeholders. This reduces responses to approximately 24KB (95% reduction). Since image citations are useful in the Web UI, the right place to strip them is in the MCP adapter layer rather than the server side.

For team distribution, we plan to bundle the opencode.jsonc configuration example and an Agent Skill summarizing how to use search vs. generate into the existing Switchyard bundle. If the agent reads this skill, it will know exactly what to do — from choosing collections to handling citations.

Using the Web UI

As an entry point for non-engineers, the bundled rag-frontend works as-is. Since it handles collection creation, document upload, and chat all in one browser session, it's the most accessible way to "just try it out."

Practical Quality Even in Japanese

If we're using this as a team, Japanese is the real test. We submitted 3 Japanese queries against the ingested Japanese corpus (4 technical articles + a self-made PDF with tables).

The third question asked to compare and contrast the unit prices of 2 model API services from a self-made PDF table ($0.09/$0.18 vs $0.14/$0.28), and not only did it cite the cell values, it also returned a ratio calculation of approximately 1.56x. Pre-ingestion embedding probes also showed strong cross-lingual alignment with cosine similarity of 0.910 for Japanese-English synonym sentence pairs, meaning Japanese queries can retrieve English documents as well.

With retrieval, extraction, and generation all functioning in Japanese, team use centered on Japanese documents is validated at a practical level based on measured results. The fact that generation uses the open-weight DeepSeek V4 Flash also seems to contribute to Japanese response quality.

Summary

We verified how practical a locally self-contained team shared RAG — covering search through generation — could be by running the NVIDIA RAG Blueprint on a two-tier DGX Spark configuration (1 service machine + 2 inference machines). To recap the key points: generation is handled by DeepSeek V4 Flash on the inference layer, embedding and reranking by NIMs on the service layer, with a combined resident footprint of approximately 22 GiB. Generation routing has a single Switchyard layer inserted for portability. The boundary is a private network via Tailscale or Cloudflare Tunnel + WARP with no public ports. Entry points are two paths — MCP (Claude Code / opencode) and Web UI — and Japanese works at a practical quality level based on measured results.

Next, we're looking to try turning this RAG into a team assistant accessible via natural language from Slack.

Reference Links

• NVIDIA-AI-Blueprints/rag (GitHub)
• NVIDIA RAG Blueprint Documentation
• MCP Server (RAG Blueprint Official Documentation)
• Nemotron Parse Extraction (RAG Blueprint Official Documentation)
• Full NVIDIA Enterprise RAG Blueprint on DGX Spark (ARM64) (Prior case from April 2026)