I built a team-shared RAG using NVIDIA RAG Blueprint × DGX Spark and connected it with MCP

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Have you ever thought it would be great to have a shared knowledge base that consolidates your team's verification notes and published articles in one place, accessible directly from Claude Code or opencode? NVIDIA's publicly available RAG Blueprint is a reference implementation that provides everything from document ingestion to search, generation, Web UI, and MCP server as a single docker compose setup — and it seems like exactly the right fit for this use case.

https://github.com/NVIDIA-AI-Blueprints/rag

My team is evaluating an opencode + NeMo Switchyard + open-weight model development environment, and we're using this RAG Blueprint on our local DGX Spark as a candidate for our shared knowledge repository, progressing through validation toward team-wide operation. The policy is local self-containment. The generative LLM that produces answers is DeepSeek V4 Flash running on the DGX Spark, and the embedding and reranking that support search also run locally as NIM containers, giving us a configuration where documents and inference never leave our environment.

To state the conclusion upfront: with a two-tier configuration of 3 DGX Sparks (1 for services + 2 for inference), everything from document ingestion to Claude Code / opencode MCP connections and Japanese queries is working at a practical level. The combined resident memory for the RAG service and search NIMs comes to approximately 22GiB as measured, and Japanese PDFs with tables can be retrieved correctly down to individual cell values. We're currently in the validation phase before opening it to the team, but it handles simultaneous access from around 10 people without any issues.

Note that I previously wrote a first-touch article about NeMo Switchyard, which we're using for routing (published 2026-07-03).

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

This article won't go into detailed construction steps or troubleshooting, but will instead introduce the overall configuration, how it works, and how to actually use it. I hope it can serve as a blueprint for anyone looking to build a locally-oriented shared knowledge base for their team.

Overall Configuration of the Team Shared RAG

First, the big picture. We're running our validation with 3 DGX Sparks divided into 2 layers with different roles.

The service layer is a node that consolidates the entire RAG Blueprint suite (API server, ingestion pipeline, vector DB, Web UI), along with the embedding/rerank NIMs that support search, the extraction NIM group used during ingestion, the MCP server, and the router Switchyard. The inference layer consists of 2 machines running only vLLM, delivering the generative LLM DeepSeek V4 Flash in parallel. The only thing that goes outside is the generation fallback destination (Fireworks AI) — document ingestion and search are fully contained within the DGX Sparks.

There are two reasons for separating the layers. The first is fault isolation: replacing or restarting vLLM models won't affect the RAG service or Web UI. The second is memory. Even with 128GB of unified memory, co-locating vLLM for a 100B-class model with the RAG stack leaves almost no free space. By offloading generation to dedicated nodes, the service layer becomes a lightweight node that runs on CPU and I/O alone.

Access from team members is designed to stay within a private network such as Tailscale or Cloudflare Tunnel + WARP. Since there are no inbound public ports, and both the Web UI and MCP are placed inside the same perimeter, operations can be kept relatively secure without having to build authentication infrastructure from scratch. For individuals or small-scale validation, Tailscale is convenient, while for team deployment, Cloudflare Zero Trust (where cloudflared on the service layer establishes an outbound Tunnel, and members connect via the WARP client) is a clean fit when combined with corporate account authentication. Whichever you choose, the RAG-side configuration remains unchanged.

What is NVIDIA RAG Blueprint?

NVIDIA RAG Blueprint is a reference implementation of an enterprise-grade RAG pipeline. The version used this time was v2.6.0 (the latest tag at the time of writing), and the main components are as follows:

• rag-server: An API server that consolidates search and generation (LLM is swappable via OpenAI-compatible interface)
• ingestor-server + nv-ingest: Document ingestion pipeline (PDF layout analysis, table extraction, etc.)
• Milvus, Redis, SeaweedFS, etcd: Vector DB and its supporting infrastructure
• rag-frontend: Bundled Web UI
• examples/nvidia_rag_mcp: A thin adapter that exposes the RAG API as MCP tools

The great thing about this stack is that all inference components — extraction, embedding, reranking, and generation — can be swapped out via environment variables. You can run them in local NIM containers, route them to NVIDIA hosted APIs (build.nvidia.com endpoints), or point them to any OpenAI-compatible server. This flexibility in swapping components was the foundation for reconfiguring the setup for DGX Spark.

Agentic RAG is also bundled as a highlight of v2.6.0. Simply adding "agentic": true to a request switches to a plan-and-execute pipeline that decomposes the question into tasks, creates a plan, repeats search and answering for each task, and then synthesizes the results. I was able to confirm locally that for composite questions like "what are the differences between these two tools and how can they be combined," it returns integrated answers correctly retrieved from each respective document. The tradeoff is that LLM calls increase, so response time is around 1 minute compared to a few seconds for standard mode. How this is positioned in our current configuration is discussed in the MCP section.

I should also mention DGX Spark-specific considerations. The core container images are officially distributed as amd64 only and won't run as-is on the arm64 DGX Spark, but all 4 core images built from source without any patches (and none of the dependency packages were incompatible with arm64). The embedding/rerank and table-structure NIM containers have official arm64 builds available and work out of the box. The one exception is the OCR NIM: the current arm64 image ships with x86_64 binaries and cannot start. Since the OCR engine itself (nemotron-ocr) is published on PyPI with arm64-compatible wheels, we worked around this by wrapping it in a thin NIM-compatible API server.

Decisions on What to Run Where

Even with 128GB of unified memory, the DGX Spark can't simply absorb the datacenter GPU configuration that the Blueprint assumes. So we made deliberate decisions about where and when to run each function.

The thinking is simple: only the heaviest workload — generation — is separated onto the 2 inference-layer machines, and everything else is consolidated on the single service-layer machine. Embedding and reranking are small (1B-class), and the per-query GPU activity is intermittent bursts of around 100ms, so keeping them always-on creates virtually no load beyond memory consumption. On the other hand, the document extraction NIM group has large images and is memory-hungry, so we operate it using compose profiles, starting it only during ingestion. Since ingestion is a low-frequency event, this is no inconvenience.

Search performance has been verified with actual measurements. Embedding single-request p50 is 88ms, reranking p50 is 31ms. Unlike external APIs, there are no rate limits, so both ingestion and search can run at the speed the hardware allows.

Here's how memory worked out: the RAG service suite takes about 12.7GiB, and the embedding + rerank NIMs take about 9.4GiB, for a combined resident total of approximately 22GiB. This stays within roughly 20% of the 128GB machine, and the numbers barely move under load. In tests that progressively applied 4 → 8 → 16 simultaneous MCP connections simulating concurrent use by 10 team members, all requests succeeded with zero errors, and host memory remained almost unchanged. There seems to be no risk of the service layer hardware giving out first.

Generation is Handled by Local DeepSeek V4 Flash

The wiring of the generative LLM is the key to this configuration. The RAG Blueprint's LLM has not just one role but four — generation, query rewriting, filter expression generation, and summarization — and each can be redirected to a different endpoint via environment variables. Rather than connecting directly to individual servers, we inserted NeMo Switchyard as a routing layer in between.

# Official env override pattern (later source wins), routing only the 4 LLM roles to the router
source deploy/compose/.env
source switchyard.env   # All SERVERURL values point to Switchyard :4200

Switchyard has 2 routes. They're split into rag-main for generation and rag-light for the lighter rewriting roles, allowing the destination of each to be tuned independently. The default destination is the inference-layer vLLM (local priority), and during inference-layer maintenance it falls back to the same V4 Flash on Fireworks AI. This is one of the benefits of open-weight models: the same model runs both locally and in the cloud, so behavior doesn't change on fallback.

The intent behind this wiring is portability. Since the connection target visible from the RAG side is always just the Switchyard route name, swapping what's behind it — from vLLM to Fireworks, or to a different model — requires zero changes on the RAG side. In practice, adjustments like adding thinking-token suppression settings to the route defaults and applying them uniformly across all roles are also handled entirely on the router side.

The V4 Flash on the inference layer has been validated running in vLLM parallel across 2 DGX Sparks (published 2026-06-30). It delivers approximately 290 tok/s total across 16 simultaneous requests — more than sufficient performance for the RAG generation role.

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-dspark/

Ingesting Documents

Moving on to usage. Ingestion has two paths — upload via the Web UI and via the API — and both go through ingestor-server into the nv-ingest pipeline. PDFs are processed by combining text extraction (pdfium) with layout analysis and table structure NIM group.

The standout quality aspect of ingestion is table handling. Tables in PDFs are structured as table elements preserving their layout, and when queried with something like "list the specific values in the table," it correctly enumerates the actual cell values from both tables. Both yearly data tables and package dependency version tables answered correctly. Speed is also practical — 2 PDFs with tables processed in 25 seconds — and the feel is that something roughly the size of one section of an internal wiki can be ingested in a few minutes. When higher-accuracy VLM-based extraction is needed, there's also a hook for plugging in Nemotron Parse.

Japanese presented no problems either. After ingesting 4 Japanese technical articles and a self-created PDF with 2 Japanese tables, both Japanese tables were extracted as table elements. Scanned PDFs can also be ingested via the custom OCR serving described earlier. Testing with an image-only PDF without a text layer, the Japanese table correctly answered cell value questions, and even from a heavily decorated A4 flyer, it correctly retrieved things like scam tactics and consultation hotline phone numbers. One caveat: for documents that don't contain tables, like flyers, you need to enable infographic extraction (APP_NVINGEST_EXTRACTINFOGRAPHICS=True). By default, only table and chart regions are OCR'd, so ingestion fails with zero extracted elements.

Connecting from Claude Code and opencode via MCP

Now for the main topic. The Blueprint ships with a thin adapter that exposes the RAG API as MCP tools, and standing it up with streamable_http transport allows remote connections from each team member's machine. Since members can reach the service layer's address directly via a private network like Tailscale or Cloudflare WARP, no authentication headers need to be configured on the MCP client side. Registration from Claude Code is done with a single command.

claude mcp add --transport http nvidia-rag http://<service layer IP>:8091/mcp

From opencode as well, it's just a matter of adding one MCP server entry to the configuration file.

// opencode.jsonc
{
  "mcp": {
    "nvidia-rag": {
      "type": "remote",
      "url": "http://<service layer IP>:8091/mcp",
    },
  },
}

This gives agents access to search (vector search + rerank) and generate (search + answer generation) as tools. The recommended approach is to use search as the default. Search returns retrieved chunks in 3–4 seconds, which the agent can then process within its own context. Even in tests with all 16 team members simultaneously firing requests, all succeeded. The intended usage pattern is to call generate only when you want the RAG side to produce the full answer text.

We decided not to use the Agentic RAG introduced earlier by default via this MCP path. Since Claude Code and opencode can run the loop of decomposing questions and repeatedly calling search externally on their own, having the RAG side also do planning and decomposition creates a double layer, adding only latency. We think Agentic RAG is best suited for single-question-single-answer entry points like Web UI or Slack.

Rather than exposing it as-is, we split the MCP server for team use into two faces. This is the key innovation in our current setup.

Since the bundled adapter exposes all 12 tools by default including destructive ones like collection deletion, the public face is limited to just 5 read-only tools (search, generate, summarize, list collections, list documents). There was also another issue: the default embedding uses a VL model with image support, so table chunk citations include base64-encoded page images, and search responses in some cases ballooned to 507KB. Since this directly impacts agent context, we added a filter to the public face that replaces images with placeholders. This brought responses down to approximately 24KB (a 95% reduction). Since image citations are useful in the Web UI, the right place to strip them is at the MCP adapter layer rather than the server side.

For distribution to the team, we plan to bundle the configuration example for opencode.jsonc and an Agent Skill summarizing when to use search vs. generate into the existing Switchyard bundle. The idea is that if agents read this skill, they won't be lost on anything from choosing collections to handling citations.

Using via the Web UI

As an entry point for non-engineers, the bundled rag-frontend works out of the box. Since you can handle everything from collection creation to document upload to chat entirely in the browser, it's the easiest way to get people to "just try it out."

Japanese Also Worked at Practical Quality

For team use, Japanese is what really matters. I ran 3 Japanese queries against the ingested Japanese corpus (4 technical articles + a self-created PDF with tables).

The third question asked to compare and contrast the unit prices for 2 model API services from a table in a self-created PDF ($0.09/$0.18 vs. $0.14/$0.28), and it not only cited the cell values but also returned the approximately 1.56x ratio calculation. Pre-ingestion embedding probes also showed strong cross-lingual alignment, with synonym sentence pairs in Japanese and English scoring a cosine similarity of 0.910, indicating the system can retrieve English documents from Japanese queries.

Retrieval, extraction, and generation all function in Japanese, and team use centered on Japanese documents can be considered practically viable based on actual measurements. The use of the open-weight DeepSeek V4 Flash for generation also seems to be contributing to the quality of Japanese responses.

Summary

We validated how far a team-shared RAG with search and generation contained locally could go in practice, by deploying NVIDIA RAG Blueprint on a 2-tier DGX Spark configuration (1 service machine + 2 inference machines). To recap the key points: generation is handled by DeepSeek V4 Flash on the inference layer, embedding and reranking by NIMs on the service layer, and total resident memory is approximately 22GiB. Generation wiring has a Switchyard layer inserted for portability. The perimeter is a private network via Tailscale or Cloudflare Tunnel + WARP with no public ports. Entry points are 2 channels — MCP (Claude Code / opencode) and Web UI — and Japanese is at practical quality as measured.

Next, I'm thinking of trying to make this RAG accessible from Slack via natural language, turning it into a team assistant.

Reference Links

• NVIDIA-AI-Blueprints/rag (GitHub)
• NVIDIA RAG Blueprint Documentation
• MCP Server (RAG Blueprint Official Documentation)
• Nemotron Parse Extraction (RAG Blueprint Official Documentation)
• Full NVIDIA Enterprise RAG Blueprint on DGX Spark (ARM64) (Prior example from April 2026)