I thought about how to use NVIDIA NeMo Relay and NeMo Switchyard as an AI Gateway

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

When you hear the term "AI Gateway," you probably picture a product that consolidates connections to AI into a single entry point. In practice, the term encompasses quite a wide range of capabilities: model routing, user authentication, budget management, and auditing. Trying to set everything up at once tends to be too heavy for a typical coding agent environment. Whether you want to switch between models, record file operations and command executions, or limit usage per team — the components you need depend on which features you actually want.

https://github.com/NVIDIA/NeMo-Relay

https://github.com/NVIDIA-NeMo/Switchyard

Until now, NeMo Relay and NeMo Switchyard have each been tried individually (the Relay article is dated 2026-08-06, and the Switchyard article is dated 2026-08-12). The Relay article introduced version 0.6.0 as a runtime for recording and controlling coding agent execution, while the Switchyard article introduced v0.2.0 as a server responsible for routing to multiple models and converting API formats. Relay has since progressed to the 0.8.x series, and this time we'll be using 0.8.1.

https://dev.classmethod.jp/articles/nvidia-nemo-relay-first-touch/

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-v020-rust-first-touch/

What's worth thinking about here is how to use these two tools together in a day-to-day AI development environment. Should you deploy just one, or does chaining them together add something meaningful? And for features neither of them covers — like user authentication or budget management — what else do you need? When I sorted this out, the conclusion wasn't really "build an AI Gateway with Relay and Switchyard," but rather "add the missing pieces using dedicated components where they're needed."

This article introduces how a single request travels to a model, written so that even readers new to NeMo Relay 0.8.1 and NeMo Switchyard v0.2.0 can follow along. It then organizes how to choose between using each tool individually versus chaining them, and how to supplement missing features with external components. This is not an article about build commands — it's about deciding what to add to your everyday environment.

Getting a Quick Grasp of Relay and Switchyard

Let me start by translating the roles of these two tools into everyday language.

Switchyard is a switchboard for AI models. It receives calls from a coding agent using a common connection name like switchyard/auto, and selects the destination model based on the content and stage of the request. In Switchyard's configuration, this connection name is called a route. Converting between the three formats — OpenAI Chat, OpenAI Responses, and Anthropic Messages — and adapting them to the destination is also Switchyard's job.

Relay handles recording and controlling the work of supported coding agents. From the moment a user submits a request, it records which LLM was called, which files were read, and which commands were executed — capturing this entire sequence as a single execution. There is middleware that can insert processing at intermediate steps, allowing LLM and tool calls to be inspected or rewritten.

Just four terms to keep in mind: a provider is a source of model APIs like OpenAI or Anthropic; a tool refers to file reads, writes, and command executions performed by a coding agent; a session is a single task; and a trace is the identifying information used to track communications generated by that task.

Watching a Single Request Travel to a Model

When using both tools, Relay is placed close to the coding agent, with Switchyard placed further along.

Say you ask a coding agent, "Find the cause of the failing tests." The flow goes like this:

1. Relay begins recording this as a single agent task
2. When the coding agent reads a file, Relay records it as a tool execution
3. When the coding agent calls an LLM, Relay passes the request to Switchyard
4. Switchyard selects a destination from candidates such as a low-cost option and a high-performance option
5. Switchyard records the selected model and token count, while Relay links the LLM call to the agent task

With only Switchyard in place, steps 3 through 5 — model selection and usage recording — are still possible. However, the agent task and tool executions from steps 1 and 2 are not visible. With only Relay in place, steps 1 and 2 plus LLM calls are recorded, but the destination is fixed separately for OpenAI-type and Anthropic-type requests, and there is no per-model routing.

In other words, there is no need to deploy both from the start.

Choosing Among Three Configurations Based on the Features You Need

For a typical environment, you can start by choosing from these three options:

The value of deploying both emerges when you want to see things like "which model was used after reading the tests" or "did the model switch mid-task" together with the agent's behavior. Personally, I think it's more practical to start with whichever side you're currently struggling with, rather than running both from the beginning.

Confirming Basic Behavior by Chaining Them Together

Here is the main part. I placed Relay 0.8.1 on a Mac mini, Switchyard v0.2.0 and a synthetic provider on a DGX Spark, and connected them in series over an SSH tunnel. I used a synthetic provider to avoid needing real API keys and to fix the provider's behavior — for example, returning 429 a specific number of times. The focus was on how requests and records are handled at each layer.

The main results were as follows:

The most notable finding was the difference between attempts and calls. Including Switchyard retries, there were 14 provider attempts for 12 completed calls. When reviewing usage, it's important to count separately: the number of calls from the user's perspective, and the number of provider attempts including retries.

This confirmed that API format conversion, routing, retry, and Relay task recording can all be placed on the same request path. At the same time, there is also information that is not automatically aligned just by chaining the two together.

Four Misalignments to Address When Combining Them

Before getting into specific header names, here is a table of what happens in practice:

A task identifier is like a tracking number on a parcel. If Relay and Switchyard are looking at different fields, even if the same string appears in the request log, it won't be used in Switchyard's session aggregation. In v0.2.0, I confirmed that passing the same ID as proxy_x_session_id will include it in session aggregation. A header adapter placed between the two to copy Relay's x-nemo-relay-session-id to this name would bridge the gap — though I did not build the adapter itself this time.

With Switchyard alone, an input traceparent passed through to the provider as-is. With Relay placed upstream, Relay becomes the origin of a new trace, and the 3 LLM calls in this test were grouped within it. It's worth noting that this is not a setup where the trace from before Relay passes through transparently.

The API key configured in Switchyard for the provider was what got sent to the provider. This means the coding agent doesn't need to hold the provider key directly — but it also means "who called Switchyard" is not verified. For shared use, user authentication needs to be added upstream.

PII is particularly easy to misunderstand. What Relay's built-in PII redaction masks is the data sent to external observability systems. It does not alter the prompt body sent to the provider. In this test as well, the synthetic email disappeared from Relay's records but remained intact in the body received by the provider. Data that needs to be masked before transmission must be handled by a separate middleware.

Supplementing Missing Features with Dedicated Components

There is no need to hand off every feature that Relay and Switchyard don't cover to a single separate LLM Gateway. You can add only the pieces you need from your existing API and observability infrastructure. The components in this table are configuration proposals not included in my actual measurements — the only things verified by measurement are the Relay and Switchyard behaviors described in the preceding sections.

Usage and budget controls don't become actual limits just by recording. You link authenticated user IDs with token counts returned by Switchyard and model selection records in a usage aggregator, then accumulate them in a usage ledger. If the API Gateway checks a control rule before accepting the next request, it can stop usage that exceeds the cap.

Switchyard v0.2.0 reads provider keys from the environment variable specified by api_key_env. When combining with a secret manager, you'll want to plan not only how to inject values as environment variables, but also how to update and restart Switchyard when a key is rotated.

If You Already Have an AI Gateway, Don't Overlap Roles

If you already have an AI Gateway that provides per-user virtual API keys, budget caps, and rate limiting, you can keep using it as-is. If its model routing meets your requirements, a configuration of local Relay → existing AI Gateway → providers — without adding Switchyard — will be simpler.

When you want to use Switchyard's content-based model classification or model selection by agent stage, the configuration becomes local Relay → existing AI Gateway → Switchyard → providers. In this case, the existing Gateway is scoped to user authentication and usage control, and Switchyard is scoped to model selection. This division matters because if two layers are both rewriting the model, it becomes difficult to trace the final selection back to a clear reason.

Scaling the Configuration as Usage Grows

Including the additional components, a team-oriented configuration looks like this:

For Personal Use, Start with One Pain Point

If you're working on your own machine, a single Switchyard or Relay is likely enough. Start with whichever one addresses your current problem, running it on loopback, and chain them together when you find you need both.

For Team Use, Place a Reception Point in Front of Switchyard

When sharing Switchyard across a team, place an API Gateway or ingress upstream and connect it to an Identity Provider. Use each user's ID as the common key for rate limiting, budget tracking, and audit records. Provider keys are passed from the secret manager to Switchyard and are not distributed to clients.

Relay stays on each developer's local machine. Even if you ran a single shared Relay, tool executions captured by each coding agent's hooks wouldn't converge there. The division becomes: execution is local, model routing and usage control are shared, and the observability backend is centralized.

The convergence point for observability is the OpenTelemetry Collector. It receives Relay's OTLP output and scrapes Switchyard's Prometheus metrics — but as noted earlier, trace origins and session headers are not automatically aligned. Add user IDs and project IDs as metadata to make the search axes explicit.

For Sensitive Data, Inspect Before It Leaves the Device

If prompts may contain personal or confidential information, call a DLP service like Presidio from Relay's request middleware, detect the relevant values, remove or replace them, and then pass the request to Switchyard. Rather than sanitizing after the data reaches the API Gateway or Switchyard, the goal is to inspect before it crosses the trust boundary. Relay's PII redaction remains in place for observability purposes, with the two serving different roles.

Even with this configuration, false positives and false negatives from the DLP, and the quality of the processed prompt, require separate evaluation. The ability to add middleware to Relay and the accuracy of a given DLP solution should be considered independently.

Beyond the Current Version, a Form Other Than Two-Stage Proxy Is on the Horizon

What was verified this time is a configuration where Relay 0.8.1 and Switchyard v0.2.0 run as separate servers in series. However, looking at how the latest versions are evolving, this two-stage configuration may not remain the only way to compose them.

In Relay 0.8, the built-in feature for connecting to a Switchyard server was removed. The Relay release notes state that Switchyard 0.3.0 is planned to distribute and document a dynamic plugin managed on the Switchyard side. This points toward running Switchyard's model selection within Relay's process. Relay itself is also advancing plugin and middleware extensions in the 0.8.x series.

Separating what is available today from the direction things are heading:

In Switchyard main, the standard session header has changed to x-switchyard-session-id, and proxy_x_session_id is now a fallback. The forward_auth feature for passing the caller's API key to the provider is also a main-branch capability. Be careful not to treat either of these as features of v0.2.0.

As of August 29, 2026, Switchyard 0.3.0 has not been released. For now, using the two-stage configuration while keeping the header adapter and API Gateway described above as swappable components seems like the right approach. Even if things are eventually unified into a plugin, the idea of separating model selection, agent execution, and user management will remain.

Summary

I've organized how to place NeMo Relay and NeMo Switchyard in a day-to-day environment as an AI Gateway.

Switchyard is a switchboard that receives requests from a common entry point and selects a model; Relay is a layer that records and controls LLM and tool executions carried out by a coding agent. If your goal is to use different models, start with Switchyard. If you want to track coding agent activity, start with Relay. Chain them together when you need both. In my measurements, I confirmed that three API formats and their streaming variants, retry, API key substitution, Relay event records, and chronological task records all work on the same request path. On the other hand, task identifiers and traces are not automatically aligned, and PII redaction does not mask data before it's sent to the provider.

User authentication, usage and budgets, secret management, pre-transmission data protection, and observability aggregation should not be crammed into Relay and Switchyard — instead, add dedicated components where they're needed. This approach lets you grow from personal use to team use by adding only the features you need, one at a time.

When Switchyard 0.3.0's dynamic plugin arrives, I'd like to revisit the differences from the two-stage configuration.

Reference Links

• NeMo Relay Overview
• NeMo Relay Release Notes — Removal of Switchyard integration in 0.8 and announcement of 0.3.0 dynamic plugin
• NVIDIA/NeMo-Relay
• NVIDIA-NeMo/Switchyard v0.2.0 server README
• NVIDIA-NeMo/Switchyard main server README — Checking unreleased specs on main
• OpenTelemetry Collector
• Presidio — PII detection and anonymization OSS, transferred from Microsoft to the Data Privacy Stack
• AI Gateway を自作して得た知見と選定ポイント — Referenced for the approach of listing requirements first and then assigning components