I thought about how to use NVIDIA NeMo Relay and NeMo Switchyard as an AI Gateway
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
When you hear the term "AI Gateway," you probably picture a product that consolidates access points to AI into one. In practice, however, this term covers a remarkably wide range of functions: model routing, user authentication, budget management, and auditing. Trying to set everything up at once inevitably feels too heavy for a typical coding agent environment. Whether you want to switch between models, log file operations and command executions, or limit usage per team — the components you need depend on the features you want.
Until now, I had been trying NeMo Relay and NeMo Switchyard separately (the Relay article is from 2026-08-06, and the Switchyard article from 2026-08-12). In the Relay article, I introduced version 0.6.0 as a runtime for recording and controlling coding agent execution; in the Switchyard article, I introduced v0.2.0 as a server responsible for routing to multiple models and converting API formats. Relay has since progressed to the 0.8 series, and this time I'll be using 0.8.1.
What I'm curious about here is how to use these two together in a typical AI development environment. Is it sufficient to use either one alone, and what does chaining them in series add? What can fill in the features neither of them covers, such as user authentication and budget management? When I thought it through, the conclusion wasn't so much "build an AI Gateway with Relay and Switchyard" as it was "add the missing features where needed using dedicated components."
In this article, for those who are new to NeMo Relay 0.8.1 and NeMo Switchyard v0.2.0, I'll first walk through the flow from a request to its arrival at a model. I'll then organize how to decide between using each alone versus chaining them, and outline a configuration proposal for supplementing missing features with external components. This is not an article about build commands — it's about deciding what to add to your everyday environment.
Getting a rough grasp of Relay and Switchyard
Let me start by translating the roles of these two tools into everyday language.
Switchyard is a switchboard for AI models. It receives calls from the coding agent using a common connection name like switchyard/auto, and selects the destination model based on the content and stage of progress. In Switchyard's configuration, this connection name is called a route. Converting among the three formats — OpenAI Chat, OpenAI Responses, and Anthropic Messages — to match the destination is also Switchyard's job.
Relay handles recording and control of coding agent work for supported agents. From the time a user's request is received, it records which LLM was called, which files were read, and which commands were executed. This entire sequence is recorded as a single execution. There are middlewares for inserting processing in between, which can also inspect or rewrite LLM and tool calls.
| Tool | In a nutshell | What it can answer | What it doesn't handle on its own |
|---|---|---|---|
| NeMo Switchyard | Switchboard for models | Which model was this request sent to? | File operations and commands executed by the coding agent |
| NeMo Relay | Recording and control layer for agent execution | How did LLMs and tools behave in this task? | Selecting a destination from multiple models |
| API Gateway etc. | Reception for team usage | Who can use it, and how much? | Agent execution records and content-based model routing |
Four terms to keep in mind: a provider is a model API supplier like OpenAI or Anthropic; a tool refers to file reads/writes and command executions performed by the coding agent. Think of a session as a single task, and a trace as identification information for tracking the communications that arose from that task — that's enough to follow what comes next.
Tracing the path of a single request to a model
When using both, you place Relay close to the coding agent, with Switchyard further along.
For example, suppose you ask the coding agent to "investigate the cause of the failing tests." The flow goes like this:
- Relay begins recording this as a single agent task
- When the coding agent reads a file, Relay records it as a tool execution
- When the coding agent calls an LLM, Relay passes the request to Switchyard
- Switchyard selects the destination from candidates such as the low-cost side and the high-performance side
- Switchyard records the selected model and token count, while Relay links the LLM call to the agent task
If only Switchyard is in place, model selection and usage recording from steps 3 through 5 are possible. However, the agent work and tool executions from steps 1 and 2 are not visible. If only Relay is in place, it can record through the LLM call in addition to steps 1 and 2, but the request destination is fixed separately for OpenAI-type and Anthropic-type, with no per-model routing.
In other words, there's no need to set up both from the start.
Choosing from three configurations based on the features you want
For a typical environment, you can start by choosing from these three:
| Configuration | Suited for | What you gain concretely |
|---|---|---|
| Switchyard alone | Consolidating to a single access point and switching between multiple models | API format conversion, model selection, retry, model and token records |
| Relay alone | Recording and controlling coding agent activity without changing model selection | Agent, LLM, and tool records; middleware insertion; external output of records |
| local Relay → Switchyard | Checking both tool execution before/after and the actually selected model along the same request path | Both sets of features above — though identification info is not automatically unified |
The value of using both becomes apparent when you want to see "which model did it send to after reading the test" or "did the model switch mid-task" alongside the agent's activity. Personally, I think it's more practical to start with whichever side you're struggling with, rather than running both from the beginning.
Verifying basic operation with a series connection
Now for the main topic. I placed Relay 0.8.1 on a Mac mini, Switchyard v0.2.0 and a synthetic provider on a DGX Spark, and chained them in series via SSH tunnel. I used a synthetic provider to avoid needing real API keys, and to fix provider-side behavior such as returning 429 a specified number of times. I'm observing how requests and records are handled at each layer.
The main results are as follows:
| What was checked | Result |
|---|---|
| API format | OpenAI Chat, Responses, Anthropic Messages, and their respective streaming variants all responded from the same Switchyard route |
| Tool call | Preserved in OpenAI Chat format. Tool calls in the other 2 formats were not verified |
| Recovery from transient errors | For the synthetic provider's 429 → 429 → 200, a single call recovered in 3 attempts |
| Provider API key | The API key configured in Switchyard was sent to the provider |
| Relay task recording | From 3 LLM calls, 10 event-format records (ATOF) and a 6-step task timeline record (ATIF) were generated |
| PII handling | Synthetic email was removed from Relay's records, but remained in the body received by the provider |
The most striking finding was the difference between attempts and calls. Including Switchyard's retries, 14 provider attempts were made for 12 completed calls. When reviewing usage, it's important to count separately the number of calls from the user's perspective and the number of attempts to the provider including retries.
This confirmed that API format conversion, routing, retry, and Relay's task recording can all be placed in the same request path. On the other hand, there is also information that doesn't line up just by chaining the two.
Four discrepancies to keep in mind when combining them
Before getting into specific header names, here's a table of what happens in practice.
| Point of concern | What happened with series connection | How to supplement |
|---|---|---|
| Task identifier | Relay's session ID is visible in logs, but does not connect to session aggregation in Switchyard v0.2.0 | Copy it to the name that Switchyard reads using a header adapter |
| Communication tracing | Relay becomes the origin of a new trace and groups the LLM calls within it | Aggregate to an OpenTelemetry Collector with Relay as the origin |
| API key | Replaced with Switchyard's provider API key | Add user authentication upstream of Switchyard |
| PII | Removed from Relay's observability data, but remains in the body sent to the provider | Add a DLP or sanitize middleware on the Relay side before sending |
The task identifier is like a tracking number attached to a package. If Relay and Switchyard are looking at different fields, even if the same string remains in the request log, it won't be used for Switchyard's session aggregation. In v0.2.0, I confirmed that passing the same ID under the name proxy_x_session_id will include it in session aggregation. Connecting them requires placing a header adapter in between that copies Relay's x-nemo-relay-session-id to this name, but I did not build the adapter itself this time.
With Switchyard alone, the input traceparent was delivered unchanged to the provider. When Relay is placed upstream, Relay becomes the origin of a new trace, and the 3 LLM calls in this case are grouped within it. It's worth noting that this is not a configuration where the trace from before Relay passes through transparently.
The API key set in Switchyard for the provider was sent to the provider. While this means the coding agent doesn't need to hold the provider key directly, "who called Switchyard" is not determined. If sharing, you'll need to add user authentication upstream.
PII is particularly easy to misunderstand. What Relay's built-in PII redaction masks is the data sent to external observability platforms. It does not alter the prompt body sent to the provider. In this case as well, the synthetic email disappeared from Relay's records, while one instance remained intact in the body received by the provider. Data you want masked before sending must be handled by a separate middleware.
Supplement missing features with dedicated components
There's no need to delegate all the features that Relay and Switchyard don't cover to a single separate LLM Gateway. You can add only the necessary components from your existing API infrastructure or observability infrastructure. The components in this table are configuration proposals not included in this round of testing — actual testing confirmed only the behavior of Relay and Switchyard through the previous section.
| Desired feature | Component to add | Where to place it | What you gain |
|---|---|---|---|
| User authentication and per-destination authorization | API Gateway or ingress + Identity Provider | Between Relay and shared Switchyard | Determine who can call which destination |
| Per-user call rate limiting | Ingress rate limiter | Upstream of Switchyard | Limit call rate per user, team, or project |
| Usage and budget caps | Usage aggregation, usage ledger, and control rule service | Between API Gateway and Switchyard | Accumulate user ID, tokens, and cost; stop requests once limits are reached |
| Provider key storage and rotation | Secret manager such as AWS Secrets Manager or Vault | Injected as environment variables at Switchyard startup | Separate API keys from config files and clients |
| PII and secret removal before provider submission | Relay request middleware + DLP service such as Presidio | Between coding agent and Switchyard | Inspect and process prompts before they leave the terminal or trust boundary |
| Observability aggregation | OpenTelemetry Collector + trace, metrics, and logs backend | Receive Relay's OTLP and scrape Switchyard | Search agent task records and model routing records in one place |
| Long-term auditing | Storage destination for ATOF and ATIF + object storage | Relay's observability output | Retain event records and task timeline records according to retention policy |
| Session connection | Header adapter via reverse proxy or Relay plugin | Between Relay and Switchyard v0.2.0 | Search Relay's task records and Switchyard's model usage under the same ID |
| Restriction of provider destinations | Limiting targets registered in Switchyard + egress firewall | Between Switchyard and the network exit | Prevent communication to any provider other than those permitted |
Usage and budget recording alone does not constitute a limit. Authenticated user IDs, token counts returned by Switchyard, and model selection records are linked in a usage aggregator and accumulated in a usage ledger. If the API Gateway side checks the control rules before accepting the next request, it can stop usage that exceeds the limit.
Switchyard v0.2.0 reads provider keys from the environment variable specified by api_key_env. When combining with a secret manager, beyond how to inject values as environment variables, you'll also want to include in your operations how to update and restart Switchyard when a key is rotated.
If you already have an AI Gateway, don't duplicate roles
If you already have an AI Gateway with per-user virtual API keys, budget caps, and call rate limiting, you can keep using it as-is. If the existing gateway's model routing meets your requirements, a configuration of local Relay → existing AI Gateway → providers — without adding Switchyard — will be smaller.
When you want to use Switchyard's content-based model classification or model selection based on agent progress stage, use local Relay → existing AI Gateway → Switchyard → providers. In this case, the existing gateway handles user authentication and usage control, while Switchyard's role is narrowed to model selection. Having two layers each rewriting the model makes it harder to trace the final selection rationale.
Scaling up the configuration as usage grows
Including the additional components, a team-oriented configuration looks like this:
For personal use, start with one pain point
If you're only using it on your own machine, a single instance of either Switchyard or Relay will often suffice. The approach is to run whichever side is bothering you on loopback, then chain them in series once you need information from both.
For team use, place a reception layer upstream of Switchyard
When sharing Switchyard across a team, place an API Gateway or ingress upstream and integrate it with an Identity Provider. Use the user's ID as a common key for rate limits, budget, and audit records. Pass provider keys from a secret manager to Switchyard, and don't distribute them to clients.
Keep Relay on each developer's machine. Even if you place a single shared Relay, the tool executions captured via each coding agent's hooks won't aggregate there. The split becomes: execution is local, model routing and usage control are shared, and the observability backend is centralized.
The convergence point for observability is the OpenTelemetry Collector. It receives Relay's OTLP and Switchyard's Prometheus metrics here, but as seen in the previous section, trace origins and session headers don't align automatically. Add user ID and project ID as metadata and make the search dimensions explicit.
For sensitive data, inspect before leaving the terminal
If prompts may contain personal or confidential information, call a DLP service like Presidio from a local Relay request middleware, and pass to Switchyard only after detected values have been deleted or replaced. The placement is to inspect before leaving the trust boundary, not to remove it after it reaches the API Gateway or Switchyard. Keep Relay's PII redaction for observability purposes, using the two for separate goals.
Even with this configuration, false positives and missed detections from the DLP, as well as prompt quality after processing, require separate evaluation. The ability to place middleware in Relay and the accuracy of the DLP you choose should be considered independently.
Beyond the current version, a shape other than two-stage proxy is visible
What I verified this time was a configuration with Relay 0.8.1 and Switchyard v0.2.0 running as separate servers in series. However, looking at the direction of the latest versions, it seems this two-stage configuration may not remain the only way to combine them.
In Relay 0.8, the previously built-in functionality for connecting to the Switchyard server was removed. Relay's release notes state that Switchyard 0.3.0 is planned to distribute and document a dynamic plugin managed on the Switchyard side. The direction is to run Switchyard's model selection within Relay's process. Relay itself is also advancing plugin and middleware extensibility in the 0.8 series.
Separating what is currently available from future directions:
| Category | Usage |
|---|---|
| Relay 0.8.1 and Switchyard v0.2.0 | Connect from Relay to the Switchyard server over HTTP |
| Switchyard main | Session header reorganization and caller API key forwarding are in progress |
| Guidance toward Switchyard 0.3.0 | A Switchyard-owned dynamic plugin running inside Relay is planned |
In Switchyard main, the regular session header has changed to x-switchyard-session-id, with proxy_x_session_id becoming a fallback. The forward_auth feature for passing the caller's API key to the provider is also a main-side feature. Please be careful not to treat either of these as features of v0.2.0.
As of 2026-08-29, Switchyard 0.3.0 has not yet been released. For now, it seems best to use the two-stage configuration while keeping the aforementioned header adapters and API Gateways as interchangeable components. Even if they are eventually unified into a plugin, the idea of separating model selection, agent execution, and user management will remain.
Summary
I've tried to organize how to place NeMo Relay and NeMo Switchyard in a typical environment as an AI Gateway.
Switchyard is a switchboard that receives requests from a common access point and selects the model; Relay is a layer that records and controls LLM and tool executions performed by the coding agent. If your goal is to switch between models, start with Switchyard; if you want to track coding agent activity, start with Relay; and chain them in series when you need both. In actual testing, I confirmed that three types of APIs with streaming, retry, API key substitution, Relay's event records, and task timeline records all work in the same request path. On the other hand, task identifiers and traces do not align automatically, and PII redaction is not masking before submission to the provider.
User authentication, usage and budget, secret management, data protection before submission, and observability aggregation are added as dedicated components in the right places, rather than crammed into Relay and Switchyard. This division allows you to incrementally add only the features you need, scaling from personal use to team use.
When Switchyard 0.3.0's dynamic plugin arrives, I'd like to try it again and see how it differs from the two-stage configuration.
Reference Links
- NeMo Relay Overview
- NeMo Relay Release Notes — Removal of Switchyard integration in 0.8 and announcement of 0.3.0 dynamic plugin
- NVIDIA/NeMo-Relay
- NVIDIA-NeMo/Switchyard v0.2.0 server README
- NVIDIA-NeMo/Switchyard main server README — Checking unreleased specs in main
- OpenTelemetry Collector
- Presidio — OSS for PII detection and anonymization, transferred from Microsoft to the Data Privacy Stack
- AI Gateway を自作して得た知見と選定ポイント — Referenced for the approach of listing requirements first and then assigning components
