I tried NVIDIA's new LLM routing infrastructure NeMo Switchyard

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

I had just published two articles about running NVIDIA LLM Router on DGX Spark, when I was immediately told that "a new routing infrastructure has been released that incorporates the LLM Router algorithms." NeMo Switchyard v0.1.0 was released on July 1, 2026, Japan time.

As someone who was preparing follow-up articles, I have mixed feelings about this, but as I actually tried it out, I kept discovering that "features I had struggled to build myself for LLM Router were already included from the start." In this article, I'll run Switchyard on my Mac and DGX Spark, and verify whether the pitfalls I hit during LLM Router testing have truly been resolved.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3/

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3-training/

What Is NeMo Switchyard?

It's a routing proxy that distributes LLM traffic, published in the NVIDIA-NeMo organization on GitHub. It's a Python package installable via pip install nemo-switchyard, with a two-layer structure where Python wraps a Rust core built with maturin. The license is Apache 2.0, version 0.1.0, and the development status is explicitly listed as Alpha.

https://github.com/NVIDIA-NeMo/Switchyard

A documentation site is also available.

https://nvidia-nemo.github.io/Switchyard/

I had the opportunity to ask NVIDIA about the relationship with LLM Router, and received the response that it's not a simple successor but rather "a more formal product that encompasses various routing technologies and the infrastructure to host and improve them." The algorithms from the LLM Router Blueprint are currently being ported to Switchyard.

For those familiar with LLM Router, here's a table comparing their characteristics.

The two major differences are "no trained router" and "built-in protocol conversion." LLM Router was designed to train a custom classifier that passes Qwen embeddings through PCA and MLP. Switchyard replaces that with signals obtained from LLM queries and agent tool execution history. Since GPU is no longer needed, it runs as-is on a Mac.

Routing: Choose from 4 Methods

The documentation describes 4 routing methods.

llm-routing summarizes the last 4 turns of conversation, passes them to a classifier model, categorizes them into 4 categories (simple / medium / complex / reasoning), then maps them to weak / strong tiers. It uses tool calling for judgment, and adopts a fail-open design where it falls back to the default tier when confidence drops below a threshold or classification fails. Three classification policies are built in—general, coding_agent, and openclaw—and it's interesting that there are pre-prepared options for coding-oriented and resident assistant use cases.

cascade is even more sophisticated, judging tool execution result signals in 3 layers: error severity, test pass/fail, and number of file edits. It immediately decides on clear-cut situations (critical errors go to strong, finishing tasks with all tests passing go to weak), then judges based on weighted scores, and only consults the LLM classifier when it can't be confident. The only dial the user touches is confidence_threshold, and the recommended value of 0.5 is described as having been calibrated on SWE-Bench Pro.

From Installation to Serve

Python 3.12 or later is required. I created an environment with uv this time.

uv init switchyard-handson && cd switchyard-handson
uv add "nemo-switchyard[server,cli]"

Wheels are available for Linux x86_64/aarch64 as well as macOS arm64, so it installs directly on Apple Silicon Macs.

The configuration has a 3-layer structure: endpoints (provider connections), targets (upstream models), and profiles (routing policies shown to clients). I assigned GLM-5.2 to the strong tier and DeepSeek V4 Flash to the weak tier. The classifier responsible for routing decisions references the same target as weak. I actually made a painful mistake in selecting this classifier model the first time, and the configuration you see now incorporates the lessons from that—I'll cover the full story in the latter half.

endpoints:
  openrouter:
    base_url: https://openrouter.ai/api/v1
    api_key: ${OPENROUTER_API_KEY}

targets:
  strong:
    endpoint: openrouter
    model: z-ai/glm-5.2
    format: openai
  weak:
    endpoint: openrouter
    model: deepseek/deepseek-v4-flash
    format: openai

profiles:
  fast:
    type: passthrough
    target: weak
  smart:
    type: llm-routing
    profile_name: coding_agent
    strong: strong
    weak: weak
    classifier: weak

There were only 2 stumbling points. The profile type name uses hyphens: random-routing, while the underscore version random_routing that appears in the quick start example is a different system for the old route bundle format. Also, classifier in llm-routing takes a target ID as a string. Both issues gave clear error messages telling you the expected format—"expected one of strong, weak, ..." and "expected a string"—so they were easy to fix. Passing the same ID as weak to the classifier is not laziness; defining two targets with the same model results in a duplicate registration error. The solution is to have a single target referenced by both the decision role and the response role.

This naming difference between the two formats wasn't noted in the quick start, so I've sent a PR to add a note to the upstream documentation.

https://github.com/NVIDIA-NeMo/Switchyard/pull/19

Starting is a single command.

uv run switchyard serve --config profiles.yaml --port 4000

This exposes all three APIs—OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses—on the same port. Even if the client-side format and the upstream-side format differ, they're mutually converted through an internal intermediate representation.

/v1/models Returns 3 Types of IDs

When you bring up serve and first look at GET /v1/models, you get the best understanding of Switchyard's design philosophy. The returned model list contains different types of IDs coexisting.

{"id": "smart", "display_name": "llm-routing", ...}
{"id": "strong", "display_name": "z-ai/glm-5.2", ...}
{"id": "z-ai/glm-5.2", "display_name": "target strong", ...}

Specifying the profile ID (smart) in model activates routing, while specifying a target ID (strong) or upstream model name bypasses routing and locks to that model. In other words, clients can choose between "I want distribution" and "I want this specific model" simply by which model name they use.

Seeing this made me stare into the distance involuntarily. This was precisely the biggest wall in the validation I was preparing as a practical follow-up to the LLM Router series.

Is LLM Router's "Ignoring the model Name" Problem Resolved?

LLM Router had a spec where it didn't look at the model field in the request body. Even if a client explicitly specified model: claude-opus-4-8, it would be overridden by auto routing, making it incompatible with clients like Claude Code that send different model names depending on the task type. In my testing, I worked around this by patching a fork with a model name bypass.

Let me run the same test on Switchyard. I sent the same prompt "Say OK only." to an llm-routing profile 5 times, changing only the model name specified.

When you want routing, you get routing; when you want a fixed model, you get a fixed model. Behavior that took patching a fork's code to achieve in LLM Router is officially supported from the start. For this point alone, I think the switch is worth making.

Connecting Claude Code with a Single Command via switchyard launch claude

Claude Code is an Anthropic API-exclusive agent, so traditionally you needed to insert a conversion proxy like CCR (Claude Code Router) to connect it to an OpenAI-compatible routing proxy. Since Switchyard has Anthropic Messages conversion built in, this becomes a single command.

switchyard launch claude

This starts a proxy on an available port and launches Claude Code with ANTHROPIC_BASE_URL and other settings replaced. The default configuration is a tested trio of Claude Opus 4.7 (strong), Kimi K2.6 (weak), and Gemini 3.5 Flash (classifier), with a status footer at the bottom of the screen showing real-time request counts and token counts per tier. It's quite satisfying to send a mix of simple instructions and heavy queries and watch the distribution numbers change in the footer.

When asked "What model are you currently running on?" within the session, it returned the route ID switchyard-deterministic-.... Claude Code itself doesn't know it's running in a proxy routing setup, while behind the scenes Kimi K2.6 is responding. This transparency is the heart of the launcher.

A smoke test is provided for connection verification. It automatically checks 8 steps from credential resolution through proxy startup to actual Claude Code responses.

[1/8] Resolving credentials...        OK
[2/8] Reaching backend...             OK (GET /models 200, 289ms)
[3/8] Probing /v1/messages support... OK (native passthrough)
[4/8] Starting proxy...               OK (127.0.0.1:51068)
[5/8] Locating claude binary...       OK
[6/8] Round-tripping chat completion... OK (reply='ok')
[7/8] Spawning claude with proxy env...  OK (10463ms)
[8/8] Tearing down proxy...           OK
verify claude: PASS (model=moonshotai/kimi-k2.6, 14322ms)

To switch between multiple routing configurations, pass a YAML called a route bundle. Registered routes appear in Claude Code's /model picker, so you can switch mid-conversation—say, "normally let the classifier decide, but force Sonnet for difficult parts." One thing to note: Claude Code's picker only displays models whose IDs start with claude or anthropic. Switchyard auto-generates aliases with a claude- prefix, but if you name your routes like claude-smart from the start, they'll match the picker display without confusion.

When I actually opened the picker, the registered routes were followed by a long list of backend catalog models with claude- prefixes. Over 340 on my machine. This means you can directly switch to models you haven't defined as routes, on a whim.

Note that route bundles are treated as legacy format and trigger a deprecated warning at startup. Since there's currently no option to pass the new-format profile config to the launcher, this is the only way to switch between multiple routes in the picker. Bundle-based startup can take nearly a minute to reach listen, and I initially thought it had hung. This is a transitional gap typical of v0.1.0. Note that this old/new relationship will be completely reversed three weeks after the article's publication. See the addendum at the end for details.

Does /effort Finally Work Properly?

Another pitfall I hit in LLM Router testing was actually on the CCR side. Claude Code can adjust thinking depth in 5 levels with the /effort command, but that value goes into output_config.effort in the body, while the thinking field always has {type: "adaptive"} attached regardless of the effort level. Since CCR's think determination only looks at the presence of thinking, even light /effort low requests were sent directly to expensive models. Working around this required writing a custom router to replace the judgment logic.

Let me reproduce the same situation in Switchyard. I sent Anthropic Messages requests to an llm-routing profile with thinking: {type: "adaptive"} attached and output_config.effort varied across 5 levels.

Since the prompt is "Say OK only." for all of them, routing all to weak based on content is the correct answer. Routing didn't misfire even with the thinking field attached. Switchyard's conversion layer is designed to first convert requests to an intermediate representation, where output_config.effort is treated as a first-class field. It seems structurally impossible for there to be a place where "thinking is present, so it must be heavy" short-circuits the decision.

Building Fully Local Routing on a Single DGX Spark

Since aarch64 wheels are available, I tried it on DGX Spark as well. Creating a venv and installing nemo-switchyard[server] is all it takes—even on GB10's aarch64 environment, import works without issues.

Since I was at it, I set up a completely local configuration using no external APIs. I used two models loaded into ollama as tier stand-ins.

targets:
  strong:
    endpoint: ollama # http://localhost:11434/v1
    model: qwen3.6:35b
  weak:
    endpoint: ollama
    model: qwen3:1.7b

When I asked the llm-routing profile "What is 2+2?", the 1.7B model answered immediately; when I threw "Prove the halting problem is undecidable using the diagonal argument," it switched to the 35B model. The routing decisions included, everything runs entirely within a single DGX Spark. For those who have been bothered by "it's wasteful to wake up the 35B for trivial questions" in local LLM operation, this is quite a compelling setup.

I learned one thing from this. Initially I assigned the 1.7B to the classifier as well, but small models ignore the forced tool calling specification (tool_choice) and respond with plain text, causing all classifications to fail. Since it's fail-open design, routing itself doesn't stop and traffic keeps flowing to the default tier, but I was able to catch it because the stats API shows classifier error counts directly. Using a model large enough to reliably handle tool calling for the classifier seems to be a key practical point.

Switching My Everyday Hermes Agent Usage to Switchyard

I've been running Hermes Agent through LLM Router. I took this opportunity to switch this everyday traffic to Switchyard as well. The change was just replacing base_url in the connection config file.

model:
  default: hermes # profile name on the Switchyard side
  provider: custom
  base_url: http://localhost:4000/v1
  api_key: dummy
  api_mode: chat_completions

Switchyard's serve doesn't require authentication from clients, so a dummy API key works fine. I assigned the openclaw policy, meant for resident assistants, to the profile. In the operational profiles.yaml, I've incorporated model names into the tier names so they're easy to understand when reviewing later.

targets:
  weak-ds:
    endpoint: openrouter
    model: deepseek/deepseek-v4-flash
    format: openai
  strong-glm:
    endpoint: openrouter
    model: z-ai/glm-5.2
    format: openai

profiles:
  hermes:
    type: llm-routing
    profile_name: openclaw
    strong: strong-glm
    weak: weak-ds
    classifier: weak-ds
    fallback_target_on_evict: strong-glm

I hit one pitfall here. When fallback_target_on_evict in llm-routing is omitted, it looks for a target named strong. The moment I changed the tier name to strong-glm, I got a startup error. If you use tier names other than the default strong/weak, explicit specification is required.

I ran everyday traffic through this configuration for about 15 hours overnight. Stats showed 56 routing decisions, 39 dispatches to weak, and 0 errors. The overall DeepSeek V4 Flash usage during the period was 95 requests, approximately 2.53 million tokens, $0.25 on OpenRouter actual billing, and it was satisfying to see the Switchyard count (56 decisions + 39 weak dispatches = 95) match the billed request count exactly. GLM-5.2 on the strong side also ran with 0 errors including tool calling and streaming. Routing decision overhead has a median of about 7.2 seconds, but Hermes traffic is mainly cron jobs and asynchronous message responses, so there's no practical impact.

Additionally, for scheduled delivery jobs where I don't want to compromise quality, I moved to specifying target IDs like strong-glm directly as the model name without going through routing. The "use profile ID when you want distribution, use target ID when you want a fixed model" approach from the first half of the article translates directly into an operational tool.

Note that the scheduled news delivery job remains on LLM Router, so I have a setup where I can see "LLM Router operation" and "Switchyard operation" running in parallel over the same period.

There's also an issue I found precisely because I put it into production. OpenRouter recorded 2.53 million tokens, but the token count for weak in Switchyard's stats showed only a fraction of that. Investigating further, I found that streaming response usage doesn't appear in stats (only buffered responses are aggregated), and confirmed that usage frames sent by the upstream are dropped. Since agent-originated traffic is almost entirely streaming, cost tracking becomes completely invisible in real-world operation. I've reported this as an issue with reproduction steps and the root cause.

https://github.com/NVIDIA-NeMo/Switchyard/issues/18

The follow-up on this issue is described in the addendum at the end.

I Actually Chose the Wrong Classifier Initially

Full disclosure: the tier configuration I've been showing so far is the second version. Initially, to make it easier to compare with the model pool in the LLM Router article, I set strong to Claude Sonnet 4.6, weak to Nemotron 3 Nano, and the classifier to Gemini 3.5 Flash. The selection rationale was: "Flash is in the name, so it must be a cheap model suitable for decision-making."

Running Hermes through this initial configuration for half a day, 125 requests showed 0 errors, 98.4% routed to weak, and routing decision overhead with a median of about 2.1 seconds—behavior was impeccable. However, when I checked the OpenRouter billing, while the weak body consumed about 5.67 million tokens for $0.32, the classifier Gemini 3.5 Flash used about 590,000 tokens for $0.70. The decision-maker was consuming more than twice the cost of the main model. Since llm-routing sends the last 4 turns of conversation to the classifier every turn, long agent contexts are passed along wholesale, exceeding an average of 4,000 tokens per call. This made me viscerally understand why features like sticky (fixing the tier after the first decision) and cascade (avoiding unnecessary classifier calls) are provided.

However, it wasn't just a traffic characteristic problem. When I laid out the pricing tables, I realized it was an error in model selection itself.

According to OpenRouter's model information, Gemini 3.5 Flash has mandatory reasoning—meaning there's no way to stop its thinking. Checking the stats from that period, of the 32,360 completion tokens from the classifier, 65% (21,184 tokens) were reasoning. This means I was paying $9.00/M in thinking fees every time for a job that only needed to "classify into 4 categories and return via tool calling." The failure was picking based on name impression and skipping the pricing table check.

What's frustrating is that this information was already in my local knowledge base. In LLM Router testing two weeks prior, I had recorded "using a reasoning model as judge causes thinking to be unstoppable and makes decisions heavy"; just days before in OCR model selection, I had also noted "Gemini 3.5 Flash has mandatory reasoning and is excessive for simple tasks." Recording information is meaningless if you can't retrieve it at the moment you're writing a config. A painful lesson.

So I unified weak and classifier to a single model, DeepSeek V4 Flash playing two roles, and reselected strong as GLM-5.2—that's the current configuration used throughout this article. GLM-5.2 achieves scores comparable to Sonnet 4.6 on Artificial Analysis metrics, while being positioned at roughly 1/3 the input price and 1/5 the output price. Comparing the decision aspects before and after the switch:

Cost per decision is approximately 1/12. Since I unified weak and classifier to the same model, I can no longer separate decision charges in OpenRouter billing, so the post-switch value is an estimate multiplying stats token counts by the official rate. The estimate for 56 decisions overnight is $0.02, so the reversal where "decision-maker costs twice the main model" has been corrected to "decision-maker costs 1/10 of main model."

However, it didn't come free. Decision latency median went from 2.1 seconds to 7.2 seconds—more than tripling. It seems surprising to get slower after eliminating reasoning, but the work of processing an average 4,500-token prompt every time remains unchanged, so the raw response speed of the model and provider shows up directly. If placing this in front of an interactive agent, decision cost and decision latency need to be weighed as separate axes.

The tier swap itself only required a few lines of YAML changes and a serve restart. First deploy a working configuration, then swap out models while watching stats and billing. I think this being straightforward is one of the benefits of routing proxy becoming a pip library.

Things I'm Concerned About

I've written a lot of positives, so let me honestly summarize the current caveats too.

First, the development status is Alpha, and known issues are publicly listed. As of 0.1.0, there are 2 cases: token counting going to 0 in Codex integration, and requests with tools failing when routed to upstream with a fixed tool schema. The latter can be triggered in agent operations that heavily use tools, so it's safest to ensure all tier models in routing destinations support tool calling.

Be careful about missing format: specifications too. Omitting it treats it as OpenAI format, and the documentation explicitly states that cache_control for Claude's prompt caching gets stripped when sending to Claude-series models. Be sure to explicitly specify format: anthropic for targets using Claude as upstream.

My customized version of LLM Router would automatically pick one from a pool of 9 models using a trained classifier. The built-in routing methods in Switchyard are all designed as binary strong/weak choices with a classifier, so for more granular selection you'd define multiple routes and have the caller explicitly choose via model name or /model picker. Automatic decision is limited to binary choice; multi-way selection is left to the caller's explicit specification. This is a similar approach to what I previously introduced with Sakana Fugu, where the client's responsibility is choosing between fugu and fugu-ultra, with orchestration running internally in the called destination.

https://dev.classmethod.jp/articles/sakana-fugu-ga-first-touch/

Does this mean automatic multi-way selection won't come back? Reading the code, I found an interesting discovery. The documentation lists 4 routing methods, but the source already has a type implemented for incorporating LMSYS's RouteLLM (a learning-based router using matrix factorization) as a profile. Even though trained routers appeared to have disappeared, the receptacle for learning-based approaches is properly prepared. This aligns with NVIDIA's response that "LLM Router algorithms are being ported to Switchyard." For those who have LLM Router training assets, this is a point I want to dig into in a follow-up article. ...Or so I wrote, but this plan didn't pan out. See the addendum at the end.

Post-Publication Updates (Addendum 2026-08-05, Supplemented 2026-08-08)

What Changed in This Month

The issue mentioned in the latter half of the article—streaming response tokens not appearing in stats—was fixed on July 14. Since agent-originated traffic is almost entirely streaming, cost aggregation is now usable in real-world operation.

There's also a follow-up on the documentation PR I sent about the random-routing vs random_routing naming difference. The maintainer indicated they wanted to "unify on the code side rather than explain in documentation," so I reworked it as a code fix (PR #22) that accepts both hyphens and underscores, which was merged on July 7. The stumbling point introduced in the main body led directly to an upstream fix.

Configuration underwent major changes. The route bundle I described as "deprecated legacy format" is the one that survived, while the profile config I was using as the new format has been removed. From the end of July, it was rebuilt as a standalone Rust server with TOML configuration. The RouteLLM integration I said I wanted to dig into in a follow-up was also deleted on July 16.

However, pip install nemo-switchyard still installs 0.1.0, and the steps in this article still work as-is. If you look at GitHub's main, it's something completely different, so if you're starting fresh, confirm whether you're dealing with the PyPI version or main first.

I've continued updating the configuration since then, switching weak to the official DeepSeek V4 Flash-0731 and strong to Kimi K3. The bundle's current default also uses this configuration. The overall picture—from the reasoning behind narrowing down to 2 models, to settings that keep data from leaving externally, to observation via routing logs—as a team AI environment is summarized in the following article (published in 2026-08).

https://dev.classmethod.jp/articles/open-weight-team-ai-environment/

Tried it on internal workloads with opencode and Fireworks

At the time I wrote the article, my local Hermes Agent was the only production use, but since late July I've been running another workload internally. The setup places Switchyard behind the OSS coding agent opencode and routes to Fireworks AI models. It's a two-role configuration with DeepSeek V4 Pro assigned to strong, and DeepSeek V4 Flash assigned to weak and classifier, with session affinity enabled.

https://opencode.ai/

https://fireworks.ai/

I ran an A/B test on coding tasks. The same task set was run across 3 arms — auto (with routing), strong fixed, and weak fixed — for a total of 39 runs. All three arms scored full marks with zero failures, and auto came in at roughly 27% cheaper than strong fixed, including the cost of the classifier. Looking at just the main body cost excluding the classifier, it's 40%.

However, measuring quality differences requires harder tasks that involve design decisions spanning multiple files. Latency was also worse with auto, with a median real-world difference of 30 seconds versus 21 seconds.

Session affinity worked straightforwardly. Classifier calls dropped by 56%, and tier switches within a session were zero. On the other hand, the same task can be pinned to either weak or strong depending on the session, so there's some variance in cost estimates.

When using DeepSeek models on Fireworks, you need to write extra_body: {} empty in the configuration. This is because the vLLM parameters that Switchyard adds automatically get rejected by Fireworks with an HTTP 400. It's been reported upstream, but for now working around it on the config side is the safe bet.

The full configuration has been published as a Docker bundle. Anyone who wants to try the same combination can get it running from here.

https://github.com/himorishige/switchyard-opencode-bundle

Summary

I ran NeMo Switchyard on a Mac and DGX Spark, and re-examined the two pitfalls I hit during LLM Router testing. The issue where model names were being ignored has been officially resolved as a distinction between profile / target IDs, and the /effort blowup no longer occurs at the design level of the conversion layer. Both the patch to the fork and the custom router I had built for CCR are now unnecessary.

It installs via pip with no GPU required, connects to Claude Code with a single command, and on DGX Spark you can set up fully local routing. While it still has rough edges befitting an Alpha release, the barrier to entry has dropped dramatically compared to the overhead of forking and nurturing a Blueprint for an LLM Router.

The classifier cost dropped to about one-twelfth by switching models. Next time, I'd like to look at the comparison between session affinity and stage_router for reducing the number of decisions themselves, using real data from the internal workload mentioned in the addendum.