I tried NeMo Switchyard v0.2.0 reborn in Rust

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA's LLM routing infrastructure NeMo Switchyard has received v0.2.0. The release date was August 10, 2026. Starting from v0.2.0, the routing engine has been rewritten from Python to Rust, and packages are now published on crates.io and PyPI.

https://github.com/NVIDIA-NeMo/Switchyard

I wrote a first-touch article back when v0.1.0 was released, and since then I've been regularly using this router for internal testing with opencode + Fireworks AI. v0.2.0 is not just a version bump — the configuration file format and the classifier algorithm are completely different. A technical blog post from NVIDIA organizing the routing concepts was also published around the same time, so I'll read through that while trying things out.

https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard

To summarize the conclusion upfront: v0.2.0 has reached the point where the idea that "the optimal model differs per task" can be put into operation with just two threshold calibrations. And the value of automatic routing that I've felt over a month and a half of use is not spotting difficult tasks and routing them to a stronger model, but rather not sending the majority of non-difficult work to expensive models. I'll elaborate on this in the analysis section.

Here is the article from v0.1.0 (written as of 2026-07-03). Since the route.yaml and classification preset discussion from back then has now become the old configuration, this article serves as a practical rewrite.

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

This article covers the problems Switchyard is trying to solve, practical usage and calibration in v0.2.0, and what I've been thinking about after a month and a half of operation.

What Problem Is Switchyard Trying to Solve?

A coding agent calls LLMs dozens of times per task. Looking at the breakdown, the majority of the work involves reading files and summarizing them, checking test results, or applying routine fixes — situations that require deep reasoning are only a small portion. Yet the operation tends to be "pick the smartest single model and hand everything to it," meaning you end up paying frontier model prices even for routine tasks. But if you go with a small model only, quality breaks down exactly when it matters most.

Switchyard's premise is that this binary choice is a false one. In the official technical blog, the argument is that each model has its own strengths, weaknesses, and cost characteristics, and rather than using a single model, you should use a "system of models" to select per request. What's interesting is the evidence: experiments using Terminal-Bench Hard show that the optimal model differs across task groups. For ML task groups it's this model, for math and science it's that one, and for the rest it's yet another — there is no single "strongest" model. The position is that the purpose of routing is not to match the strongest model, but to select the model that satisfies the required quality at the lowest cost.

Concrete reduction numbers are also presented as case studies. In a LangChain multi-turn agent evaluation, a 74% cost reduction compared to frontier model alone was achieved with only 7% of calls going to the frontier model. In a Cognition evaluation, approximately 28% cost reduction was reported within a 2.8 percentage point accuracy difference. The reading here is not that you throw away quality to cut costs, but that you measure how much quality drops and then trim accordingly.

So what signals does routing use? According to the official documentation, there are 3 tuning-free routers and 1 learning-based router.

This article focuses on the LLM classifier, which in v0.2.0 has two modes: capability and escalation. The concepts corresponding to the 1st and 3rd rows of the table above have been integrated into a single route type.

The learning-based prefill router appears to be only conceptually introduced at the time of writing and has not yet been included in a release. However, this slot looks familiar. It's in the same lineage as the NVIDIA LLM Router I previously tested on DGX Spark — training a dedicated classification model for routing on your own workload. Since Switchyard was originally described to me as infrastructure that would "incorporate the LLM Router algorithm as well," I expect the learning-based approach to eventually merge into this prefill router slot. My testing at the time is summarized in two articles (both written as of 2026-06-21).

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3/

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3-training/

v0.2.0 Was Reborn as a Rust Server

Switchyard v0.1.0 was a Python package, but in v0.2.0 this Python routing implementation has been removed from main, and the successor is a standalone Rust binary called switchyard-server. The routing logic is extracted as a provider-agnostic SDK, and the server is its reference implementation, accepting requests in three formats: OpenAI Chat Completions, Responses, and Anthropic Messages.

Here is a table summarizing the main changes and how knowledge from the previous version maps over.

Since everything from the configuration file format to the classification algorithm has changed, this is closer to a rebuild than a migration. That said, as I'll describe later, the calibration approach is considerably more organized than in the previous version, and personally I think it was a worthwhile rebuild.

Boots Up in 84 Seconds from cargo install

This is the most welcome change in v0.2.0. Previously, running the Rust server required pinning a specific git commit and building the entire workspace, with the initial build taking 10–20 minutes. Now that it's published on crates.io, a single cargo install specifying the release version is all you need.

ARG RUST_VERSION=1.96.1
FROM rust:${RUST_VERSION}-bookworm AS builder

ARG SWITCHYARD_VERSION=0.2.0
RUN cargo install --locked switchyard-server \
        --version "${SWITCHYARD_VERSION}" \
        --root /opt/out

FROM debian:bookworm-slim
COPY --from=builder /opt/out/bin/switchyard-server /usr/local/bin/switchyard-server
COPY routes.toml /app/routes.toml
EXPOSE 4100
ENTRYPOINT ["switchyard-server"]
CMD ["--config", "/app/routes.toml", "--host", "0.0.0.0", "--port", "4100"]

In my local measurements, building this image took 84 seconds total, with cargo install taking 70 seconds. The image size is 187MB. Compared to when it used to take 10–20 minutes, the psychological barrier to rebuilding after changing configuration is in a completely different league.

Once you've written your configuration, you can validate it with --dry-run before starting. routes.toml is designed to error on unknown fields rather than silently ignoring them, so typos show up as startup errors rather than mysterious runtime behavior. That's quietly appreciated.

switchyard-server --config routes.toml --dry-run

routes.toml Is Written in 3 Layers

The configuration has a 3-layer structure: llm_clients, targets, and routes. Connection definitions, model definitions, and routing definitions are separated, which makes it feel cleaner than v0.1.0's route.yaml. Official documentation also gained a new TOML schema reference in v0.2.0.

Here is a version extracted from the configuration I'm running in actual operation.

schema_version = 1

[llm_clients.fireworks]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"

# Dedicated client definition for the classifier (judge). Since targets are
# managed as (llm_client, model ID) pairs, if you want to use the same model
# as weak for the judge, keep both by giving the client a different name
[llm_clients.fireworks_judge]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"

[targets.classifier]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"
llm_client = "fireworks_judge"
extra_body = { reasoning_effort = "none", temperature = 0 }

[targets.strong]
id = "accounts/fireworks/models/kimi-k3"
llm_client = "fireworks"

[targets.weak]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"
llm_client = "fireworks"

[routes.auto]
id = "auto"
type = "llm_classifier"
mode = "capability"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.75
threshold_step = 0.1
session_affinity = true
recent_turn_window = 4

[routes.auto-esc]
id = "auto-esc"
type = "llm_classifier"
mode = "escalation"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
escalation = { confirmations = 2 }

[routes.strong-only]
id = "strong-only"
type = "passthrough"
target = "strong"

[routes.weak-only]
id = "weak-only"
type = "passthrough"
target = "weak"

The route id becomes the model name visible to clients. From the opencode side, you simply select something like switchyard/auto to switch between automatic routing and fixed routes. Keeping passthrough fixed routes alongside allows you to compare behavior without going through the classifier, which also helps with troubleshooting.

The extra_body in targets is a knob for injecting additional parameters into requests to that target. Here it's used to stop the classifier from thinking. Judgment is just a job of returning a single probability, so there's no point in letting a reasoning model think for a long time. The effect of this one line is measured in the calibration section.

The Capability Classifier Estimates the "Probability of Being Solvable"

v0.1.0's classifier categorized tasks into 4 categories from SIMPLE to REASONING, then routed to tiers using a preset mapping table. The v0.2.0 capability mode changes the concept: a judge LLM estimates the "probability that the weak model can complete this task" — internally called p_solve — and compares it against a threshold to decide routing.

The threshold is not uniform. The judge simultaneously determines the type of task: for domains where weak is expected to be strong, the base_threshold applies as-is; for domains where the judge is less confident, a stricter threshold elevated by threshold_step is applied. With my configuration, this creates a staircase: 0.75 for familiar domains, 0.85 for uncertain ones, 0.95 for unfamiliar ones. The design has "when in doubt, route to strong" built in.

Another important factor is how much of the conversation the judge sees. By default, it only sees the first user message and the most recent user message. Setting recent_turn_window passes the last N messages — including assistant responses and tool execution results — to the judge.

This made a real difference. Here are the results from taking 320 judgments on 40 real-world coding agent conversation shapes, with and without the window (including re-runs with reversed arm order to eliminate ordering confounds).

Routing destinations barely changed, yet the judge's thinking dropped to roughly 1/7 and judgment became about 6x faster. With default input, the judge is "estimating completion probability from only the initial request," and the lack of context seems to cause longer deliberation. When decisive material like tool execution results is included, it can decide immediately — that's my interpretation.

However, this is a result for code work. When I opened the same window for design consultation-style conversations, I did observe a phenomenon where the judge leaned toward weak because the weak model's responses looked reasonable. Please read the numbers in this article with the assumption of a coding agent workload.

session_affinity = true is also a practically required setting. Agents call LLMs dozens of times per task, so judging every turn would make costs proportional to the number of turns. Enabling affinity locks the tier at the start of the session and skips subsequent judgments.

There Is No Default Threshold Value — Calibrate with Your Own Workload

The base_threshold for capability mode is a required field, and upstream provides no recommended value. At first this felt unhelpful, but my view changed after calibrating. This value has such wildly different optimal points depending on workload that providing a default would actually be dishonest.

For calibration, I used a set of 87 judgments: 40 conversation shapes from actual operation and 47 standalone prompts. First, measuring the p_solve distribution showed that real-world coding operation shapes cluster around a median of 0.85–0.86, while deep business-oriented brainstorming conversations drop to around 0.55. The results of sweeping the threshold against this distribution were as follows:

• At 0.5 (as in the documentation example), only 9 out of 13 deep brainstorming questions reach strong
• Raising to 0.75 captures 12 questions, while all 40 real-world coding operation shapes stay on weak
• From 0.80 onward, real-world operation shapes start flowing to strong, working against cost goals

So the answer for my environment was base_threshold = 0.75. The point here is not the value 0.75 itself, but that this number will be a different value in your environment. The distribution of p_solve depends on both the model pairing and the shape of tasks, so measuring once with a sample of real traffic is ultimately the fastest approach.

Classifier thinking suppression was also measured alongside calibration. Comparing the presence and absence of extra_body = { reasoning_effort = "none", temperature = 0 } set up in the routes.toml section across the same 87 judgments: judge latency was 2.2 sec vs. 10.4 sec at p50, and 4.4 sec vs. 44.5 sec at p90. The routing destinations for real-world operation shapes matched in 30 out of 30 cases, so stopping the thinking did not degrade judgment quality for code work. temperature 0 is to prevent judgments near the threshold from flipping on re-execution.

Escalation Mode Promotes Only Stuck Trajectories

If capability is "prediction before starting," escalation mode is "promotion based on evidence of actually getting stuck." All sessions start on weak, the judge monitors the trajectory, and when it produces 2 consecutive escalation judgments, that session is locked to strong.

I measured this by creating a stuck trajectory: the same ImportError repeated 4 times, an unrelated file edited, and a completion declared despite the issue not being fixed — an overtly problematic conversation.

Judgment quality was better than expected. In "healthy friction" conversations where relevant fixes accumulate while tests keep failing, no escalation occurred at all. Only truly stuck trajectories with sufficient evidence latched within 2 turns. The judge's prompt uses the criteria "only clearly stuck patterns, no escalation on isolated failures, don't escalate when unsure," and in practice it behaved accordingly.

However, there are trade-offs. Here is a comparison with capability mode:

The loss of streaming was confirmed in testing: a response that flows in as 361 chunks with weak fixed arrives as a single chunk with auto-esc. In interactive use, the screen freezes during long generation, so personally I think this is an option better suited to non-interactive workloads like cron jobs or batch processing.

The Router Isn't the Only Routing Entity

From here, I'll share what I've been thinking about after operating this router for a month and a half.

The thing that left the strongest impression from operating the previous version was how quietly the classifier could change. A concrete example: when I swapped the classifier model to a minor update version, judgments on the same 50 conversations flipped from weak 39 to weak 1 — all the way to one end. Not a single error occurred, and confidence remained high. To check whether this was model-specific behavior, I ran 1,392 judgments across 8 models × 2 prompts × 87 judgments, and found that even with the same prompt and same input, weak judgment rates scattered from 0–100% depending on the model. What the judge reads and how it reads it turned out to be a far larger variable than the threshold.

That's precisely why ongoing observation of the judgment distribution is necessary in operation. In v0.2.0, /v1/stats provides tier distribution and classifier overhead, --routing-log-file leaves per-request JSONL, and classifier calls themselves are recorded in logs. I aggregate these logs weekly, and it was this habit that allowed me to notice the judgment shift. v0.2.0's design looks good in light of this experience: the judge's input is narrowly defined, calibration is consolidated into 2 thresholds, making it clear what to re-measure when the model changes.

On the other hand, the limits of predictive routing itself have also become apparent. When I ran 14 hard competitive programming problems through it, the capability judge routed all 14 to weak. This is because tasks where the output format is fully specified as a specification and can be mechanically verified by tests appear to the judge as "having high probability of being solvable." Tasks where difficulty lies in algorithmic insight rather than specification complexity are, in principle, hard to detect with this approach.

So my current thinking is "you don't need to have the router do everything." I think of there being 4 subjects for escalation: the router's predictive routing is auto, the router's trajectory judgment is auto-esc, and explicit human switching is passthrough fixed routes or agent-side mode settings. A fourth path is also conceivable: the model itself consulting a higher-tier model via tools. I've settled on a configuration where I manually fix brainstorming and planning to strong, and leave everyday coding to auto.

Since settling on this configuration, my understanding of strong and weak has also changed. Strong is less a "safety net for hard problems" and more a "source of specific capabilities." The official benchmarks also show that the optimal model differs across task groups, which aligns with the view that model superiority isn't a one-dimensional strong/weak ranking but rather a landscape of capability strengths and weaknesses. Put weak at the center of daily work, and explicitly source only the capabilities that are lacking. The value of auto lies more in "preventing over-escalation that sends routine work to strong" than in "protecting quality on hard tasks" — that's my current feeling.

The Next Battlefield for 30B-Class Models Looks Like Specialized Agents

Extending this "source of specific capabilities" view, the models lined up beyond the router will probably not just be a vertical ladder of strong and weak. Right around the time I was writing this article, Fastino announced models with domain-specific training of Nemotron 3.5 Lightning in finance and healthcare respectively, in collaboration with NVIDIA, released under Apache 2.0. The base is a lightweight MoE (Mixture of Experts) model at 30B with 3B active parameters, and according to Fastino's announcement, the financial benchmark FinQA moved from 15.9% to 59.2%. This is a concrete example of a lightweight model matching a large model in its specific domain when the domain is narrowed. This is exactly the kind of use case enabled by open models with publicly available weights, and the more privacy-sensitive or heavily regulated the industry, the more this approach will proliferate.

https://fastino.ai/blog/fastino-nemotron-3-5-lightning-finance-and-healthcare

What becomes interesting then is the idea of treating a business domain or an organization itself as a single large model. Specialized lightweight models become the experts within it, and the router becomes the gating that decides which expert to call. It's the MoE structure being externalized from inside a single model to the architecture of the entire system. Looking at the combination of Nemotron 3.5 Lightning as the "lightweight execution unit" and Switchyard as the "supervisor that allocates work," NVIDIA may already have moved beyond the competition to "pick the single strongest model" and into agent design that decomposes planning, execution, and verification across separate models.

As with Meta Superintelligence Labs' recently open-weight released Muse Glimmer 30B, it's becoming increasingly difficult for models of this size to compete at the top in overall intelligence. The battleground is shifting to "which domain to become an expert in," and it seems likely that the next frontier for 30B-class models will be as specialized agents. I have a feeling that the day when domain-specific targets are lined up in my local Switchyard routes.toml isn't too far off.

Summary

Here are the actual measurements comparing 4 routes on real coding agent tasks. I ran 10 code modification runs and 3 tool call runs on each route, with all 52 runs completing without failures.

The −98.0% figure is appealing, but since this task set was at a difficulty level where weak alone also achieves perfect scores, this number is not proof that "quality was maintained while cutting costs" — it's proof that "tasks that didn't need to go to strong were fully routed to weak." Strong calls in auto were zero. In real-world operation with hard tasks mixed in, the reduction rate will naturally be lower; in fact, our team's opencode + Fireworks testing is mentioned in the NVIDIA official blog case studies as achieving 27% reduction. Reading it as landing somewhere in between depending on task composition is the honest interpretation.

To summarize impressions of v0.2.0: the distribution format reachable via cargo install and calibration consolidated into 2 thresholds have clearly brought it closer to a "tool you can operate" compared to v0.1.0. Since it's a fast-moving project as noted in the pre-alpha disclaimer, the realistic approach is to pin a release version and re-measure calibration on your own workload each time you upgrade.

Next, we'll have a full month's worth of real operation logs from the team, so I'd like to verify tier distribution and reduction rates on real traffic rather than benchmarks. I also plan to cover escalation mode deployment for non-interactive workloads in a follow-up.

Reference Links

• NVIDIA-NeMo/Switchyard
• switchyard-server - crates.io
• Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard — Official overview of routing concepts and 4-type classification
• NVIDIA Blog - Nemotron Lightning and NeMo Switchyard — Includes mention of deployment case studies
• Fastino x NVIDIA Collaboration — Domain-specific variants of Nemotron 3.5 Lightning
• Building a Team AI Coding Environment with Open-Weight Models (written as of 2026-08-08)