I tried NeMo Switchyard v0.2.0, reborn in Rust

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA's LLM routing platform NeMo Switchyard has received v0.2.0. The release date is August 10, 2026. Starting from v0.2.0, the routing engine has been rewritten from Python to Rust, and packages are now published on crates.io and PyPI.

https://github.com/NVIDIA-NeMo/Switchyard

I wrote a first-touch article back in v0.1.0, and since then I've been regularly using this router for internal validation with opencode + Fireworks AI. v0.2.0 is not merely a version bump — the configuration file format and the classifier algorithm are both completely different. A NVIDIA technical blog post summarizing the routing philosophy was also published around the same time, so I'll read through that alongside my testing.

https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard

To state my conclusion upfront: v0.2.0 has reached the point where the idea that "the optimal model differs per task" can be put into production with just two threshold calibrations. And the value of automatic routing that I've come to appreciate over a month and a half of use lies not so much in identifying difficult tasks and routing them to a powerful model, but in not sending the vast majority of non-difficult work to an expensive model. I'll elaborate on this in the analysis section.

Here is my article from the time of v0.1.0 (written as of 2026-07-03). Since the route.yaml and classification preset discussion from back then has now become the old configuration, this article effectively serves as a rewrite.

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

This article covers the problem Switchyard is trying to solve, how to use v0.2.0 and calibrate it in practice, and what I've thought about during a month and a half of operation.

What Switchyard Is Trying to Solve

A coding agent calls an LLM dozens of times in a single task. Looking at the breakdown, the majority of work consists of reading and summarizing files, checking test results, and applying routine fixes — deep reasoning is only required in a fraction of cases. Yet in practice, operations tend to default to "pick the smartest model and hand everything to it," meaning you end up paying frontier-model rates even for routine tasks. On the other hand, going all-in on a small model causes quality to collapse at critical moments.

Switchyard's premise is that this binary choice is a false one. The official technical blog frames it as: each model has its own strengths, weaknesses, and cost characteristics, and rather than relying on a single model, you should use a "system of models" that selects the right one per request. What's interesting is the supporting evidence — experiments using Terminal-Bench Hard show that the optimal model differs depending on the task group. ML-type task groups favor this model, math and science favor that one, and the rest favor yet another. There is no single "best" model. The position is that the purpose of routing is not to find the strongest model, but to select the model that meets the required quality at the lowest cost.

Concrete reduction figures are also presented as examples. In a LangChain multi-turn agent evaluation, they report a 74% cost reduction compared to using a frontier model alone, with only 7% of calls going to the frontier; in a Cognition evaluation, a roughly 28% cost reduction with a quality difference of within 2.8 points. This can be read as: rather than sacrificing all quality for cheapness, the approach measures how much quality degrades before cutting costs.

So what signals does routing use? According to the official breakdown, there are three tuning-free routers and one learning-based router.

This article focuses on the LLM classifier, which in v0.2.0 has two modes: capability and escalation. The ideas corresponding to the first and third rows of the table above have been integrated into a single route type.

The learning-based prefill router appears to be only conceptually introduced at the time of writing and not yet included in the release. However, this slot looks familiar. It's in the lineage of the NVIDIA LLM Router I previously validated on DGX Spark — training a dedicated classification model on your own workload. I was told that Switchyard was originally a platform designed to "also absorb the LLM Router algorithm," so I expect the learning-based approach to eventually merge into this prefill router slot. My validation at the time is written up in two articles — a foundations piece and a training piece (both written as of 2026-06-21).

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3/

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3-training/

v0.2.0 Has Been Reborn as a Rust Server

v0.1.0's Switchyard was a Python package, but in v0.2.0 that Python routing implementation has been removed from main, and the successor is a standalone Rust binary called switchyard-server. The routing logic has been extracted as a provider-agnostic SDK, with the server serving as its reference implementation, capable of accepting requests in three formats: OpenAI Chat Completions, Responses, and Anthropic Messages.

Here is a summary of the major changes and how prior knowledge maps to them.

Since everything from the configuration file format to the classification algorithm has changed, it's closer to a rebuild than a migration. That said, as I'll describe later, the calibration approach is much more organized than the old version, and personally I think it was a worthwhile rebuild.

Up and Running in 84 Seconds from cargo install

This is the most welcome change in v0.2.0. Previously, running the Rust server required pinning a specific git commit and building the entire workspace, with initial builds taking 10–20 minutes. With the package now published on crates.io, a single cargo install targeting the release version is all it takes.

ARG RUST_VERSION=1.96.1
FROM rust:${RUST_VERSION}-bookworm AS builder

ARG SWITCHYARD_VERSION=0.2.0
RUN cargo install --locked switchyard-server \
        --version "${SWITCHYARD_VERSION}" \
        --root /opt/out

FROM debian:bookworm-slim
COPY --from=builder /opt/out/bin/switchyard-server /usr/local/bin/switchyard-server
COPY routes.toml /app/routes.toml
EXPOSE 4100
ENTRYPOINT ["switchyard-server"]
CMD ["--config", "/app/routes.toml", "--host", "0.0.0.0", "--port", "4100"]

In my own measurements, the full image build took 84 seconds, with cargo install accounting for 70 of those. The image size is 187MB. Compared to when it used to take 10–20 minutes, the psychological barrier to changing the configuration and rebuilding is completely different.

Once you've written your configuration, you can validate it with --dry-run before starting. routes.toml will error on unknown fields rather than silently ignoring them, so typos surface as startup errors rather than mysterious behavior at runtime. A small but genuinely appreciated touch.

switchyard-server --config routes.toml --dry-run

routes.toml Is Written in Three Layers

The configuration has a three-layer structure: llm_clients, targets, and routes. Since connection endpoint definitions, model definitions, and routing definitions are all separated, I find it more readable than v0.1.0's route.yaml. The official documentation has also added a new TOML schema reference with v0.2.0.

Here is an excerpt from my production configuration, distilling the key points.

schema_version = 1

[llm_clients.fireworks]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"

# Dedicated client definition for the classifier (judge). Since targets are managed
# as (llm_client, model ID) pairs, if you use the same model for both the judge and
# weak, you need to keep separate client entries for both.
[llm_clients.fireworks_judge]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"

[targets.classifier]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"
llm_client = "fireworks_judge"
extra_body = { reasoning_effort = "none", temperature = 0 }

[targets.strong]
id = "accounts/fireworks/models/kimi-k3"
llm_client = "fireworks"

[targets.weak]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"
llm_client = "fireworks"

[routes.auto]
id = "auto"
type = "llm_classifier"
mode = "capability"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.75
threshold_step = 0.1
session_affinity = true
recent_turn_window = 4

[routes.auto-esc]
id = "auto-esc"
type = "llm_classifier"
mode = "escalation"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
escalation = { confirmations = 2 }

[routes.strong-only]
id = "strong-only"
type = "passthrough"
target = "strong"

[routes.weak-only]
id = "weak-only"
type = "passthrough"
target = "weak"

The route's id becomes the model name visible to clients. From the opencode side, you simply select something like switchyard/auto, allowing you to switch between automatic routing and fixed routes. Keeping passthrough fixed routes alongside automatic ones lets you compare behavior without going through the classifier, which also helps with troubleshooting.

The extra_body in targets is a knob that injects additional parameters into requests sent to that target. Here it's used to suppress the classifier's reasoning. Since the judgment is just returning a single probability, there's no point in making a reasoning model think at length. I'll measure the effect of this one line in the calibration section.

The Capability Classifier Estimates "Probability of Solving"

v0.1.0's classifier categorized tasks into four categories from SIMPLE to REASONING, then mapped them to tiers using a preset lookup table. v0.2.0's capability mode takes a different approach: the judge LLM estimates what it internally calls p_solve — the probability that the weak model can complete the task — and compares that against a threshold to decide routing.

The threshold is not uniform. The judge simultaneously determines the task type, and the base_threshold applies as-is for domains where weak should be capable, while a threshold_step stricter threshold applies for domains where the judgment is less confident. With my configuration, this creates a staircase: 0.75 for familiar domains, 0.85 for uncertain ones, and 0.95 for unfamiliar ones. "When in doubt, route to strong" is built into the design.

Another important factor is what part of the conversation the judge sees. By default, it only sees the first user message and the most recent user message. Setting recent_turn_window passes the last N messages to the judge — including assistant responses and tool execution results.

This made a real difference. I ran 320 judgments on 40 real coding agent conversation shapes with and without the window (including a re-run with reversed arm order to eliminate ordering confounds).

The routing destination barely changed, yet the judge's reasoning shrunk to about 1/7 and judgment became about 6x faster. With only the first request in view, the judge apparently needs to think longer because it lacks sufficient material. When decisive evidence like tool execution results is included, it can reach a conclusion immediately — that's my interpretation.

However, these are results for coding workloads. For brainstorming conversations like design consultations, I also observed a phenomenon where opening the same window made the judge lean toward weak, possibly because the weak model's responses look plausible. Please read the numbers in this article with the caveat that they assume a coding agent workload.

session_affinity = true is also effectively a required setting. Since an agent calls the LLM dozens of times per task, judging every turn scales cost proportionally with turn count. Enabling affinity locks in the tier at the start of the session and skips subsequent judgments.

There Is No Default Threshold Value — Calibrate With Your Own Workload

base_threshold in capability mode is a required field, and upstream provides no recommended value. At first this felt unhelpful, but after going through calibration my view changed. This value's optimal point varies too much by workload for a default value to be anything but misleading.

For calibration I used a set of 87 judgments: 40 real-world conversation shapes from production and 47 standalone prompts. First I measured the p_solve distribution: production coding conversation shapes clustered around a median of 0.85–0.86, while deep brainstorming sessions on business topics dropped to around 0.55. The results of sweeping the threshold against this distribution were as follows.

• At 0.5, which appears in the documentation examples, only 9 of 13 deep brainstorming questions reached strong
• Raising to 0.75 captured 12 of them, while all 40 production coding shapes remained on weak
• At 0.80 and above, production shapes began flowing to strong, working against cost efficiency

So the answer for my environment was base_threshold = 0.75. The point I want to make is not the value 0.75, but that this number will be different in your environment. The p_solve distribution depends on both the model pairing and the shape of the tasks, so the fastest approach is ultimately to measure once with a sample of real traffic.

I measured classifier reasoning suppression alongside calibration. Comparing the extra_body = { reasoning_effort = "none", temperature = 0 } setting from the routes.toml section against its absence, across the same 87 judgments: judge latency was 2.2 sec vs. 10.4 sec at p50, and 4.4 sec vs. 44.5 sec at p90. The routing destination for production shapes matched in 30 out of 30 cases, so suppressing reasoning did not degrade judgment quality for coding tasks. Setting temperature to 0 prevents judgments near the threshold from flipping on re-runs.

Escalation Mode Only Escalates on Evidence of Being Truly Stuck

If capability is "prediction before starting," escalation mode is "escalation based on actual evidence of being stuck." All sessions start on weak, the judge monitors the trajectory, and if it produces two consecutive escalate judgments, that session is pinned to strong.

I built a stuck trajectory and measured it. It was a deliberately bad conversation: the same ImportError repeated 4 times, an unrelated file edited, and a completion declared despite the error not being fixed.

Judgment quality was better than expected. Conversations with "healthy friction" — repeated test failures while making relevant fixes — never escalated. Only the clearly stuck trajectory with accumulated evidence latched after 2 turns. The judge's prompt specifies: "only escalate on clear stuck patterns, not on isolated failures, and don't escalate when uncertain" — and measurements confirmed it behaves exactly that way.

However, there are trade-offs. Here's a comparison with capability mode.

The loss of streaming was confirmed empirically: a response that flows as 361 chunks on weak-fixed arrives as 1 chunk on auto-esc. For interactive use, the screen freezes during long generation, so personally I think this is an option suited to non-interactive workloads like cron jobs or batch processing.

The Router Is Not the Only Source of Routing Decisions

From here, I'll share what I've been thinking about during a month and a half of running this router.

The thing that stands out most from running the old version is that classifiers can change silently. Here's a real example. When I swapped the classifier model for a minor update version, the judgments for the same 50 conversations shifted from 39 weak to 1 weak in a complete reversal. Not a single error was thrown, and confidence remained high. To check whether this was a model-specific quirk, I ran 1,392 judgments across 8 models × 2 prompts × 87 judgments, and the weak judgment rate scattered from 0% to 100% across models using the same prompt and same input. What the judge reads and how it reads it turned out to be a far larger variable than the threshold.

That's precisely why continuous observation of the judgment distribution is essential in operations. In v0.2.0, /v1/stats gives you tier distribution and classifier overhead, --routing-log-file writes per-request JSONL, and the classifier calls themselves are also logged. I aggregate these logs weekly, and it was this habit that allowed me to notice judgment shifts. Looking at v0.2.0's design through this experience, it's a step in the right direction — the judge's inputs are narrowly defined and calibration is consolidated into two thresholds, making it clear what to re-measure when you change models.

At the same time, the limits of predictive routing itself have become apparent. When I ran 14 hard competition programming problems through it, the capability judge routed all 14 to weak. Tasks where the output format is fully specified and can be mechanically verified by tests appear to the judge as having "high probability of success." Tasks where difficulty lies in algorithmic insight rather than specification complexity are, I believe, fundamentally hard to detect with this approach.

So my current framing is that "you don't need the router to do everything." I think there are four agents of escalation: router-based upfront prediction is auto, router-based trajectory judgment is auto-esc, and human-driven explicit switching is done via passthrough fixed routes or mode settings on the agent side. A fourth path is also conceivable: the model itself consulting a higher-tier model via a tool call. I've settled on a configuration where brainstorming and planning are human-decided strong-fixed, while everyday coding is left to auto.

Since settling into this configuration, my view of strong and weak has also changed. Strong is less a "safety net for hard problems" and more a "procurement source for specific capabilities." The official benchmarks show the same result — optimal models differ by task group — which aligns with the view that model superiority is not a one-dimensional scale of strength but a profile of capability peaks and valleys. Establish weak as the primary workhorse for everyday tasks, and explicitly procure only the capabilities it lacks. The value of auto lies more in "preventing over-escalation that sends routine work to strong" than in "protecting quality on hard tasks" — that's my current sense of it.

The Next Battleground for 30B-Class Models Will Be Specialized Agents

Extending this "procurement source for specific capabilities" framing, the models lined up behind the router will likely not remain just a vertical ladder of strong and weak. Around the time I was writing this article, Fastino — in collaboration with NVIDIA — published domain-specialized variants of Nemotron 3.5 Lightning, fine-tuned separately for finance and healthcare, under the Apache 2.0 license. The base is a lightweight MoE (Mixture of Experts) model with 30B total and 3B active parameters, but according to Fastino's announcement, it moved the financial benchmark FinQA from 15.9% to 59.2%. A concrete example of a lightweight model matching large models in a narrowly-scoped domain. This is precisely what's possible with open-model weights that are publicly available, and this approach will only grow in industries with strong privacy or regulatory requirements.

https://fastino.ai/blog/fastino-nemotron-3-5-lightning-finance-and-healthcare

What becomes interesting then is the idea of treating a business domain or even an organization itself as one large model. Domain-specialized lightweight models become the experts inside it, and the router becomes the gating mechanism that decides which expert to call. It's as if the MoE structure is being externalized from inside a single model out to the architecture of the overall system. Looking at the combination of Nemotron 3.5 Lightning as the "lightweight execution worker" and Switchyard as the "supervisor that assigns work," NVIDIA may already be moving away from the competition to "pick the one strongest model" and into an agent design that decomposes planning, execution, and verification across separate models.

As with Muse Glimmer 30B, the open-weight model Meta Superintelligence Labs recently released, it's becoming increasingly hard for models of this size to compete with the top tier in a general-purpose intelligence contest. The battleground is shifting toward "which domain to become an expert in," and the next frontier for 30B-class models looks like it will be their role as specialized agents. The day when a domain-specialized target lines up in my local Switchyard routes.toml doesn't feel far off.

Summary

Here are my measurements from comparing 4 routes on real coding agent tasks. I ran 10 code-editing runs and 3 tool-call runs through each route, and all 52 runs completed with zero failures.

The −98.0% figure is compelling, but since this task set is at a difficulty level where weak alone achieves perfect scores, this number is not proof of "reducing costs while protecting quality" — it's proof of "routing all tasks that don't need strong to weak." Strong calls from auto were zero. In a real-world workload with a mix of hard tasks, the reduction rate will naturally be smaller; indeed, in my team's opencode + Fireworks validation, NVIDIA's official blog mentions us as an example of a 27% reduction. Somewhere in this range depending on task composition is the honest read.

To summarize my impressions of v0.2.0: with the cargo install distribution model and calibration consolidated into two thresholds, it has moved clearly closer to being an "operational tool" compared to v0.1.0. Given that this project moves fast, as the pre-alpha caveat implies, the realistic approach is to pin a release version and re-run calibration on your own workload each time you upgrade.

Next, a month's worth of team production operation logs will accumulate, and I'd like to look at tier distribution and cost reduction rates with real traffic rather than benchmarks. I also plan to cover applying escalation mode to non-interactive workloads in a follow-up.

Reference Links

• NVIDIA-NeMo/Switchyard
• switchyard-server - crates.io
• Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard — Official breakdown of the routing philosophy and four-category classification
• NVIDIA Blog - Nemotron Lightning and NeMo Switchyard — Includes mention of deployment examples
• Fastino x NVIDIA Collaboration — Domain-specialized variants of Nemotron 3.5 Lightning
• Building a Team AI Coding Environment with Open-Weight Models (written as of 2026-08-08)