I tried replacing the Classifier of NeMo Switchyard used for LLM routing in NemoHermes with TypeSafe's Jev and measured the results
This page has been translated by machine translation. View original
Introduction
Hello, I'm Shimada from Classmethod's Manufacturing Business Technology Department.
In a previous article, I placed NeMo Switchyard in front of a Slack resident agent (NemoHermes) and used a post-trained judge model to route requests as either weak or strong.
The judge takes a median of 1.8 seconds per decision.
Since Hermes calls Switchyard every time it invokes a tool, a single Slack message triggers 3 to 4 judgments, and that overhead adds to the response wait time.
This time, I'll try replacing this judge role with TypeSafe's Jev.
Jev is a model that doesn't generate text — it directly returns choices and confidence scores.
Recently, our colleague Morrissey implemented a library to integrate Jev as a classifier into Switchyard and submitted a PR, so I built it on DGX Spark and compared its judgments against the same agent configuration we use in actual operation.
Note that at the time of writing this article, TypeSafe's API is in early access (waitlist), and the returned model name is jev-1.13.0.
Switchyard is built from source on the PR branch (switchyard-server 0.3.0, commit b0ad365).
Since the PR contents may change before merging, please check PR #739 for the latest status.
TypeSafe and Jev
TypeSafe AI is an AI lab building models designed to make decisions within software.
On September 15, 2026, they announced a classification called System One models and its first model, Jev.
System One models read natural language input and return typed decisions with probabilities rather than generated text.
The official documentation provides three primitives:
- Choice: Selects one option from given choices and returns it with a confidence score
- Score: Returns a numerical score based on a rubric
- Noul: Returns the truth value of a sentence as a value from 0 to 1
Output is extracted directly from the sampling layer, so there is no step to parse generated text into JSON.
According to published figures, response time is 70 to 500 milliseconds, the input token price is $0.042 per million tokens, and output is free.
There is a single API endpoint: send state (the text to be judged) and questions (primitives and choices) to POST /v1/systemone.
The Switchyard judge's task — "is this request enough for weak?" — is exactly a two-option Choice.
The replacement here is swapping out a generative model doing the light work of classification with a model purpose-built for classification.
What changes with the replacement
While I'm not actually replacing this in production, let me outline what changes aside from speed.
The judgment input leaves the DGX.
In the previous configuration, the judge was the only component reading the content of all requests, so it was kept local.
Since Jev is a TypeSafe API, the text used for judgment (the beginning of the request and the latest user message) is sent to TypeSafe.
The executor was already going to Fireworks from the start, so the type of information leaving the system doesn't increase, but one more destination is added.
The reasoning behind judgments becomes unreadable.
The judge writes a crux (the hardest requirement in this request) in the verdict, so you could trace why each request was routed to strong.
Jev only returns a label, confidence score, and per-option probabilities.
Judgment criteria are now expressed in text rather than trained into the model.
The judge was a post-trained model with a Capability Card in the system prompt.
With Jev, the description text (criteria) for each option becomes the judgment standard directly.
When you want to change the criteria, you edit the text in the configuration file.
About the aforementioned PR
The PR adds a route type called type_safe_classifier to Switchyard.
The implementation is organized into three layers:
libsy(the routing core) contains an HTTP-independentTypeSafeProvidertrait and theTypeSafeTaskClassifierthat uses it- A new crate
switchyard-typesafe-clientholds the HTTP implementation that actually callsPOST /v1/systemone switchyard-runner(the layer that reads configuration and assembles routes) gains a[type_safe_client]table andtype_safe_classifierroute configuration
The judgment flow is as follows.
The initial request and the latest user message are flattened to plain text and sent as state, and the options and descriptions written in options are passed as Choice criteria.
If the returned confidence meets or exceeds base_threshold, the request is routed to the first target in the models group corresponding to that label.
When confidence falls below the threshold, when a label cannot be resolved, or when the API fails, all cases are routed to default_target.
This classifier does not error out a request on judgment failure.
The API key is only read from environment variables.
There is no field to write it in the configuration file, and it does not appear in logs.
Building
Jev support is included in the Rust version of switchyard-server.
Install rustup on DGX Spark (aarch64, Ubuntu 24.04) and build the PR branch.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
. ~/.cargo/env
git clone --branch feature/typesafe-classifier-routing --depth 1 \
https://github.com/cm-morrissey/Switchyard.git
cd Switchyard
cargo build --release --locked -p switchyard-server
ls -la target/release/switchyard-server
On the 20-core DGX Spark, the build completed in 36 seconds.
The resulting binary is 23 MB.
Writing routes.toml
Configuration for the Rust version is TOML.
It's written in three sections: llm_clients (connection targets), targets (models), and routes (routing).
schema_version = 1
[type_safe_client]
api_key_env = "TYPESAFE_API_KEY"
# base_url = "https://api.typesafe.ai" # default
# model = "jev-latest" # default
[llm_clients.fireworks]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"
[targets.weak]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"
llm_client = "fireworks"
[targets.strong]
id = "accounts/fireworks/models/kimi-k3"
llm_client = "fireworks"
[routes.switchyard]
id = "switchyard"
type = "type_safe_classifier"
default_target = "strong"
base_threshold = 0.6
question = "Please select which model tier is needed when a Slack internal assistant answers this request."
context_window = 262144
tool_calling = true
vision = true
[routes.switchyard.options]
weak = "Requests with clear procedures and well-defined completion conditions, such as short questions, terminology checks, translation and summarization, or searching and citing Backlog or internal wikis."
strong = "Requests requiring deep reasoning or multi-step work, such as design decisions and policy consultations, rubber-duck discussions, investigations cross-referencing multiple documents, code design or review, or requests where accuracy errors are costly."
[routes.switchyard.models]
weak = ["weak"]
strong = ["strong"]
any = ["strong", "weak"]
I set default_target to strong.
If a cheap model receives traffic when judgment fails, the failure becomes invisible to users.
The judge configuration also routed to strong on failure (fail-open), so this behavior is consistent.
I wrote the options descriptions in Japanese.
Since requests coming into Slack are in Japanese, I thought writing the judgment criteria in the same language would make them easier to read and revise later.
The next section's results will show how Jev handles Japanese criteria.
Starting up and testing connectivity
Set the keys as environment variables, then validate the configuration with --dry-run first.
export FIREWORKS_API_KEY="..."
export TYPESAFE_API_KEY="..."
./target/release/switchyard-server --config routes.toml --dry-run
# server OK: switchyard
According to the documentation, a configuration missing [type_safe_client] will fail at this validation stage because the type_safe_classifier route cannot be assembled.
At startup, I added the option to record routing events to a JSONL file.
./target/release/switchyard-server --config routes.toml \
--host 127.0.0.1 --port 8011 --routing-log-file routing.jsonl
Connectivity testing is done by sending an OpenAI-compatible chat/completions request with model set to the route id.
curl -sS http://127.0.0.1:8011/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"switchyard","messages":[{"role":"user","content":"Explain idempotent in one sentence"}]}'
The model in the response was accounts/fireworks/models/deepseek-v4-flash-0731, meaning weak.
The server log prints one line with the judgment result.
Model selected algorithm="type_safe_task_classifier" target=accounts/fireworks/models/deepseek-v4-flash-0731 confidence=1.0
Comparing judgments using the same requests as the judge
I prepared 13 scenarios modeled after requests that come into the team's Slack.
Eight expected weak (greeting, Backlog status check, terminology explanation, summarization, internal wiki search, meeting URL check, custom emoji existence, translation), and five expected strong (technical rubber-duck discussion, code design review, cross-referencing multiple documents, comparative architecture design, non-technical rubber-duck discussion).
For comparison, I also set up a configuration using type = "llm_classifier" on the same Rust switchyard-server, with the conventional judge (post-trained Nemotron Lightning running on vLLM) as the decision maker.
The llm_clients and targets are the same; only the judge in routes differs.
Both configurations ran through the 13 scenarios 3 times each, with executor generation cut off at max_tokens=16.
Judge results
The judge routed correctly on 36 of 39 cases.
The median response time including judgment was 2.31 seconds, with the judge verdict alone at a median of 1.47 seconds.
This is shorter than the 1.8 seconds mentioned at the beginning because these scenarios are short single-turn requests, and the judge's input is shorter than in real operation (which includes recent turns and tool results).
The 3 misses were: "meeting URL check" routing to strong 2 out of 3 times, and "Backlog status check" routing to strong 1 out of 3 times.
Although temperature is 0, judgments vary across runs.
Reading the verdict's crux, the meeting URL cases noted "no means to read calendars listed in the Capability Card."
This is not a judge error — it is a judgment consistent with the capabilities described in the Card.
Jev results
Jev routed correctly on all 39 of 39 cases.
The median response time including judgment was 1.17 seconds, about 1.1 seconds shorter than the judge.
Since response time includes executor generation, the generation time needs to be subtracted to isolate the difference in judgment.
/v1/stats reports executor call time and total request time separately, so I compared the difference (time spent on judgment and conversion).
| Judge | Judgment and conversion time (median) | Correct |
|---|---|---|
| judge (llm_classifier) | 1.55 seconds | 36/39 |
| Jev (type_safe_classifier) | 0.27 seconds (strong) to 0.42 seconds (weak) | 39/39 |
To observe the judgment time itself, I also measured the TypeSafe API directly across 13 scenarios × 10 runs.
| Scenario | Expected | Jev choice | confidence | Judgment time (median) | Input tokens |
|---|---|---|---|---|---|
| Greeting | weak | weak (10/10) | 0.99 | 0.492 sec | 505 |
| Backlog status check | weak | weak (10/10) | 1.00 | 0.530 sec | 510 |
| Terminology explanation | weak | weak (10/10) | 1.00 | 0.512 sec | 502 |
| Summarization | weak | weak (10/10) | 0.98–1.00 | 0.515 sec | 692 |
| Internal wiki search | weak | weak (10/10) | 1.00 | 0.510 sec | 519 |
| Meeting URL check | weak | weak (10/10) | 0.99 | 0.484 sec | 507 |
| Custom emoji existence | weak | weak (10/10) | 0.99 | 0.531 sec | 500 |
| Technical rubber-duck discussion | strong | strong (10/10) | 1.00 | 0.510 sec | 588 |
| Code design review | strong | strong (10/10) | 1.00 | 0.488 sec | 599 |
| Cross-referencing multiple documents | strong | strong (10/10) | 0.99 | 0.512 sec | 571 |
| Comparative architecture design | strong | strong (10/10) | 0.99–1.00 | 0.516 sec | 582 |
| Non-technical rubber-duck discussion | strong | strong (10/10) | 0.99 | 0.522 sec | 544 |
| Translation | weak | weak (10/10) | 1.00 | 0.521 sec | 521 |
All 130 judgments succeeded and matched expectations.
Judgment time was a median of 0.514 seconds, and p95 was 0.576 seconds.
Input tokens ranged from 500 to 692, costing approximately $0.000023 per call, and $0.003 for all 130 calls.
This measurement, calling one at a time from Python, re-establishes a TLS connection each time.
Since Switchyard reuses connections, the 0.27 seconds for strong in the table above is closer to the judgment time as seen from Switchyard, consistent with the 0.25 to 0.28 seconds in Morrissey's article.
Confidence was 0.98 or higher across all scenarios.
The "meeting URL check" that varied across judge runs was placed at weak by Jev with 0.99 confidence.
Since Jev doesn't have a Capability Card, it doesn't check "whether the agent has that capability."
It judges solely based on whether the request fits the criteria description: "requests with clear procedures and well-defined completion conditions."
Which is correct depends on whether the agent actually has the means to read calendars.
Both rubber-duck discussion scenarios were routed to strong with confidence of 0.99 or higher.
Since I wrote "rubber-duck discussion" on the strong side of the criteria, it was picked up directly.
In judge operations, the team had noted "requests with no completion condition are hard for weak to forecast, and weak tends to get routed there."
In these 13 scenarios the judge also routed rubber-duck discussions to strong, but being able to express judgment criteria in text makes it easier to fix how this type of request is handled.
Limitations discovered through use
Requests with images are passed to Jev with images replaced by the text "(image)".
Since Jev doesn't support image input, the PR converts image blocks to placeholders.
It's not possible to route based on image content, but the fact that an image is attached can still be used in judgment.
Judgment reasoning is not saved in JSONL.
The --routing-log-file records the selected model and token counts, while labels and confidence scores only appear in server INFO logs.
The PR review also notes that retaining probability distributions and recording judgment latency are future work items.
If you need to trace "why it was routed here" in production, you'll need to collect logs separately for the time being.
Also, Jev is currently early access.
API keys are distributed via waitlist invitations, and pricing and limits may change in the future.
Conclusion
Replacing the judge with Jev brought the median time the router spends on judgment from 1.55 seconds down to around 0.3 seconds, and all 39 of the 13 scenarios × 3 runs were routed as expected.
The cost per judgment is approximately $0.000023, which is not a reason to hold back the replacement.
In exchange, you need to accept that judgment input goes to TypeSafe and that judgment reasoning (crux) is no longer readable.
The first point was the reason the previous article kept the judge local, so actually making the switch requires a decision on whether it's acceptable to send judgment input outside.
Because the router was placed as an intermediate layer, swapping the judge required only one route in the configuration file and one environment variable.
With judgment criteria moving from trained Capability Cards to criteria text, changing the criteria becomes a configuration file edit.
Judgment speed and keeping input within DGX are incompatible in this configuration.
As a way to achieve both, we are also exploring using a small locally-trained classifier as the judge.
Together with how many seconds are saved by removing the reasoning (crux) from the judge's output, we will compare these in the next article.
References
- Implementing a TypeSafe (Jev) integration library for NeMo Switchyard (DevelopersIO)
- Testing how much faster and cheaper model routing becomes when replaced with TypeSafe (Jev) (DevelopersIO)
- feat(routing): add type_safe_classifier backed by TypeSafe's Jev (NVIDIA-NeMo/Switchyard PR #739)
- feat(routing): add a TypeSafe-backed classifier router (NVIDIA-NeMo/Switchyard Issue #723)
- Introducing System One Models & Jev (TypeSafe AI Blog)
- System One (TypeSafe AI Docs)
- Placing a post-trained judge on NeMo Switchyard to route Slack agent requests (DevelopersIO)