I tried replacing model routing with TypeSafe (Jev) to see how much faster and cheaper it would get

This is Morinaga from Classmethod Malaysia.

In a previous article, "Verifying NeMo Switchyard with a First Touch," when testing the llm-routing method of NVIDIA's LLM routing infrastructure NeMo Switchyard, it was reported that the choice of classifier (the component that determines which model to route to) led to a trade-off between "fast but expensive" and "cheap but slow."

In this article, I tested the hypothesis that "replacing the classifier component with the Choice primitive of TypeSafe, a structured decision-making specialized model that does not generate text, could resolve this trade-off entirely," by actually calling the TypeSafe API.

The Problem with Model Selection in the Original Article

According to the original article, NeMo Switchyard's llm-routing works by summarizing the last 4 turns of conversation, classifying them into 4 levels — simple / medium / complex / reasoning — using a classifier model, and then routing the request to the corresponding tier (from weaker to stronger models). It is a fail-open design that automatically falls back to the default tier if classification fails.

The following results were reported depending on which classifier was used.

In other words, using an LLM that allows reasoning is fast but drives up costs, while switching to an LLM that can disable reasoning reduces costs but significantly worsens latency — they were stuck in a "binary choice within the same LLM." The root cause is that, even though the classifier should be a lightweight task of simply "choosing one of 4 options," it gets dragged along by the LLM's inherent text generation and reasoning mechanisms.

What is TypeSafe?

TypeSafe is an AI platform specialized in structured decision-making. Its flagship model "Jev" is positioned as "the first System One Model," and the official documentation describes it as "unstructured state in, typed probabilistic decisions out."

The key point is that rather than generating text and then parsing it like a typical LLM, it returns typed values and probability distributions directly from the sampling layer. There are 3 available primitives:

• Choice: Selects one option from a list of choices (with probability/confidence)
• Score: Scores along a spectrum based on a rubric
• Noul: Judges whether a statement is true or false as a value from 0 to 1

This Choice is designed almost identically to the classifier's job in NeMo Switchyard — "receiving input and selecting one option from a predefined set" — which maps directly to "classifying into simple/medium/complex/reasoning."

The pricing is $0.042 per 1 million input tokens, with output tokens being free. The official blog claims end-to-end latency of 70–500ms, and 20–200x faster than comparable LLMs.

After signing up for the waitlist, I was able to access it right away, so let's test it!

Actually Trying It

Scope of Verification

What I did this time was to directly call the TypeSafe API and measure the latency, cost, and classification results of 4-tier classification using Choice on its own. This article does not cover actually integrating Jev as the classifier within NeMo Switchyard (embedding it into Switchyard).

After a brief investigation, I found that Switchyard's classifier is designed to be configured as a target that speaks a specific format such as OpenAI Chat Completions, and it was not possible to directly specify TypeSafe's /v1/systemone as the base_url. Connecting them would require inserting an adapter in between or incorporating Switchyard as a library and calling it directly from within the routing logic, which turned out to be more effort than expected, so I decided to defer the Switchyard integration to a future article. Therefore, the latency and cost figures below are measured values from calling TypeSafe directly on its own, and will differ from the latency when actually used via Switchyard (there should be additional overhead from the routing layer and adapter). Please keep this in mind when reading.

Verification Approach

Mimicking NeMo Switchyard's llm-routing, I sent requests to TypeSafe's actual API (POST https://api.typesafe.ai/v1/systemone) that pass "text summarizing the recent conversation" and classify it into the 4 tiers simple / medium / complex / reasoning using Choice.

For each tier, I prepared one representative conversation summary, and mapped the criteria directly to NeMo Switchyard's 4 classifications as follows:

TIER_CRITERIA = {
    "simple": "A simple one-question-one-answer exchange or short confirmation response. No tool execution or multi-step planning required.",
    "medium": "Requires explanation or light advice, but does not involve multi-step investigation or tool integration.",
    "complex": "Requires multi-step investigation or execution, such as cross-referencing multiple files, logs, and tool execution results.",
    "reasoning": "Requires deep logical reasoning and simultaneous consideration of multiple constraints, such as comparing trade-offs or making design decisions.",
}

The request itself is very simple:

payload = {
    "state": state_text,  # Text summarizing the recent conversation
    "model": "jev-latest",
    "questions": {
        "tier": {
            "type": "choice",
            "instructions": "Based on the recent exchanges in this conversation, determine the difficulty tier of the response needed next.",
            "criteria": TIER_CRITERIA,
        }
    },
}

I called the API 10 times each for 4 patterns (conversation samples expected to result in simple/medium/complex/reasoning), for a total of 40 calls, recording latency, classification results, confidence, and cost.

Results

All 40 calls succeeded, and the classification results for all 4 patterns matched the expected tier perfectly, 10/10 times.

Placing these alongside the classifier actual measurements from the original article:

The latency (median-based 0.643–0.674 seconds, including average values 0.652–0.690 seconds) was approximately 3x faster than the Gemini 3.5 Flash classifier that allows reasoning, and approximately 10–11x faster than the DeepSeek V4 Flash classifier with reasoning disabled. Regarding cost, a direct comparison is difficult because the NeMo Switchyard side is measured in "per session" units while this measurement is "per call," but it was confirmed that the cost per single judgment differs by orders of magnitude.

Points to Note

There are several points worth noting about the results of this verification.

• Medium tier has lower confidence: While simple/complex/reasoning were classified without hesitation at confidence 1.0, medium returned slightly lower values of 0.57–0.67. This straightforwardly reflects in the probability distribution that borderline cases tend to be more uncertain, which I felt was an advantage not commonly seen in LLM-based text classification. It seems easy to operate with a confidence threshold, such as "fall back to the safer tier if confidence is low."
• Not integrated into Switchyard: As mentioned, the figures here are measured values from calling TypeSafe directly on its own. They are not latency figures for running it as the classifier in NeMo Switchyard, so please do not use them as-is for production deployment figures.
• This verification is not a comprehensive benchmark of classification accuracy: Since I prepared one straightforward conversation sample for each of the 4 tiers, this is not an accuracy verification with a large number of borderline cases. It is also important to note that TypeSafe's selling point is not accuracy itself but rather speed and cost. Even in the vendor's own published workflow benchmarks, Jev at 76.0% does not clearly outperform other models compared to GPT-5.6 Luna at 76.1% or DeepSeek V4 Flash at 76.8%. Furthermore, in independent verification by a third party (Every), the gap widens slightly, with Jev at 67.8% versus the best comparison at 74.1%. The reality seems to be "overwhelmingly superior in speed and cost, but accuracy is on par or slightly inferior," so for use cases where accuracy could be a bottleneck, additional thorough verification is recommended.

Summary

Against the trade-off in the original article's llm-routing — "lightening the classifier worsens latency, while allowing reasoning drives up costs" — by replacing it with the Choice primitive of TypeSafe (Jev), a structured decision-making specialized model that does not generate text, the measured results showed it to be faster than either configuration in the original article while also significantly reducing costs.

As hypothesized, an architecture like Jev's that "does not generate reasoning tokens" appears to be an approach that resolves the speed and cost trade-off inherent in LLM-based classifiers from entirely outside the playing field. On the other hand, it should be kept in mind that TypeSafe itself is still in early access, that this verification is a standalone benchmark of TypeSafe and not figures from actually integrating it into Switchyard, and that it is not a comprehensive accuracy benchmark either.

For those running multi-model routing infrastructure and struggling with classifier speed and cost, it seems worth keeping as one of the options. In the next article, I plan to go further by actually integrating it into NeMo Switchyard as a library and running Jev as the classifier.