I tried implementing a library that integrates TypeSafe (Jev) into NeMo Switchyard as a classifier

This is Morinaga from Classmethod Malaysia.

In the previous article "Tried replacing model routing with TypeSafe (Jev) to see how much faster and cheaper it would get", I tried calling TypeSafe's Choice primitive directly on its own. I got a good sense that this could resolve the trade-off that model routing had been struggling with — fast but expensive, or cheap but slow. However, I ended up stopping short of actually integrating it into Switchyard's libsy.
Leaving it at that felt unsatisfying, so I went ahead and implemented a library that actually integrates Jev as a classifier into Switchyard.

Looking at the Switchyard repository, I found Issue #723, which contains a Feature Request almost identical to what I had in mind.

• Proposal: Integrate TypeSafe's System One into Switchyard's routing
• Benchmark listed in the issue: TypeSafe accuracy 10/10, average 281ms, p95 375ms; GPT-5.6-sol accuracy 10/10, average 1653ms, p95 2576ms
• Implementation requirements: Send normalized conversation state to TypeSafe as a type-safe Choice. Define stable labels and semantic criteria for each target. Route only when confidence is high; fall back when it's low
• Security requirements: Retrieve credentials from environment variables; do not store them in TOML configuration or logs
• Architectural constraints: "switchyard-libsy maintains an I/O-free design. Implement as an external HTTP judgment step or as a runner-owned classification provider; do not make direct HTTP calls from libsy"

So this time, I decided to implement following the requirements outlined in this issue.

Investigating Switchyard's Architecture

First, I read through the libsy (the routing algorithm core) code to get a concrete grasp of the constraints.

Reading libsy, it is indeed designed to be I/O-free. Algorithm::route simply calls driver.call_model(request, models), and the actual HTTP calls are handled by the libsy-llm-client implementation.

Another challenge is the types of wire formats that the existing libsy-llm-client's Backend expects.

// crates/libsy-llm-client/src/backend.rs
enum Backend {
    OpenAiChat,
    OpenAiResponses,
    Anthropic,
}

TypeSafe's POST /v1/systemone doesn't match any of these formats. It's neither a chat completion nor a responses API — it uses its own request/response format of {state, model, questions}. I also considered adding TypeSafe as a fourth wire format to switchyard-translation, but that would require writing a new codec supporting both buffering and streaming, and given that TypeSafe calls aren't streamed in the first place (the judgment is returned immediately in a single request), it seemed like a poor fit.

So I looked at the existing llm-routing equivalent implementation, LlmTaskClassifier in algorithms/llm_class.rs. This implementation constructs a chat completion request with structured output enforced via JSON Schema targeting a model in the judge category, sends it via driver.call_model, and then parses the returned text as JSON to make a policy decision.

Implementation Approach

I implemented the following to satisfy both options indicated in the issue (external HTTP judgment step and runner-owned classification provider).

libsy side: I/O-free port and classification logic

First, I defined a trait in libsy with no HTTP dependencies whatsoever. This serves as the entry point for the external HTTP judgment step mentioned in the issue.

// crates/libsy/src/algorithms/util/typesafe_provider.rs
#[async_trait]
pub trait TypeSafeProvider: Send + Sync {
    async fn classify(
        &self,
        input: TypeSafeClassifierInput,
        options: &[TypeSafeOption],
    ) -> Result<TypeSafeVerdict, TypeSafeProviderError>;
}

libsy only depends on this trait and holds no implementation that actually makes HTTP calls. TypeSafeOption (label + description), TypeSafeClassifierInput (question + context), and TypeSafeVerdict (label + confidence) are all plain data types.

On top of that, I implemented TypeSafeTaskClassifier, which has the same skeleton as LlmTaskClassifier.

// crates/libsy/src/algorithms/type_safe_class.rs
pub struct TypeSafeClassifierConfig {
    pub options: Vec<TypeSafeOption>,
    pub question: String,
    pub base_threshold: f64,
    pub default_target: Category,
    pub classify_trigger: ClassifyTrigger,
    pub message_hash_fallback: bool,
    pub recent_turn_window: Option<usize>,
}

When the provider (TypeSafeProvider) is called and the confidence falls below base_threshold, or the provider returns an error, or the returned label cannot be resolved to any Category, all such cases are treated as indeterminate and routed to the DefaultCategoryClassifier side. It is a fail-open design that never returns Err no matter what happens.

The conversation windowing logic (trim_messages/task_messages) and the AffinityRouter construction logic were reused as-is from the existing llm_class.rs by making them pub(crate), avoiding duplicate implementations.

switchyard-typesafe-client: A new crate that actually makes HTTP calls

The runner-owned classification provider mentioned in the issue is this new crate. I added switchyard-typesafe-client as a sibling crate to libsy-llm-client.

// crates/switchyard-typesafe-client/src/lib.rs
#[async_trait]
impl TypeSafeProvider for TypeSafeHttpClient {
    async fn classify(
        &self,
        input: TypeSafeClassifierInput,
        options: &[TypeSafeOption],
    ) -> Result<TypeSafeVerdict, TypeSafeProviderError> {
        // POST { state, model, questions: { "route": { type: "choice", instructions, criteria } } }
        // to {base_url}/v1/systemone,
        // and map answers.route.{choice, confidence} into TypeSafeVerdict
        ...
    }
}

The request/response format is identical to what was verified in the previous article. The API key is designed to be retrieved only from environment variables via TypeSafeHttpClient::from_env(variable_name), and cannot be written in TOML configuration files at all.

switchyard-runner: Wiring into TOML configuration

Finally, I wired things up in switchyard-runner so that this feature can be used from the actual deployment configuration (TOML).

[type_safe_client]
api_key_env = "TYPESAFE_API_KEY"

[routes.switchyard]
type = "type_safe_classifier"
default_target = "efficient"
base_threshold = 0.6
question = "Which model tier does this conversation need?"

[routes.switchyard.options]
capable = "Needs multi-step reasoning, ambiguous instructions, or high-stakes correctness."
efficient = "A short, well-specified request."

[routes.switchyard.models]
capable = ["gpt-5.5"]
efficient = ["gpt-5.5-mini"]
any = ["gpt-5.5", "gpt-5.5-mini"]

If a route with type = "type_safe_classifier" is defined without a [type_safe_client] table, it will be rejected as a configuration error at startup. Validations such as requiring each key in options to also have a matching group name in models were implemented by sharing the existing llm_classifier configuration validation logic.

Benchmark

The benchmark in the previous article measured direct calls to the TypeSafe API from Python (urllib). This time, I ran the same benchmark again after actually integrating it into Switchyard. The goal was to confirm that speed had not degraded as a result of wrapping it in a library and switching to a Rust implementation.

Verification Method

The conditions are exactly the same as last time. I prepared 4 patterns modeled after NeMo Switchyard's llm-routing (conversation summaries expecting simple/medium/complex/reasoning respectively), and called TypeSafeHttpClient::classify 10 times each, for a total of 40 calls.

Results

All 40 calls matched the expected tier (accuracy 40/40).

※ The p95 for simple (0.747s) is an outlier from just 1 out of 10 runs. It was the very first call made by the process that ran the benchmark, so it likely incurred the cost of TLS handshake and connection establishment. The other 9 runs were in the 0.2–0.4 second range and had almost no impact on the median/mean figures.

Comparing against the direct API measurements from the previous article and the TypeSafe standalone benchmark figures listed in Issue #723 (average 281ms, p95 375ms):

The results this time (0.25–0.32s) are considerably faster than the measurements from the previous article (0.64–0.69s). This is not a reflection of implementation quality but is simply due to differences in the client execution environment (Rust + reqwest vs. Python + urllib) and network/API conditions at each point in time; these are not benchmarks with conditions aligned for strict comparison between the two. However, the values are close to the TypeSafe standalone benchmark listed in Issue #723 (average 281ms), and I believe this confirms that the library integration and switch to a Rust implementation have not compromised TypeSafe's inherent speed. (Cost was not calculated this time, but since the state text used in this benchmark is identical to the previous article, the cost should be considered the same.)

Summary

I performed actual connectivity verification with the TypeSafe API and ran a benchmark under the same conditions as the previous article, confirming that Jev's inherent speed (median 0.25–0.28s, classification accuracy 40/40) has not been compromised.
When operating an LLM routing infrastructure like NeMo Switchyard, I now have a strong sense that Jev could be a very compelling option.

I submitted the PR and all sorts of interesting discussions have started, which is fun.