Automatically routing Slack agent requests with NeMo Switchyard and a post-trained judge model

While building an agent that routes requests in NeMo Switchyard, I documented my experience of dramatically improving the weak/strong allocation from 1:9 to 3:7 by replacing the routing model with a post-trained version.

嶋田怜央成 / Reona Shimada

2026.08.28

This page has been translated by machine translation. View original

Introduction

Hello, I'm Shimada from Classmethod's Manufacturing Business Technology Department.

Last time, I set up a Slack resident agent with NemoClaw/NemoHermes.

This time, I'm inserting NeMo Switchyard in front of it to route requests to different models depending on the task.

The requests coming in from the team's Slack are wide-ranging.
Some requests, like "list the in-progress issues by assignee," are complete once you retrieve and organize the necessary information, while others, like "propose next quarter's resource allocation with trade-offs," require careful, deliberate thinking — and both arrive at the same entry point.
Using an expensive model for the former is wasteful, and routing the latter to a cheap model will almost never yield the expected response.

So I placed a gatekeeper role that reads each request first and decides where to route it.
In this configuration, only this gatekeeper runs locally on the DGX Spark.

The reason for keeping the gatekeeper local is because this role is the one that reads the contents of every single request first.
The executor roles can be sent externally depending on the task, but the routing decision itself is completed on-premises.

Note that at the time of writing this article, Switchyard is at 0.2.0 and vLLM is at v0.27.1.
Since Switchyard is under active development, the setup caveats described below may be resolved in future versions.
Please check the official repository for the latest information.

A vanilla model can't serve as the gatekeeper

Switchyard's LLM classifier approach works by giving the model a judgment prompt and a structured JSON schema, asking it to answer "which tier is sufficient for this request?"
Initially, I had the same local model used for the weak tier (Nemotron 3.5 Lightning) double as the gatekeeper.
The one-off tests during setup passed, so I let the team use it for a few days.

The result: the /v1/stats distribution looked like this.

tiers.weak.calls    = 23     (7.6%)
tiers.strong.calls  = 279   (92.4%)

Over 90% was routed to strong, making the router essentially pointless.
Three issues were compounding.

The first was that the actual user request was not reaching the gatekeeper.
The classifier reads a summary JSON of the original request, but this summary has a max_request_chars limit (default 16,000 characters), and anything beyond that is truncated from the end.
The beginning of the summary is Hermes's system prompt.
As seen last time, this alone is 16,000 tokens (about 50,000 characters), so every time it would cut off mid-system-prompt, and the actual request text never reached the gatekeeper.
The evidence was that the classifier prompt in stats was consistently stuck at an average of 4,100 tokens (equivalent to 16,000 characters).

The second was that the vanilla model lacked the vocabulary for the gatekeeper role.
It could comply with the format, but it had no sense of "the probability that a lighter model can complete this task."
A similar phenomenon is described in detail in an article written by my colleague Morishige: when using vanilla Lightning directly as the gatekeeper, the judgment category match rate was 17.3%, with nearly every case falling into "unable to judge."

The third was the maintainability of the preset.
Reading the profile: general implementation, the tier mapping only covers SIMPLE→weak; MEDIUM, COMPLEX, and even the fallback on indeterminate judgments all go to strong.
It was designed so that any ambiguity at all would fall through to strong.

"A gatekeeper that isn't reading the request, returning ambiguous judgments without the right vocabulary, and routing everything ambiguous to strong" — a three-layered problem.
The first and third can be fixed with configuration and a translation layer, but the second remains a model problem.

Capability Card and p_solve

I replaced the gatekeeper with a model post-trained (LoRA SFT) specifically for judging.
The aforementioned article covers the procedure for building a judge for coding tasks; this time I used a judge model trained on the same methodology but on this team assistant's traffic (Backlog searches, team RAG, Slack requests).
The base is Nemotron 3.5 Lightning, and after merging it weighs 21.6GB in NVFP4.

This judge's determination is not "is this task difficult?"
It is designed as a capability forecast: "what is the probability that the current agent configuration can complete this task end-to-end?"

The system prompt includes a Capability Card — a document describing the skills, tools, and permissions available to the agent; judgments are made against this Card
Output is structured JSON: {crux, primary_rule, capability_boundary, p_solve}. crux is "the hardest requirement that determines success," primary_rule is "the rule ID behind the judgment," capability_boundary is "the judgment category," and p_solve is "the probability of completing it in one shot"
Routing uses a threshold approach. The baseline is 0.70, with 0.10 added per level of uncertainty (within capability 0.70, uncertain 0.80, outside capability 0.90). If p_solve meets or exceeds the threshold, route to weak; otherwise route to strong
When the environment's capabilities change, rewrite the Card rather than retrain the model

{
  "crux": "Sort each row descending, then for each column take the maximum value across rows and sum those maxima.",
  "primary_rule": "SUP-3",
  "capability_boundary": "supported",
  "p_solve": 0.88
}

That last point is the key to this design.
Since skills are expected to grow over time, a design requiring retraining every time they do is not operationally sustainable.

Spinning up a dedicated vLLM for judging

I serve the judge via vLLM.
Since judgments are short and low-parallelism, an 8K context is sufficient.

services:
  vllm-judge:
    image: vllm/vllm-openai:v0.27.1-aarch64
    ports:
      - "127.0.0.1:8002:8000"
    volumes:
      - ${HOME}/models/judge-router:/model:ro
    command:
      - --model
      - /model
      - --served-model-name
      - judge
      - --max-model-len
      - "8192"
      - --gpu-memory-utilization
      - "0.22"
      - --kv-cache-memory
      - "4294967296"     # KV fixed at 4GiB
      - --trust-remote-code
      - --moe-backend
      - marlin
      - --mamba-backend
      - flashinfer

I hit one snag with the memory settings.
Running with just --gpu-memory-utilization 0.25 allocated 12GiB to the KV cache, leaving less than 3GB free in unified memory.
So I pinned KV to 4GiB with --kv-cache-memory, but then it wouldn't start.

ValueError: Free memory on device cuda:0 (39.19/121.69 GiB) on startup is less than
desired GPU memory utilization (0.92, 111.95 GiB).

Specifying --kv-cache-memory causes --gpu-memory-utilization to revert to its default of 0.92, resulting in a situation where the KV is pinned but the startup free-memory check still runs against the 0.92 baseline and fails.
The correct approach is to specify both explicitly.
With this configuration, the judge's actual consumption stays within 16.3GiB for weights and 4GiB for KV.

There's one more caveat with unified memory.
When starting vLLM on DGX Spark, you must not let other processes change their model load state during initialization.

AssertionError: Error in memory profiling. Initial free memory 82.25 GiB,
current free memory 92.36 GiB. This happens when other processes sharing
the same container release GPU memory while vLLM is profiling ...

This happened because Ollama, which was co-residing, auto-unloaded its model after 5 minutes of idle — and that timing coincided with vLLM's initialization.
On a dGPU, another process freeing RAM has no effect on VRAM, but with unified memory they are directly linked.

Setting up Switchyard

Installation

Running uv tool install --python 3.10 "nemo-switchyard[cli]" as described in the Getting Started with Switchyard guide stalled at dependency resolution.
As of version 0.2.0 on pip at the time of writing, there are four discrepancies between the documentation and the implementation:

Python 3.12 or higher is required (the documentation says 3.10)
The routes.toml shown in the documentation is the format for the Rust-based switchyard-server (via cargo); the pip CLI's switchyard serve accepts a YAML bundle via --routing-profiles
PyYAML is required to read YAML files but is not declared as a package dependency
The serve command requires the server extra (fastapi, uvicorn)

The two missing dependencies only became apparent when actually running the tool.

ModuleNotFoundError: No module named 'yaml'
ModuleNotFoundError: No module named 'uvicorn'

The command to resolve everything at once is as follows:

uv tool install --python 3.12 --with pyyaml "nemo-switchyard[cli,server]"

Making Switchyard speak the judge's contract

Here, simply pointing classifier.model at the judge is not enough.

The pip version of Switchyard 0.2.0's classifier strictly parses a fixed JSON schema per profile ({recommended_tier, confidence, abstain, ...}).
The verdict the judge returns ({crux, primary_rule, capability_boundary, p_solve}) is a different contract.
While the judgment prompt can be swapped out, the response schema and threshold logic cannot.

So I inserted a small proxy (shim) between the two to translate the contracts.

The shim is about 300 lines using only the Python standard library, and does four things:

Restore the judgment material: Extract the message list from the summary JSON, discard the system prompt, and reassemble "the initial request, recent exchanges, and the latest request" to fit within the judge's 8K context
Call the judge: Pass the Card as the system message and the verdict schema as response_format, calling at temperature 0
Validate and threshold: Validate the verdict's consistency (rule-to-boundary correspondence, p_solve range), compare against the threshold, and determine weak vs. strong. Invalid verdicts or judge failures fail-open to strong
Translate the contract: Pack the result into the JSON Switchyard expects. Returns values that always map to SIMPLE for weak and always to COMPLEX for strong (this neutralizes the third issue described earlier)

The core of the routing logic is just this:

STEPS = {"supported": 0, "uncertain": 1, "unmatched": 1, "unsupported": 2}
threshold = 0.70 + STEPS[verdict["capability_boundary"]] * 0.10
route = "weak" if verdict["p_solve"] >= threshold else "strong"

Since the judge can only evaluate text, requests containing image or PDF attachments bypass the judge entirely and go straight to strong.

routing-profiles.yaml

Since the YAML bundle schema is not documented, I reverse-engineered it from the loader implementation inside the wheel.
The final form is as follows:

defaults:
  timeout_secs: 300

routes:
  switchyard:                      # inbound model id (Hermes calls with model="switchyard")
    type: deterministic            # LLM classifier approach
    profile: general
    fallback_target_on_evict: strong  # fallback when a tier fails (required field)
    classifier:
      model: judge-shim
      base_url: http://127.0.0.1:8003/v1
      api_key: local-noauth        # required field even without auth (dummy value)
      fail_open: true              # on shim or judge failure, fall through to strong
      max_request_chars: 200000    # leaving this at default 16000 means the request never reaches the gatekeeper
      recent_turn_window: 4
    weak:
      model: accounts/fireworks/models/deepseek-v4-flash-0731
      base_url: https://api.fireworks.ai/inference/v1
      api_key: ${FIREWORKS_API_KEY}
    strong:
      model: accounts/fireworks/models/kimi-k3
      base_url: https://api.fireworks.ai/inference/v1
      api_key: ${FIREWORKS_API_KEY}

I discovered that fallback_target_on_evict and the classifier's api_key are required fields only after seeing startup errors.

error: invalid route bundle: switchyard.fallback_target_on_evict must be a non-empty string
error: invalid route bundle: switchyard.classifier.api_key must be a non-empty string

The weak and strong tiers are also automatically registered as standalone passthrough targets.
Querying /v1/models shows the route name switchyard alongside the two tier backends and Fireworks's catalog, so if you want to pin a specific task to always use strong, you can specify model="strong" directly.

The choice of Fireworks's DeepSeek V4 Flash for the weak tier also has a calibration rationale.
Since the same V4 Flash served as the teacher during judge training, the p_solve calibration directly corresponds to the actual capability of the executor.

Switching NemoHermes's inference target

I redirect Hermes's inference target to Switchyard.
This was the most time-consuming step in the entire setup.

Pitfall: allowed ports for the loopback bridge

After setting up Switchyard at localhost:4000 and re-onboarding, the router inside the sandbox tried to connect to the container's own localhost:4000 and failed.

[sandbox] [INFO] [openshell_router] routing proxy inference request (streaming)
  endpoint=http://localhost:4000/v1 method=POST path=/v1/chat/completions
[sandbox] [OCSF] NET:FAIL [LOW] inference.local:443

Reading the NemoClaw source, the bridge logic that rewrites loopback URLs to the sandbox-accessible host.openshell.internal only activates when the port is 11434, 11435, or 8000.
Port 4000 was not covered.

The fix was to swap ports: judge on 8002, Switchyard on 8000.
Once the rewrite kicks in, the endpoint in the same log changes to http://host.openshell.internal:8000/v1.

Pitfall: dual listen address requirement

The onboarding process validates localhost:8000 from the host side, while the sandbox accesses port 8000 on host.openshell.internal (Docker bridge at 172.18.0.1).
This means the service needs to listen on both addresses.
Binding to 0.0.0.0 would handle both at once, but that would expose an unauthenticated router to the internal LAN, so I opted against it.
Instead, I bind to the bridge IP and forward the loopback side via socat.

switchyard serve --routing-profiles routing-profiles.yaml --host 172.18.0.1 --port 8000
socat TCP-LISTEN:8000,bind=127.0.0.1,fork,reuseaddr TCP:172.18.0.1:8000

The reason this issue doesn't arise when using Ollama as a provider is that NemoClaw itself resolves it by setting up an authenticated proxy at 0.0.0.0:11435.
With a custom endpoint, you have to handle it yourself.

Pitfall: config changes causing delayed crashes

When expanding the context window, you might be tempted to run this inside the sandbox:

# Do NOT do this
nemohermes team-assistant exec -- hermes config set providers.compatible-endpoint.models.switchyard.context_length 262144

The command succeeds and the setting is applied.
However, the config integrity hash at /sandbox/.hermes/.config-hash is not updated.
The startup script performs hash verification first, so the next time the gateway restarts, verification fails and it enters a crash loop.

[SECURITY] Hermes config hash does not match persisted inputs
[SECURITY] HERMES_MCP_CONFIG_DRIFT: MCP intent cannot be matched to the persisted gateway state

What makes this insidious is that nothing happens immediately after making the change.
It keeps running in a broken state and only surfaces the next time it restarts.
The correct approach is to use the host-side command, which updates both the setting and the hashes in a single transaction.

export PATH="$HOME/.local/bin:$PATH"   # without openshell in PATH, you get ENOENT
nemohermes team-assistant config set --key <dotpath> --value <value> --restart

Recovery requires a rebuild (restart and recover won't fix it).
To make things worse, the state backup cannot be taken while the crash loop is active, and the rebuild itself aborts — so you need to stop the loop with docker update --restart=no, manually back up with docker cp, and then recover with rebuild --force.

I've adopted an operational rule: for any command that directly modifies config files inside the sandbox, always check whether there's an equivalent entry point on the host side first.

Verification

The startup banner shows the gatekeeper and both tier assignments.

  switchyard  ready  →  switchyard

  profiles  ▶ switchyard  (default)
                llm-classifier
                strong      accounts/fireworks/models/kimi-k3
                weak        accounts/fireworks/models/deepseek-v4-flash-0731
                classifier  judge-shim
                profile     general

After switching to the judge, each judgment can be traced individually through the verdict log.
This is the biggest difference from the vanilla classifier — crux explains why a request was routed the way it was.

{"route": "weak", "latency_ms": 1353.5, "capability_boundary": "supported",
 "primary_rule": "SUP-1", "p_solve": 0.77,
 "crux": "The request asks for a list of in-progress issues in project TASKHUB
          grouped by assignee, which requires a read-only Backlog API call and
          formatting of the result."}

{"route": "strong", "latency_ms": 1813.8, "capability_boundary": "unsupported",
 "primary_rule": "LIM-1", "p_solve": 0.31,
 "crux": "The request asks for a 3-month cross-project issue trend analysis and
          a resource allocation plan for the next period. The main deliverable
          is deliberative analysis and a proposal, not a retrieval."}

The issue list request is complete with a Backlog skill read and formatting, so p_solve 0.77 clears the 0.70 threshold and routes to weak.
The trend analysis and resource allocation proposal is fundamentally about reasoning rather than retrieval, so it's judged outside the capability boundary, falls short of the 0.90 threshold, and routes to strong.
In E2E tests through Switchyard, the former was answered by DeepSeek V4 Flash in 7 seconds, and the latter by Kimi K3.

Judgment latency, including reading the Card (~2,600 tokens) every time, is p50 1.8 seconds and p95 2.5 seconds.
This is on par with the average 1.7 seconds when using vanilla Lightning as the gatekeeper — meaning only the quality of judgment improved.

The distribution also changed dramatically.
Here is the breakdown for 140 judgments after introducing the judge:

	Vanilla Lightning	Post-trained judge
weak	23 cases (7.6%)	93 cases (66.4%)
strong	279 cases (92.4%)	47 cases (33.6%)

The breakdown by boundary was: supported 93, unsupported 37, unmatched 8, uncertain 2.
This matches the intended design: "tasks within capability (Backlog search/formatting, web search) go weak; tasks that are fundamentally about reasoning go strong."

Closing

The value of inserting a router was not what I originally expected — "it will be cheaper."
What actually mattered was being able to see in numbers exactly what was flowing where.
Without the stats showing that 92.4% bias, I would never have noticed that the gatekeeper wasn't reading the requests, or that a vanilla model lacks the vocabulary for the job.

Replacing the gatekeeper with a post-trained model feels worth the effort.
Designing judgment not as "guessing task difficulty" but as "a capability forecast for whether this agent configuration can complete the task" made it possible to trace each judgment individually with its reasoning.
And when capabilities change, you can follow along by rewriting the Capability Card rather than retraining the model.

The effect of keeping a single unified entry point is also significant.
Since we started operating, the executor models have been swapped out several times, but the Hermes configuration has never been touched.
Because model="switchyard" stays constant, swapping means editing a few lines in the route definition and restarting the router.

NemoClaw's custom endpoint has two underlying constraints to be aware of: the bridge only allows ports 11434/11435/8000, and the service must listen on both localhost and the bridge IP. Not knowing these will leave you stuck when switching.
Config changes made from inside the sandbox also silently cause delayed crashes.

Next time, I'll have the agent manipulate Backlog using a mechanism called skills.
The topics will be designing write access while blocking deletes, and iteratively refining skills based on real-world operational feedback.