Automatically routing Slack agent requests with NeMo Switchyard and a post-trained judge model

Introduction

Hello, I'm Shimada from Classmethod's Manufacturing Business Technology Department.

Last time, I set up a Slack resident agent with NemoClaw/NemoHermes.

https://dev.classmethod.jp/articles/reona-01-dgx-spark-nemohermes-slack-agent/

This time, I'll insert NeMo Switchyard in front of it to route requests to different models.

The requests coming from the team's Slack cover a wide range.
Things like "list the in-progress issues by assignee" — where you just need to fetch the necessary information and format it — and things like "propose next quarter's resource allocation with trade-offs" — where the thinking itself is the main task — all come through the same entry point.
Using an expensive model for the former is wasteful, and routing the latter to a cheap model produces unusable results.

So I place a role at the front that reads each request and decides where to route it (the judge).
In this setup, only this judge runs locally on the DGX Spark.

The reason the judge is placed locally is that this role is the first to read the contents of every request.
The execution roles are sent out externally depending on the task, but the routing decision itself is completed on-premises.

Note that at the time of writing this article, Switchyard is at 0.2.0 and vLLM is at v0.27.1.
Since Switchyard is under active development, the setup caveats described below may be resolved in future versions.
Please check the official repository for the latest information.

A bare model cannot serve as the judge

Switchyard's LLM classifier approach works by giving the model a judgment prompt and a structured JSON schema, and having the model answer "which tier is sufficient for this request?"
Initially, I had the same local model used for weak (Nemotron 3.5 Lightning) double as the judge.
The standalone test during setup passed, so I had the team use it for a few days as-is.

As a result, the distribution from /v1/stats looked like this:

tiers.weak.calls    = 23     (7.6%)
tiers.strong.calls  = 279   (92.4%)

Over 90% was flowing to strong, making the router pointless.
Three causes were compounding each other.

The first was that the user's request was not included in the judgment material.
The classifier reads a summary JSON of the original request, but this summary has a max_request_chars limit (default 16,000 characters), and anything beyond that is truncated from the end.
The beginning of the summary is Hermes's system prompt.
As seen last time, this alone is 16K tokens (about 50K characters), so every time it was cut off mid-system-prompt, and the actual request text never reached the judge.
The trace was that the classifier prompt in stats was consistently stuck at an average of 4,100 tokens (equivalent to 16,000 characters).

The second was that the bare model lacks the vocabulary for the judge role.
While it can follow the format, it has no sense for estimating "the probability that this task can be completed with a lightweight model."
The same type of phenomenon is described in detail in the following article — when connecting bare Lightning directly as the judge, the judgment category match rate was 17.3%, with nearly all cases falling into "judgment not possible."

https://dev.classmethod.jp/articles/dgx-spark-nemotron-lightning-switchyard-classifier-finetune/

The third was the maintainability of the presets.
Reading the implementation of profile: general, tier routing maps only SIMPLE→weak; MEDIUM, COMPLEX, and the fallback on judgment failure all go to strong.
The design was that any ambiguity in judgment would fall through to strong.

It was a three-layer setup: "a judge that can't see the request returns ambiguous judgments without the right vocabulary, and ambiguous judgments go to strong."
The first and third causes can be fixed with configuration and a translation layer, but the second remains a model-level problem.

Capability Card and p_solve

I place a model post-trained (LoRA SFT) specifically for judgment in the judge role.
The article cited above includes a procedure for building a coding-oriented judge, and this time I used a judge trained with the same method on this team assistant's traffic (Backlog searches, team RAG, Slack requests).
The base is Nemotron 3.5 Lightning, and after merging it is 21.6 GB in NVFP4.

This judge's decision is not "is this task difficult?"
It is designed as a capability forecast: "what is the probability that this agent setup can complete this task end-to-end?"

• The system prompt includes a Capability Card: a document describing the skills, tools, and permissions available to the agent, and judgments are made against this Card
• The output is a structured JSON of {crux, primary_rule, capability_boundary, p_solve}. crux is "the hardest requirement that determines success," and p_solve is "the probability of completing it in one shot"
• Routing uses a threshold approach. Starting from a baseline of 0.70, it adds 0.10 depending on the confidence of the judgment (within capability: 0.70, uncertain: 0.80, outside capability: 0.90). If p_solve is at or above the threshold, it routes to weak; below, to strong
• When the environment's capabilities change, rewrite the Card rather than retraining the model

That last point is the key to this design.
In the third and fourth installments, skills will keep increasing, so a design requiring retraining every time would not be operationally sustainable.

Standing up a dedicated vLLM for judgment

I serve the judge with vLLM.
Since judgments are short text and parallelism is low, 8K context is sufficient.

services:
  vllm-judge:
    image: vllm/vllm-openai:v0.27.1-aarch64
    ports:
      - "127.0.0.1:8002:8000"
    volumes:
      - ${HOME}/models/judge-router:/model:ro
    command:
      - --model
      - /model
      - --served-model-name
      - judge
      - --max-model-len
      - "8192"
      - --gpu-memory-utilization
      - "0.22"
      - --kv-cache-memory
      - "4294967296"     # KV fixed at 4GiB
      - --trust-remote-code
      - --moe-backend
      - marlin
      - --mamba-backend
      - flashinfer

I ran into one snag with the memory settings.
Using only --gpu-memory-utilization 0.25 at startup caused the KV cache to take 12 GiB, leaving less than 3 GB of free unified memory.
So I tried fixing the KV to 4 GiB with --kv-cache-memory, but this time it wouldn't start.

ValueError: Free memory on device cuda:0 (39.19/121.69 GiB) on startup is less than
desired GPU memory utilization (0.92, 111.95 GiB).

Specifying --kv-cache-memory resets --gpu-memory-utilization to its default of 0.92, causing the KV to be fixed while the startup free-memory check still runs against the 0.92 baseline and fails.
The correct approach is to explicitly specify both.
With this configuration, the judge's actual consumption stays within 16.3 GiB for weights and 4 GiB for KV.

There is one more caveat with unified memory.
When starting vLLM on DGX Spark, you must not move other processes' model load state.

AssertionError: Error in memory profiling. Initial free memory 82.25 GiB,
current free memory 92.36 GiB. This happens when other processes sharing
the same container release GPU memory while vLLM is profiling ...

This happened because co-residing Ollama auto-unloaded its model after 5 minutes of idle, coinciding with the initialization.
On a dGPU, another process releasing RAM doesn't affect VRAM, but with unified memory they are directly linked.

Standing up Switchyard

Installation

Following the Getting Started with Switchyard guide and running uv tool install --python 3.10 "nemo-switchyard[cli]" stalled at the dependency resolution stage.
At the time of writing, the pip release 0.2.0 has the following four discrepancies between the documentation and the implementation:

• Python 3.12 or higher is required (the documentation says 3.10)
• The routes.toml described in the documentation is for the Rust-based switchyard-server (cargo) format; the pip CLI's switchyard serve accepts a YAML bundle via --routing-profiles
• PyYAML is required to load the YAML, but it is not declared as a package dependency
• serve requires the server extra (fastapi, uvicorn)

The two missing dependencies were not apparent until I actually ran it.

ModuleNotFoundError: No module named 'yaml'
ModuleNotFoundError: No module named 'uvicorn'

The command that resolves everything at once is as follows:

uv tool install --python 3.12 --with pyyaml "nemo-switchyard[cli,server]"

Getting Switchyard to speak the judge's contract

Here, simply pointing classifier.model at the judge is not enough to make it work.

The classifier in pip-based Switchyard 0.2.0 strictly parses a fixed JSON schema per profile ({recommended_tier, confidence, abstain, ...}).
This is a different contract from the verdict the judge returns ({crux, primary_rule, capability_boundary, p_solve}).
While the judgment prompt can be swapped out, the response schema and threshold logic cannot.

This is a limitation specific to the pip version.
The Rust-based switchyard-server can hold thresholds (base_threshold, threshold_step) in route definitions, so a judge of this form can be integrated through configuration alone.
Since I was proceeding with the rest of the setup using the pip CLI, I chose to insert a translation layer instead.

So I inserted a small proxy (shim) between the two that translates the contracts.

The shim is about 300 lines using only the Python standard library, and it does four things:

1. Restoring the judgment material: Extract the message list from the summary JSON, discard the system prompt, and reassemble "the initial request, recent exchanges, and the latest request" to fit within the judge's 8K context
2. Calling the judge: Pass the Card as the system message and the verdict schema as response_format, calling at temperature 0
3. Validation and thresholding: Validate the verdict's consistency (correspondence between rule and boundary, p_solve range), compare against thresholds, and decide between weak and strong. Invalid verdicts or judge failures fail-open to strong
4. Contract translation: Repack the result into the JSON Switchyard expects. Return values that always map to SIMPLE for weak and COMPLEX for strong (this nullifies the third cause mentioned above)

The core of the judgment is just this:

STEPS = {"supported": 0, "uncertain": 1, "unmatched": 1, "unsupported": 2}
threshold = 0.70 + STEPS[verdict["capability_boundary"]] * 0.10
route = "weak" if verdict["p_solve"] >= threshold else "strong"

Since the judge can only evaluate text, requests that include image or PDF attachments bypass the judge and go directly to strong.

routing-profiles.yaml

Since the YAML bundle schema is not documented, I reverse-engineered it from the loader implementation inside the wheel.
The final form is as follows:

defaults:
  timeout_secs: 300

routes:
  switchyard:                      # inbound model id (Hermes calls with model="switchyard")
    type: deterministic            # LLM classifier approach
    profile: general
    fallback_target_on_evict: strong  # escape destination on tier failure (required field)
    classifier:
      model: judge-shim
      base_url: http://127.0.0.1:8003/v1
      api_key: local-noauth        # required field even without auth (dummy)
      fail_open: true              # on shim or judge failure, fall to strong
      max_request_chars: 200000    # keeping the default 16000 means the request never reaches the judge
      recent_turn_window: 4
    weak:
      model: accounts/fireworks/models/deepseek-v4-flash-0731
      base_url: https://api.fireworks.ai/inference/v1
      api_key: ${FIREWORKS_API_KEY}
    strong:
      model: accounts/fireworks/models/kimi-k3
      base_url: https://api.fireworks.ai/inference/v1
      api_key: ${FIREWORKS_API_KEY}

I learned that fallback_target_on_evict and the classifier's api_key are required fields from startup errors.

error: invalid route bundle: switchyard.fallback_target_on_evict must be a non-empty string
error: invalid route bundle: switchyard.classifier.api_key must be a non-empty string

weak and strong are also automatically registered as standalone passthrough targets.
Querying /v1/models shows the route name switchyard alongside the actual tier entities and Fireworks's catalog, so when you want a task to always go to strong, specifying model="strong" will pin it there.

The choice of Fireworks DeepSeek V4 Flash for weak also has a reason related to alignment with the judge side.
The teacher during training for this judge was the same V4 Flash, so the calibration of p_solve directly corresponds to the actual capability of the execution model.

The reason for choosing Kimi K3 for strong is that it can handle images natively.
Since the judge can only read text, requests with attachments bypass the judge and go directly to strong.
If the receiving end cannot read images, the route doesn't function as intended.
DeepSeek V4 is text-only, so the option of using the same model family for both weak and strong was never available.

However, generation is not fast.
For the same 300-token generation, weak's V4 Flash takes 3.1 seconds, while K3 takes 14–18 seconds.
In a real-world agent, one turn involves hitting the model multiple times for tool calls, so the measured average for the strong side was 32.9 seconds, with a single-request maximum of 133 seconds.
When selecting the strong model, generation speed affects the user experience as much as capability does.

Switching NemoHermes's connection target

I point Hermes's inference target to Switchyard.
This was the most time-consuming step in this entire setup.

Pitfall: Allowed ports on the loopback bridge

When I stood up Switchyard on localhost:4000 and re-onboarded, the router inside the sandbox tried to connect to the container's own localhost:4000 and failed.

[sandbox] [INFO] [openshell_router] routing proxy inference request (streaming)
  endpoint=http://localhost:4000/v1 method=POST path=/v1/chat/completions
[sandbox] [OCSF] NET:FAIL [LOW] inference.local:443

Reading the NemoClaw source, the bridge logic that rewrites loopback URLs to host.openshell.internal accessible from the sandbox only activates when the port is 11434, 11435, or 8000.
Port 4000 was not included.

The fix was to swap the ports.
I placed the judge on 8002 and Switchyard on 8000.
Once the rewrite takes effect, the endpoint in the same log changes to http://host.openshell.internal:8000/v1.

Pitfall: Dual listen addresses

Onboarding validates from the host side against localhost:8000, and the sandbox accesses port 8000 on host.openshell.internal (Docker bridge 172.18.0.1).
This means listening on both addresses is necessary.
Binding to 0.0.0.0 handles both at once, but exposing an unauthenticated router to the internal LAN was not acceptable, so I passed on that.
I bind to the bridge IP and forward the loopback side with socat.

switchyard serve --routing-profiles routing-profiles.yaml --host 172.18.0.1 --port 8000
socat TCP-LISTEN:8000,bind=127.0.0.1,fork,reuseaddr TCP:172.18.0.1:8000

The reason this problem doesn't arise when using Ollama as a provider is that NemoClaw itself stands up an authenticated proxy at 0.0.0.0:11435 to solve it.
With a custom endpoint, you have to manage this yourself.

Pitfall: Configuration changes that cause delayed crashes

When expanding the context window, you might be tempted to run this inside the sandbox:

# Do NOT do this
nemohermes team-assistant exec -- hermes config set providers.compatible-endpoint.models.switchyard.context_length 262144

The command succeeds and the setting is applied.
However, the integrity hash at /sandbox/.hermes/.config-hash is not updated.
The startup script runs hash verification first, so the next time the gateway restarts, it fails verification and enters a crash loop.

[SECURITY] Hermes config hash does not match persisted inputs
[SECURITY] HERMES_MCP_CONFIG_DRIFT: MCP intent cannot be matched to the persisted gateway state

The tricky part is that nothing happens immediately after the change.
It continues running in a broken state and only surfaces at the next restart.
The correct approach is using the host-side command, which updates both the configuration and the hashes in a single transaction.

export PATH="$HOME/.local/bin:$PATH"   # openshell must be in PATH or you get ENOENT
nemohermes team-assistant config set --key <dotpath> --value <value> --restart

Recovery requires a rebuild (restart and recover will not fix it).
Worse, during the crash loop, the state backup cannot be taken and the rebuild itself is aborted, so you need to stop the loop with docker update --restart=no, manually back up with docker cp, and then recover with rebuild --force.

I adopted an operational rule: before directly touching configuration files inside the sandbox, first check whether there is an equivalent entry point on the host side.

Verification

The startup banner shows the judge and both tier assignments.

  switchyard  ready  →  switchyard

  profiles  ▶ switchyard  (default)
                llm-classifier
                strong      accounts/fireworks/models/kimi-k3
                weak        accounts/fireworks/models/deepseek-v4-flash-0731
                classifier  judge-shim
                profile     general

After switching to the judge, I can trace each judgment via verdict logs.
This is the biggest difference from the bare classifier — crux explains why it was routed to that side.

{"route": "weak", "latency_ms": 1353.5, "capability_boundary": "supported",
 "primary_rule": "SUP-1", "p_solve": 0.77,
 "crux": "The request asks for a list of in-progress issues in project TASKHUB
          grouped by assignee, which requires a read-only Backlog API call and
          formatting of the result."}

{"route": "strong", "latency_ms": 1813.8, "capability_boundary": "unsupported",
 "primary_rule": "LIM-1", "p_solve": 0.31,
 "crux": "The request asks for a 3-month cross-project issue trend analysis and
          a resource allocation plan for the next period. The main deliverable
          is deliberative analysis and a proposal, not a retrieval."}

The issue list request completes with a Backlog skill read and formatting, so p_solve 0.77 exceeds the threshold of 0.70 and goes to weak.
The trend analysis and resource allocation proposal, where the thinking is the substance rather than the retrieval, is judged as outside capability scope, fails to reach the threshold of 0.90, and goes to strong.
In the end-to-end test via Switchyard, the former was answered by DeepSeek V4 Flash in 7 seconds, and the latter by Kimi K3.

Judgment latency, including reading the Card (approximately 2,600 tokens) every time, is 1.8 seconds at p50 and 2.5 seconds at p95.
This is on par with the average of 1.7 seconds when using bare Lightning as the judge, meaning only the quality of judgment improved.

The distribution also changed markedly.
Here is the breakdown of 156 judgments recorded over 4 days of normal team usage after switching to the judge (including those made during setup verification):

The breakdown by boundary was: supported 96, unsupported 45, unmatched 11, uncertain 3, and 1 that failed validation.
This is exactly the intended split: "tasks within capability (Backlog searches and formatting, web searches) go to weak; tasks where thinking is the substance go to strong."

On the other hand, a cost comparison is not available.
Switchyard's /v1/stats has a cost_estimate per tier, but since model unit prices are not configured, it remains at 0.
The distribution improvement described in this article is purely "how many times each model was called," not a comparison of billed amounts.

Conclusion

The value of inserting a router was not the "cost savings" I had originally envisioned.
What actually helped was being able to see in numbers exactly what was flowing where.
Without the stats showing a 92.4% bias, I would never have noticed that the judge wasn't reading the request text, nor that the bare model lacked the vocabulary for the role.

Replacing the judge with a post-trained model feels worth the effort.
By designing the judgment not as "guessing task difficulty" but as "forecasting whether this agent setup can complete the task," each judgment became traceable with a rationale.
And changes in capability can be tracked by rewriting the Capability Card rather than retraining the model.

The benefit of keeping a single entry point is also significant.
Since going into operation, the execution-side model has been swapped several times, but I haven't touched the Hermes configuration once.
Since model="switchyard" never changes, swapping is just a few lines in the route definition and a router restart.

What I want to try next is making weak fully local.
The DeepSeek V4 Flash-0731 chosen for weak has a track record of running across two DGX Sparks in-house.

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-0731/

The 284B (active 13B) weights are 155 GiB in the officially distributed FP8 and MoE FP4 formats, which doesn't fit in a single machine's unified memory.
Connecting two machines directly via QSFP and splitting with vLLM's tensor parallel yields 76 tok/s decoding on short prompts and still 69 tok/s with a 900K-token context.
If weak can be brought back here, by the same logic as the judge, the contents of everyday requests would also never leave the DGX.
Since the router is already in place, swapping at that point is just changing the base_url in the route definition.

Next time, I'll have the agent operate Backlog via skills.
The topics will be the design for allowing writes while blocking deletes, and growing the skills through real-world operational feedback.

References

• NeMo Switchyard (GitHub)
• Training Nemotron 3.5 Lightning as a Switchyard Judge (DevelopersIO)
• NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (Hugging Face)
• DeepSeek V4 Flash (Fireworks)
• Kimi K3 (Fireworks)
• Kimi K3 vs DeepSeek V4 (Orca Router)
• Running 284B DeepSeek V4 Flash-0731 Across Two DGX Sparks (DevelopersIO)