I tried repurposing Nemotron Lightning as a dedicated judge for a team assistant using SFT and Capability Cards

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.

In my previous article, I used LoRA SFT to turn NVIDIA's Nemotron 3.5 Lightning into a judge for a coding agent router. As I wrote at the end of that piece, "the first production use case is the resident agent," and I had already been reusing that coding judge for routing decisions in a Slack-based team assistant.

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

The majority of requests this assistant receives are routine tasks like issue lists and standup summaries. A cheap locally running model (weak) can handle those just fine. On the other hand, when deliberative requests like strategy brainstorming get routed to weak, it returns plausible-sounding generalities that people tend to believe. So for each request, I insert a judge to determine "is weak enough, or do we need the expensive model (strong)?" — and the accuracy of that judgment sets the ceiling for both response quality and cost.

Was it really acceptable to leave this critical judge as a repurposed coding judge? To answer that question with actual measurements, I created a dedicated evaluation set for the resident assistant and re-trained a dedicated judge from the same Nemotron 3.5 Lightning base.

Let me state the conclusion upfront: the repurposed coding judge passed 84% of validity checks on 189 dedicated evaluation cases, with 12 critical misses routing brainstorming and strategy discussions to the cheap model. By externalizing runtime environment knowledge into a single document called a Capability Card, and running SFT on 2,122 dedicated training samples labeled with weak's empirical success rates, we achieved 100% validity, 1–2 critical misses, and a median judgment latency of 1.57 seconds — and the addition of web search capability was reflected in judgments simply by rewriting the Card, without any retraining.

This is a continuation of the previous article, so please refer to that piece for the LoRA SFT recipe and NVFP4 quantization steps, and the NemoHermes article for the resident assistant's architecture (those articles are from August 2026 and June 2026, respectively).

https://dev.classmethod.jp/articles/dgx-spark-nemotron-lightning-switchyard-classifier-finetune/

https://dev.classmethod.jp/articles/dgx-spark-nemohermes-openshell-hermes-agent/

This article covers: the design of externalizing runtime environment information as a Capability Card, building training data from empirical labels, and verifying that adding web search capability is reflected in judgments simply by rewriting the Card. I hope it resonates with anyone trying to build a judge or classifier for their own domain.

The repurposed coding judge hit a ceiling at 84%

The target resident assistant is a read-only assistant that receives Slack mentions, reads issues in the team's Backlog space, and returns per-assignee lists and standup summaries. The router (NeMo Switchyard) calls the judge for each request, and the judge returns a JSON containing p_solve — the probability that weak will complete the request in one shot — and a judgment category indicating which decision rule applies. The router compares p_solve against a threshold to route the request to either weak or strong.

The judge only sees the request text, the last 4 turns of conversation, and a single document called a Capability Card summarizing the runtime environment. I'll explain the Card's contents in the next section, but the architecture is the same as the previous article — the only thing I'm changing this time is the judge.

The question is whether it's acceptable to leave the coding judge in place as the decision-maker. So I froze an evaluation set of 189 cases for the resident assistant (the creation process is described later) and measured both the repurposed coding judge and the vanilla Lightning with only the Card provided, under identical conditions. I used four metrics: validity check pass rate (how many responses conformed to the JSON format), judgment category match (how often the correct decision rule was applied), weak retention rate (how often requests that weak could handle were kept with weak), and the number of critical misses — cases where requests that should go to strong, like brainstorming and strategy discussions, were sent to weak instead.

The vanilla Lightning doesn't know the decision rules and judgment category mappings at all, so 73% of responses were invalid. The fail-safe routes almost everything to strong, which avoids disasters, but with a weak retention rate of 0% and everything going to the expensive model, there's no point having a router. The pattern seen in the previous article — "vanilla Lightning doesn't work as a judge" — held true even when environment information was provided via the Card.

The repurposed coding judge passes 84% on format, but its vocabulary was still that of the coding domain. In the coding training, brainstorming was categorized as "doesn't match any rule," whereas in the assistant domain, brainstorming is the primary battleground. As a result, 12 critical misses occurred at exactly the boundary we most needed to protect.

In other words, the Card can provide material for judgment, but it cannot create the judgment behavior itself. Behavior is built through SFT. Separating these two layers is the core of this design.

All environment information goes into the Capability Card

The judge is not given the execution-side agent's system prompt or tool definitions. Instead, all information about the runtime environment that the judge needs to evaluate is compiled into a single document called a Capability Card, which is rendered into the judge's system prompt. For the judge, this Card is the entirety of the environment information.

It is used in exactly one place. The rendered Card is inserted wholesale into the prompt field for the judge in Switchyard's route configuration. Each time a request arrives, the router calls the judge with the Card as the system message and the Slack request text plus recent turns as the user message, then compares the returned JSON's p_solve against the threshold to choose weak or strong. The flow looks like this:

The format of requests to the judge and its responses are as follows (excerpts):

{
  "messages": [
    { "role": "system", "content": "<rendered Capability Card, ~11,000 chars>" },
    { "role": "user", "content": "List in-progress issues in TASKHUB" }
  ],
  "temperature": 0,
  "response_format": { "type": "json_schema", "json_schema": { "name": "CapabilityClassifierDecision", "strict": true } }
}

{
  "crux": "Retrieve in-progress issues using the valid Backlog skill and format as a list",
  "primary_rule": "SUP-1",
  "capability_boundary": "supported",
  "p_solve": 0.77
}

The four response fields use the same judgment contract as the previous article. The Card is not passed to the execution-side weak or strong models. Requests flow directly to the execution model, and only the judge reads the Card.

The Card has two parts:

The key to the fixed section is the success condition — the definition of SUCCESS. Success means selecting the correct skill, having the retrieved data match the request, and ultimately delivering a Slack reply. Plausible-sounding generalities without grounding count as failures, no matter how well-written. The p_solve the judge outputs is "the frequency at which this SUCCESS is naturally achieved," not the judgment model's confidence score. Leaving this ambiguous causes drift in the labeling process described in the next section.

The dynamic environment profile is held as JSON and the system prompt is mechanically rendered from a template. After rendering it's about 11,000 characters; for a single request, the full prompt is around 2,600 tokens. Since it's a static document, vLLM's prefix cache applies and it barely adds to judgment latency.

This swappable profile structure supports the counterfactual pairs that are central to the design. Pairs where only the Card's profile differs for the same request text are included in both training data and the evaluation set. For example, the request "List in-progress issues in TASKHUB" should return "can do it" when the Backlog skill is enabled in the Card, and "cannot do it" when disabled. A judge that can make this distinction is demonstrably reading the Card to make its determination, rather than memorizing request patterns. In evaluation, this is measured as "Card sensitivity" across 20 pairs.

A note on the name and portability: "Capability Card" is a name I coined, inspired by the A2A protocol's Agent Card (a mechanism where agents declare their capabilities in JSON). The difference is that this document is meant to be read by the judge to understand the execution-side environment, and it includes success conditions and decision rules. Also, since it's essentially a system prompt for the judge, it isn't specific to Switchyard. Any router or proxy that can call an OpenAI-compatible judge endpoint and branch on p_solve can use the same Card. What's specific to Switchyard is the judgment JSON contract and how thresholds and fail-open behavior are handled.

Training data is built from weak's empirical success rates

In the previous article, the target was a coding benchmark, so weak's empirical accuracy could be measured mechanically by test pass/fail. In this domain, the output is a "Slack reply," so I needed to build the success rate measurement methodology from scratch. Training data has a three-tier structure:

The first tier is the test questions. I reviewed 458 Golden Set cases and froze 189 for evaluation, excluding them from training data via hash matching as in the previous article.

The second tier's synthesis used NeMo Data Designer, generating 34 types of request patterns typical for the assistant (issue retrieval, standup summaries, brainstorming, ambiguous requests, write requests, etc.) using two generation models, and pairing them with Card-swapped counterfactual partners. Generated requests were passed through a teacher filter that kept only those where a teacher model's judgment matched the expected label. This filter would later trip me up — but that's covered in the next section.

https://github.com/NVIDIA-NeMo/DataDesigner

The third tier is the crux. For p_solve labels, I use empirical measurements of the actual weak model (deepseek-v4-flash-0731, same as in production) solving the evaluation set's requests. Since the production space can't be used, I set up a mock Backlog environment with a fictional project containing 203 issue fixtures, ran 312 replays, and scored them on 8 criteria to assign success or failure. I also ran an additional 33 runs against a validation real API space to confirm that mock scoring didn't diverge from the real environment. The per-category success rates thus obtained were fed as p_solve labels into the training data.

The 8th scoring criterion was added partway through. When scoring with the original 7 criteria (correct skill selection, arguments, grounding, etc.), runs that "correctly reported they couldn't execute because of read-only restrictions" counted as successes, leading to success rates above 0.6 for write-type requests — a perverse outcome. Properly declining a request and completing it are different things. Adding "request completion" as the 8th criterion brought out-of-capability categories to a flat 0 success rate, and judgment categories aligned with p_solve.

Of these, only the 47 brainstorming and strategy runs were scored manually by my own eyes. Agreement with LLM-based scoring was 25 out of 45 cases — leaving this domain to automation produces systematic drift.

Three training runs with different data mixes, and the dedicated data won

The training recipe is reused from the previous article (Megatron-Bridge LoRA SFT, rank 8 / alpha 32 / 2 epochs). Three things changed: sequence length was extended from 3,072 to 4,096 to handle Slack threads; inputs were changed from standalone instruction texts to actual message sequences; and tool call arguments were held in structured form rather than strings. Hardware was 2 H100s rented via NVIDIA Brev, with all three training runs costing about $25 total.

The three runs differed in their data mix. Run 1 used only dedicated data: 1,659 cases. Run 2 added 500 cases of coding training data — a technique called rehearsal, mixing in some previous training data to prevent forgetting, which could also enable one model to serve both domains. Run 3 targeted the 4 remaining misses from Run 1 with 402 additional cases including duplications of uncertain examples, totaling 2,122 cases.

First, rehearsal was counterproductive. The match rate improved slightly, but the critical brainstorming misses went from 2 to 5. The coding data's judgment distribution skews toward "weak can handle it," which pulled brainstorming judgments toward the optimistic side as well. In post-training that replaces the entire judgment vocabulary, rehearsal brings in bad habits from the old domain. Fortunately, since the judge is a dedicated routing component, forgetting coding capabilities isn't a cost.

For the third targeted run, I hit the trap of the teacher filter mentioned earlier. The hard cases I was targeting are brainstorming requests that look like work instructions — things like "reschedule it by one day and rebuild the budget at ¥600,000." This is also a domain where the teacher model itself makes judgment errors. So the teacher filter systematically discards exactly these hard cases. Even replacing the teacher with a higher-tier model for second-pass arbitration, only 1 of 25 cases survived. I ultimately read all 24 remaining cases myself and manually restored 23 to the training data. Automatic teacher-model filtering quietly strips out the hard cases that are hardest for the teacher itself — the very cases you most want to include.

The results are as shown in the table: judgment category match 92.1%, critical misses down to 2. All 4 targeted cases were fixed, but 2 different cases newly regressed — a swap. This swap was only visible because the evaluation set was frozen. I judged that further refinement against these 189 cases would lead to overfitting the evaluation set, and stopped adding synthetic data here.

With the adopted configuration decided, I also recalibrated the threshold. Using the coding judge's threshold of 0.80 as-is drops weak retention to 1%. In this domain, weak's empirical success rate for routine requests tops out around 0.77, so p_solve clusters between 0.70 and 0.77 — it just can't reach 0.80. I swept the threshold and decided to operate at 0.70. Changing domains means recalibrating thresholds.

Arrived at a 21GB, 1.57-second judge

The finishing step is the same NVFP4 quantization via Model-Optimizer as the previous article, bringing the ~66GB BF16 model down to 21GB.

One note on evaluation methodology: up to this point, evaluations were run in parallel to save time. Since vLLM batches multiple requests together, inputs that land exactly on the threshold can have unstable judgments even with temperature 0. When I re-ran the 189 cases one at a time, 8 judgments shifted. In production, the router calls the judge one request at a time, so there's no practical impact — but evaluations that inspect threshold-adjacent cases should be measured with batch=1, matching production. All finalized numbers below were measured sequentially.

After quantization, one piece of instability remained that persisted even with sequential single-request evaluation. Running the same input 30 times showed that 4 inputs whose p_solve lands exactly at the threshold 0.70 oscillate between judgments 13–43% of the time. I suspect this is due to parallel summation in the quantized MoE kernel, but all oscillations went toward the strong side. The practical impact is a "safe failure" — requests that weak could handle occasionally go to strong at slight extra cost.

Conversely, in BF16, one specific input went rogue 30/30 times — after finishing the JSON output, it kept emitting whitespace until the limit, causing one judgment to take 55 seconds. This disappeared after NVFP4 quantization, though I believe the quantization shifted the trigger condition rather than actually fixing it. Like in the previous article, this is also a safe failure — the fail-safe routes it to strong.

Comparing all configurations under identical conditions reveals the full picture of the retrained judge's improvement. The calibration column is the Brier score — lower means p_solve is closer to empirical success rates. Note that this table uses parallel evaluation relative comparisons and includes batch noise of ±a few cases.

The finalized numbers for the adopted configuration, measured sequentially with batch=1 and NVFP4, are as follows. Judgment latency settled at a median of 1.57 seconds.

Like the coding judge, this fits on a single DGX Spark at 21GB, with speed equal to or better than the current judge.

This judge was built for team use and is already deployed in the team's router. My colleague Shimada has written an article covering the setup procedure including Switchyard configuration and Card handling, including a breakdown of how over 60% of 156 recorded judgments from normal team usage were routed to weak (article dated 2026-08-28).

https://dev.classmethod.jp/articles/reona-02-dgx-spark-switchyard-routing/

Note that this judge does not replace the coding judge. Both are deployed as separate judges — the same base model with different LoRA adapters and different thresholds (0.80 and 0.70). The option of mixing both domains into one judge proved counterproductive in the rehearsal measurements, and since retraining costs around $10 per run, the practical approach is to maintain separate Card + adapter pairs per domain.

Adding web search only required rewriting one paragraph of the Card

Finally, I wanted to verify that this design works as intended. The team had plans to add web search to the assistant. When that day came, what would need to happen with the judge? I tested this in advance to see whether retraining could be avoided.

What I did was simply render an updated Card with web_search enabled in the profile, and run 24 counterfactual probes against both old and new Cards. The result: judgments for web-dependent requests correctly flipped from "cannot do it" (p_solve 0.01) to "can do it" (0.77), while the remaining 22 items remained unchanged. This confirmed that even capability descriptions that didn't exist at training time are reflected in judgments through Card comprehension.

When the day of capability addition arrives, what's needed is not retraining — just rewriting one paragraph of the Card and restarting the router. Externalizing environment information into the Card rather than baking it into the weights was precisely for this day. One caveat: writing a capability into the Card before it actually exists causes judgments to get ahead of reality. So until web search was implemented, we kept the old Card in operation; once the implementation was complete, we reconciled the actual skill documentation with the Card text and finalized the updated version. There were no regressions in the 24-case probe at finalization, and this version is now the default in the distributed Card.

Summary

The repurposed coding judge scored 84% validity and 12 critical misses on 189 dedicated evaluation cases. By externalizing runtime environment information as a Capability Card and running SFT on 2,122 dedicated training samples labeled with weak's empirical success rates, we achieved a dedicated judge with 100% validity, 94.7% strong recall, and 1.57-second median judgment latency. Training took 3 runs, about $25, and 2 working days. We also confirmed that adding web search capability is reflected in judgments simply by rewriting the Card.

Three lessons emerged from this work. The Card can provide material for judgment, but SFT is what builds the judgment behavior. Mixing in coding data via rehearsal was counterproductive and required recalibrating the threshold, so maintaining separate Card + adapter pairs per domain is the practical approach. And automatic teacher-model filtering quietly strips out the hard examples that are hardest for the teacher — keeping a human review point in the pipeline turned out to be the most reliable approach.

To be transparent about what's unresolved: brainstorming misses remained at 1–2 cases until the end, and all evaluations in this article were conducted on synthetic data. How things look on the team router's real traffic is left to Shimada's article. Once judgment logs accumulate from team usage, I'd like to add actual request patterns to the training data and see whether the remaining misses can be fixed.

Reference Links

• NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (Hugging Face)
• NVIDIA-NeMo/Nemotron — Repository containing LoRA SFT usage cookbook
• NVIDIA-NeMo/Switchyard
• Automatically routing Slack agent requests with NeMo Switchyard and a post-trained judge model — Article on building the team's router, which uses this judge as the decision-maker (2026-08-28)
• Agent2Agent (A2A) Protocol Specification — Agent Card spec; the inspiration closest to the Capability Card concept
• NVIDIA-NeMo/DataDesigner — Used for synthetic data generation
• NVIDIA/Model-Optimizer — NVFP4 quantization recipes are included here