Trying a 2B Model Distilled from DeepSeek V4 Flash's Judgments as the Classifier in NeMo Switchyard

Introduction

Hello, this is Shimada from the Manufacturing Business Technology Department at Classmethod.

I have placed NeMo Switchyard in front of a Slack resident agent (NemoHermes) and use it to route requests to one of two models, weak and strong.
The judge role was initially a post-trained judge model (a generative model), and in the previous article I replaced it with TypeSafe's Jev (a judgment-only API that does not generate text).

https://dev.classmethod.jp/articles/reona-02-dgx-spark-switchyard-routing/

https://dev.classmethod.jp/articles/reona-dgx-spark-switchyard-jev-classifier/

Jev was fast and accurate, but the judgment input (the request text) is sent to TypeSafe.
That single point was the reason I had kept the judge local, so speed and "not sending input outside the DGX" could not both be achieved.

This time I replaced the judge role itself with a different small model (2B and 4B), and placed on the DGX Spark a classifier trained (distilled) on the judgments of a large model.
Like Jev, it does not generate text and returns only the choice and a confidence score, and the input does not leave the machine.
As a result, the time per judgment went from about 1.5 seconds with the judge to around 40 milliseconds, and the agreement rate of judgments on the validation data exceeded that of the teacher model's immediate-answer mode.

What prompted this, and what this article compares

The trigger was a post on X.
Based on the fact that Jev appears to be trained on synthetic data and labels from a large-scale model, the post described distilling the judgments of DeepSeek V4 Flash on a DGX Spark into a 4B model, which exceeded the accuracy of the teacher's immediate-answer mode at about 22 milliseconds per judgment.

https://x.com/taroleo/status/2101106887840370919

Switchyard's judgment, "is weak enough for this request?", is a classification with two choices, so the same approach applies directly.
Our environment already had the judgment question and the text of the choices, the 13 evaluation scenarios used in earlier verification, and the measured values for the judge and Jev, so the only missing pieces needed to follow the post's procedure were the training data and the training code.

This article lines up the following five judge roles.

1. judge (generative). In the current output format, writing the basis for its judgment (crux)
2. judge (generative). With crux removed from the output format. No training
3. The judge retrained to read out a single token
4. The distilled classifier (this time). Returns only the choice and a confidence score
5. Jev. Returns only the choice and a confidence score

The difference from 1 to 2 is "how many seconds are saved by no longer writing the basis", the difference from 2 to 3 is "how many seconds are saved by no longer generating at all", and the difference from 3 to 4 is "how many seconds are saved by going from a 30B to a 2B judge".
The point of this article is not which is fastest, but looking at these three differences.

Models that can be used as teachers, and those that cannot

Distillation uses the teacher model's outputs as training data.
Some models and APIs prohibit using their outputs to train other models in their terms, so I checked this first.

Sending request text to Fireworks is the same path I already use for the strong executor.
Only synthesized request text is used for training this time, so no internal information newly leaves the company.

Architecture

To hook it into Switchyard, I use the llm_classifier route that currently calls the judge, unchanged.
In front of the classifier I place a small server that returns responses in the same format as the judge, so no changes are needed on the Switchyard side.

Synthesizing request text

The training data was made purely from synthetic data.
The logs of the judge in operation keep only the judgment results (boundary, probability, crux) and not the request text itself.

I decided on 30 types of requests that come to Slack (greetings, terminology checks, translation, summarization, Backlog status checks, internal wiki searches, schedule checks, technical sounding-board discussions, code review, error triage, cross-checking multiple documents, architecture comparisons, estimates, risk identification, short messages with ambiguous intent, thread continuations, attachment-only requests, English mixed in, and so on) and 8 writing styles (a short one-liner, polite business prose, casual tone, with typos or abbreviations, with a deadline or priority added, and so on), and had DeepSeek V4 Flash write a Japanese request for each combination.
I did not let it use real person or customer names, and used fictitious names instead.

I matched the shape of the request text to the state that Switchyard sends to the judge role (plain text with the speaker and body joined by line breaks).
If the input format differs between training and judgment, accuracy drops at that point.

There were two things I fixed after trying the generation.
First, at temperature 1.0 the responses sometimes came out in Chinese, so I added an instruction to write in Japanese and excluded request texts that contained no kana.
Second, when I had it return 10 items at once as a JSON array of objects, the JSON sometimes broke, for example the closing bracket of the array was dropped for types with long bodies, so I changed it to an array of just the request strings.
The final count was 2,731 items.

Labeling with the teacher

I used the judgment question and the choice descriptions exactly as written in the configuration for Jev.
If the criteria are not aligned, it becomes unclear what differs when comparing with Jev later.

問い: Slack の社内アシスタントがこの依頼に答えるとき、どのモデル階層が必要かを選んでください。
weak: 短い質問、用語の確認、翻訳・要約、Backlog や社内 wiki の検索や引用のような、手順が明確で完了条件がはっきりした依頼。
strong: 設計判断や方針の相談、壁打ち、複数の資料を突き合わせる調査、コードの設計やレビュー、正確さの誤りが高くつく依頼など、深い推論や多段の作業が必要な依頼。

For each item I had DeepSeek V4 Flash (with thinking) judge three times, returning a weak or strong label and the probability that weak suffices as JSON.
If all three agreed, that was the label.
For items where the three split, and items where the average probability was between 0.4 and 0.6, I had Kimi K3 judge once and adopted its result.
186 items went to K3 arbitration.

For the comparison in this article, I also took one judgment per item from DeepSeek V4 Flash without thinking (immediate answer) on the same request text.
With the Fireworks API, passing {"thinking": {"type": "disabled"}} brought reasoning tokens to 0.

The cost was about $0.6 for synthesis and about $2.4 for labeling (about $1.7 for DeepSeek and about $0.7 for K3).
Even with thinking, the reasoning for a judgment was short, around 75 tokens per item, and the cost came in well under the prior estimate (about $10).

The label breakdown is 1,663 weak and 1,068 strong.
By type, greetings and terminology checks are almost all weak, sounding-board discussions and architecture comparisons are almost all strong, and short messages with ambiguous intent, attachment-only requests, thread continuations, and English-mixed requests are split between weak and strong.

Training the classifier

I prepared two students, Qwen3.5-2B and Qwen3.5-4B.
For both, I added a classification head that outputs 2 classes from the state of the last token of the backbone, and trained with LoRA (r=16) for 3 epochs.
For labels I used the ratio of the 3 votes as is (soft labels), and made only the items arbitrated by K3 hard labels.
After training, I merged the LoRA and saved it as a single model, and calibrated the confidence by temperature scaling on the validation data.

The training data is 2,275 items and the validation data is 456 items (one sixth taken from each type).
Training time on a single DGX Spark was 17 minutes for 2B and 28 minutes for 4B.
This was while sharing the machine with other services, so it should be somewhat shorter if used exclusively.

For the training environment, I used the vLLM container image that runs the judge on the DGX Spark (with torch and transformers), adding peft and flash-linear-attention.
flash-linear-attention affects the inference speed discussed in the next section.

Results

Agreement of judgments

On the 456 validation items, I looked at agreement with the teacher's labels (3 votes with thinking plus K3 arbitration).
For comparison, I also list the agreement of the teacher's own immediate-answer mode (without thinking).

Both students exceeded the teacher's immediate-answer mode.
This is the same picture as in the X post.
Since they were trained on three rounds of judgments with thinking, I think their judgments are more consistent than a teacher's single immediate answer.

Looking at the misses, of the 18 items that both 2B and 4B got wrong, 16 were weak requests sent to strong.
The errors lean toward the side that raises cost, and there are few on the side that lowers quality (strong requests sent to weak).

In the results of judging the 13 scenarios three times each, both 2B and 4B got all 39 as expected.
Even "checking a meeting URL", which the judge wavered on depending on the run, was placed in weak all three times by the classifier.
Because it does not generate, the same input returns the same probability, and there is no variation from temperature.

Judgment time

I measured the time per judgment by sending requests directly to the judge role's server.

The 4B did not reach the 22 milliseconds in the X post.
To isolate the reason, I measured the time of one forward pass by input length.

Units are milliseconds, bf16, batch size 1.

I learned two things.
Qwen3.5 has linear attention layers, and without the dedicated kernel (flash-linear-attention) it falls back to the torch implementation and the time doubles.
Even with the kernel installed, the time is almost the same as Qwen3-4B with only standard attention, so the remainder is the prefill computation on the GB10.
Since the time is almost proportional to the input length, it can also be read that the limit is computation, not reading out the weights.

The input sent to Jev averaged 549 tokens, so with 4B it takes around 100 milliseconds.
To get under 50 milliseconds, it has to be 2B.
The difference in agreement is 4 items out of 456, so I decided to use 2B for the judge role.
The comparisons below also use 2B numbers.

Lining up the five judge roles

These are the measured values for the five judge roles.
The first row for the judge is the same 13-scenario measurement as in the previous article, and the second row is a measurement in which three requests were sent directly to the judge twice each (same Capability Card, temperature 0, without thinking).
The third row is from a colleague's article, and its evaluation data and measurement conditions differ from the other rows.

Removing the crux from the judge reduces the output from 70 to 90 tokens down to 31 tokens, and the judgment goes from 1.49 seconds to 0.88 seconds.
This is 0.6 seconds shorter without any training, but the generation time still remains.
Once generation is dropped, even the same 30B judge takes 0.25 seconds, and Jev takes around 0.3 seconds.
Making the judge role 2B brings it to 0.03 seconds, and the remaining difference is the prefill computation.

If you look only at speed, the 2B classifier wins, but I have not compared the quality of judgments on equal footing.
The single-token judge has learned the rules of the Capability Card, and its judgment includes "whether this agent has that means".
The classifier this time is only trained on two descriptions of "is this a request that weak can handle", does not look at the agent's capabilities, and its evaluation is also on synthetic data and the 13 scenarios.

To get this speed without sending the judgment input outside, I had to train the classifier myself.
That comes with the trade-offs described in a later section.

Hooking it into Switchyard

I added an endpoint that maps the classifier's output to the judge's verdict (JSON with 4 fields) and returns it, and call it from the llm_classifier route in place of the judge.
Switchyard itself is unmodified.

{"crux": "classifier route-clf: p_weak=0.9812", "primary_rule": "SUP-1", "capability_boundary": "supported", "p_solve": 0.9812}

I put the (calibrated) probability that weak suffices into p_solve, always set capability_boundary to supported, and leave the threshold to Switchyard's base_threshold.
crux cannot be empty (at least 1 character in the schema), so I made it a fixed sentence containing the probability.

[llm_clients.clf]
format = "openai_chat"
base_url = "http://127.0.0.1:8016/v1"   # Classifier on the DGX. No authentication

[targets.clf]
id = "route-clf"
llm_client = "clf"

[routes.switchyard]
type = "llm_classifier"
mode = "capability"
classifier_target = "clf"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5      # p_solve is a calibrated probability, so cut at 0.5, the same as argmax
threshold_step = 0.10
# Do not set recent_turn_window. Only the opening request and the latest user message are passed, which matches the input format used in training

There was one snag.
Switchyard checks the combination of the verdict's primary_rule and capability_boundary, and none can only be paired with unmatched.
When I first returned none with supported, the verdict was treated as invalid and every request was routed to the default, strong.
It passes once corrected to the pair SUP-1 and supported.

With this setup, I sent the 13 scenarios three times each, and all 39 were as expected.
According to Switchyard's statistics, the classifier call has a median of 43 milliseconds, and the total overhead from judgment to route decision has a median of 44 milliseconds (maximum 69 milliseconds).
By the same measure, the previous Jev was 0.27 to 0.42 seconds, and the judge was 1.55 seconds.

Trade-offs found in use

The judgment criteria go into the weights.
With Jev, rewriting the choice descriptions (criteria) changed the criteria, but with the classifier, every change to the criteria requires relabeling with the teacher and retraining.
At this scale, labeling takes a few dollars and an hour and training takes 20 minutes, so it does not stop operations.
Still, it is not suited to a period when the Capability Card is being revised frequently.

The basis for judgments cannot be read.
There is no way to write out "the hardest requirement in this request" the way the judge's crux does, so why something was sent to strong has to be inferred from the confidence score and the input.
This is the same trade-off as with Jev.

Both the training data and evaluation data are synthetic.
I have trained and evaluated on neither actual Slack requests, so accuracy may drop for types of requests not included in the synthesis.
The 456 validation items are request texts made by the same procedure as the training data, and the 13 scenarios are also short single-exchange requests.
This article does not verify quality on evaluation data made up of real requests or difficult requests.
For requests that look clearly procedural but actually need a means the agent does not have, I expect the classifier is likely to send them to weak, since it does not look at capabilities.
If a change is made to keep operational request texts in the logs, I can evaluate on real data and mix it in to retrain.

Input is truncated at 512 tokens.
For deployment I prioritized 50 milliseconds and truncated the input at 512 tokens.
It is almost enough as long as the opening request and the latest message fit, but for requests with a long pasted body, the latter half of the body is not seen.

Conclusion

By training a 2B classifier on a large model's judgments, I brought Switchyard's judgment from about 1.5 seconds with the judge to around 40 milliseconds, and the routing of 13 scenarios × 3 runs was all 39 as expected.
The agreement rate on the validation data exceeded the teacher's immediate-answer mode, and the cost of creating the training data was about $3.

There were four ways to speed up the judgment.
Removing the basis from the judge's output gives 0.88 seconds, retraining the judge for single-token readout gives 0.25 seconds, sending it to Jev gives 0.3 seconds, and training a small model gives 0.03 seconds.
With the premise of not sending the judgment input outside, two remain: retraining the judge or training a small model.
In both, changing the criteria means retraining rather than editing text, and the basis for judgments becomes unreadable.
The difference between the two is speed (0.25 seconds versus 0.03 seconds) and whether the judgment includes the rules of the Capability Card.
Comparing quality on real operational requests is a task for the future.

The "this much is possible with just a local LLM" in the X post was a story of doing even the teacher's labeling locally.
We put the teacher on Fireworks, so that part is not the same.
The teacher's role is only to label the synthesized request text, and the student sees only those labels, so as long as it is the same model, the training result is expected to be the same wherever the teacher runs.

References

• X post (distilling DeepSeek V4 Flash's judgments into 4B)
• I made Nemotron 3.5 Lightning's judge role a non-generating single-token judgment like Jev (DevelopersIO)
• Replacing NeMo Switchyard's judge role with TypeSafe's Jev to decide Slack agent routing without generation (DevelopersIO)
• Loading a post-trained judge into NeMo Switchyard to route Slack agent requests (DevelopersIO)
• feat(routing): add type_safe_classifier backed by TypeSafe's Jev(NVIDIA-NeMo/Switchyard PR #739)
• TypeSafe Master Customer Agreement
• DeepSeek-V4-Flash(Hugging Face)
• Kimi-K3 LICENSE(Hugging Face)
• Qwen3.5-2B(Hugging Face)
• flash-linear-attention(GitHub)