I tried using Nemotron 3.5 Lightning as the judge, making it a single-token judgment that doesn't generate like Jev
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
In August, I wrote an article about fine-tuning Nemotron 3.5 Lightning with LoRA to serve as a classifier for an LLM router (article dated 2026-08-16). That classifier has since spent over a month routing coding agent requests in NeMo Switchyard's production path. Switchyard v0.3.0 was released on September 22 and we switched production over, but the model serving as classifier is the same.
During that month, the landscape around the classifier changed somewhat. TypeSafe's Jev released a non-generative judgment as a product, implementations calling themselves OpenJev followed, and I wrote an article comparing them (article dated 2026-09-18). Meanwhile, my own classifier was taking 1.2 seconds to write JSON for each request. I wanted to know how fast it could get if I stopped making it write JSON, and whether judgment quality would hold up. I wanted to verify this with the same weights used in production.
To state the conclusion upfront: with the same weights, judgment time shrank from 1.18 seconds to 0.95 seconds with speculative decoding, and down to 0.25 seconds by limiting output to 1 token. The 1-token version's judgment quality nearly matches the current version, and the remaining gap in my evaluation set is just one step away. My assessment from this experiment is that speed is determined by the form of readout, and judgment quality is determined by the form of problems in the training data. Combined with the results of delegating the same classifier role to Jev, this article introduces how I measured and separated classifier speed from judgment quality, and what conclusions can be drawn. I hope it resonates with people who want to make the judgment component of routers and agents smaller and faster.
80% of Judgment Time Was Spent Writing JSON
The classifier is Nemotron 3.5 Lightning 30B-A3B fine-tuned with LoRA and quantized to NVFP4. It reads the request text and a capability card (a system prompt describing the weak model's strengths and weaknesses, and whether tools like reference lookup and test execution are available in the environment; hereafter "Card"), and returns a 4-field JSON. Breaking down one judgment cycle using vLLM metrics:
| Phase | Average | Contents |
|---|---|---|
| prefill | 0.210 sec | Reading approximately 1,250 input tokens |
| decode | 0.972 sec | Writing approximately 78 tokens of JSON at 12.7 ms per token |
| Total (p50) | 1.18 sec | Consistent at 1.18 sec across 3 restarts |
82% of the time is decode — that is, the time spent writing JSON. The prefix cache for the input side doesn't fill the 4,176-token cache block for this classifier's Mamba-layer-containing architecture, so even though the Card is the same every time, hits were 0. To make it faster, there's no choice but to reduce decode.
20% Reduction with Speculative Decoding; Grammar Constraints Caused Runaway
As an approach that required neither changing weights nor the output format, I added DSpark, the official Lightning speculative decoding draft. This is the one that achieved 1.45x generation speed in the first-touch article (article from 2026-08).
With DSpark k=3, the median went from 1.18 seconds to 0.95 seconds, and average decode from 0.97 seconds to 0.72 seconds — a 20% overall reduction. The average number of draft tokens accepted per verification was 1.71; since the JSON output is short, this was more modest than the official benchmark numbers. This configuration was promoted to production on September 21, and the production median went from 1.16 seconds to 0.99 seconds.
As a shortcut to reduce the output itself, I also tried a configuration using JSON schema to limit the reasoning field to 200 characters. In 66 out of 1,152 judgments, the model started outputting 4,000+ newlines the moment it hit the limit and ran to the maximum token count. This is the same pattern as the key-order runaway I described in the August article — adding grammar constraints at inference time that aren't in the training distribution causes blank runaway. This solidified the conclusion that if you want to shorten output, you need to do it on the training side.
Measuring Sticks for This Article
From here on I'll be comparing judgment quality, so let me first summarize the evaluation sets and terminology used.
| Evaluation Set | Contents | What I Want to See |
|---|---|---|
| 87 production cases | 87 requests that flowed through the production path, with their expected judgments | Number of misroutes and safe-side cases |
| 30 implementation tasks | Implementation tasks empirically known to be solvable by weak, with 2 versions varying the environment section | Number routed to weak and Card adherence |
| 14 hard problems | Difficult problems where weak failed twice but strong passed (same as August article) | Number kept for strong |
| 10 design consultations | Design discussions that don't write code; require strong | Number kept for strong |
| 40 environment pairs | 20 pairs with the same request but only the tool availability in the environment section changed | Whether judgment changes based only on environment section |
Misrouting means sending a request that needs strong to weak; safe-side means sending a request where weak would suffice to strong. I only count misroutes as errors in one direction; safe-side cases are not counted as individual errors, but viewed as missed savings when they accumulate too much. Also, the NVFP4 classifier shifts a few judgments across vLLM restarts even with the same weights and same input, so I write numbers as ranges across 3–4 restarts, and only call a difference a difference when it falls outside that range.
Delegating the Classifier Role to Jev
I first tried non-generative judgment with the official Jev. I passed exactly the same input as the production classifier, bundled the Card into Jev's state, and the question was a binary choice: "Is weak sufficient for this request?" I ran 528 cases × 3 iterations through OpenRouter's decisions endpoint, at a cost of $0.16, with a median response time including network of 0.28 seconds. The rules for whether to adopt Jev's choice as-is or cut by confidence were frozen before looking at the results using a development split, and became: "adopt Jev's choice only when confidence is 0.9 or above; otherwise send to strong."
Jev excels at stability in keeping design consultations with strong. It kept all 10 with strong, and misroutes in the 87 production cases were also 0. On the other hand, safe-side cases were 11 (the current classifier has 1), and implementation tasks routed to weak were 10 out of 30 (the current classifier routes 20). You could say confidence 0.9 is too conservative, but relaxing it brought back misroutes in the development set, so for this use case the two couldn't coexist.
The difference was larger than expected on the 14 hard problems. Jev confidently said weak for 12 of the 14. These are the difficult problems that the August classifier could keep 5 for strong, and the current classifier now keeps 8–11 for strong. The Card has no vocabulary for difficulty, and Jev doesn't infer hardness from the request text. Similarly, in the environment pairs, there was almost no reaction when tool availability in the environment section changed. I took these two as judgments specific to the job that the current classifier acquired by training on empirically measured labels and environment lines — things a general-purpose decision model wouldn't naturally have. When I ran 14 production conversations through Jev, only 1 exceeded confidence 0.9, and 13 flowed to strong at a list price of $2.47; the current classifier sent 13 to weak at $0.32. This isn't a story about Jev being bad; it's about what happens when you delegate a classifier role built on the premise of reading a Card to a general-purpose model — it tilts toward safe-side and the savings disappear.
Reducing Lightning's Output to 1 Token
So can we make Lightning itself a non-generative classifier? Since Jev's own mechanism isn't public, I follow the approach shown by OpenJev implementations — specifically Avi Chawla's article and ekzhang/openjev-sglang. Map candidates to 1-token labels and softmax the logprob of the next token immediately after prefill, restricted to the candidates only. The ekzhang README states it's a job with only prefill and reading out the first token — no generation, no chain of thought.
For my classifier, candidates are the combinations of 10 judgment rules × weak or strong = 20 patterns, each assigned a single letter A through T. I append this legend to the end of the request text, and by specifying max_tokens: 1 and logprob_token_ids with the 20 token IDs in vLLM 0.27.1's chat completions, I get logprobs for all 20 candidates from a single prefill. Since Switchyard only reads the JSON body, I placed a small translation proxy in between to map to 4-field JSON, leaving Switchyard itself unmodified.
I first applied the production weights as-is to 1-token readout without any training. Speed was a median of 0.249 seconds. Consistent across 3 restarts — about one-quarter of the 0.95 seconds with DSpark. However, judgment quality collapsed. Safe-side cases in the 87 production examples exceeded 40, and only 1–4 of the 14 hard problems were kept for strong. Weights trained to write JSON don't know how to answer by reading a legend. If you change the readout form, you have to retrain in that form. This led to proceeding with SFT to teach 1-token labels.
The training data follows the same lineage as the August judge data: 3,000 rows of supervised data, with the assistant side replaced from 4-field JSON to a single character representing "whether this request is routed to weak at the production threshold." The recipe kept LoRA rank 8, took under 3 hours for one run, followed by NVFP4 quantization, then ran the same evaluation sets. After the first run, judgment returned to parity with the JSON classifier, with a NVFP4 median speed of 0.249 seconds, with 97% of the time in prefill. Regardless of whether training was done, if the readout form is the same, it converged to 0.249 seconds.
Note that adding DSpark to the 1-token version only increased p95 by 41 ms. Since only 1 token is requested, no draft tokens are accepted — speculative decoding works for the JSON-writing classifier, but for the 1-token classifier it only adds cost.
What Was Good and What Was Lacking in Each of the Three Classifiers
Laying out the current JSON classifier, Jev, and the 1-token version side by side, the strengths and weaknesses broke down as follows.
| Classifier | What Was Good | What Was Lacking | Current Status |
|---|---|---|---|
| Current JSON classifier | Almost no misroutes; can read hard problems and Card environment section. 0.95 sec with speculative decoding | Takes about 1 second per judgment, with 80% of that spent writing JSON | Running in production |
| Jev 1.13 | 0 misroutes. Keeps all design consultations with strong. 0.28 sec including network | Tilts safe-side, cutting requests routable to weak in half. Judges 12 of 14 hard problems as weak; doesn't read Card environment section | For general-purpose judgment. Not a replacement for this classifier |
| 1-token version | 0.25 sec, one-quarter of current. Hard problem discrimination and Card adherence equal to or better than current | Slightly more safe-side cases and environment pair misroutes. Requests not in training data require additional training data | Candidate. One step away from production deployment |
Supporting numbers are in the table below. ✅ means equal to or better than current, 🟡 means slightly short, ❌ means significantly short.
| Evaluation Set | Current JSON Classifier | Jev 1.13 (confidence 0.9) | 1-Token Version (NVFP4) |
|---|---|---|---|
| Misroutes / safe-side in 87 production cases | ✅ 0–1 / 1–2 | 🟡 0 / 11 | 🟡 1–2 / 4–5 |
| Number of 30 implementation tasks routed to weak | ✅ 19–23 | ❌ 10 | ✅ 30 |
| Number of 14 hard problems kept for strong | ✅ 8–11 | ❌ 2 | ✅ 11–12 |
| Number of 10 design consultations kept for strong | ✅ 9–10 | ✅ 10 | ✅ 10 |
| Environment pair misroutes | ✅ 1 | ✅ 1 | 🟡 3 |
| Adherence to Card environment section | ✅ +9 | ❌ No reaction | ✅ +20 |
| Time per judgment (median) | 0.95 sec | 0.28 sec (including network) | 0.25 sec |
The seconds for the current classifier and Jev measure different things, so I won't give a multiplier.
The 1-token version has two 🟡 items that are just one step away in my evaluation set. Some of the judgments that differed from the current version were actually correct in the 1-token version, and combined with the 4x speed improvement, I think one or two more rounds of training data adjustment would make it ready for production. Jev met the most important criterion of 0 misroutes, but was weak on the amount routed to weak and hard problem classification, and couldn't deliver the savings-side benefit.
Speed Is Determined by Readout Form; Judgment Is Determined by Training Data Form
Stepping back and looking at all this, two axes separated out cleanly.
The speed axis is determined not by weights but by readout form. Whether I applied the current weights without training to 1-token readout, or used the retrained version, the median converged to 0.249 seconds. As long as you make it write JSON, you can only shave 20% of decode with speculative decoding, and forcing it shorter with grammar causes runaway. The moment you stop making it write, decode disappears and only prefill remains. Jev's internals aren't public, but since it also doesn't generate, the fact that it achieves 70–500 ms on official benchmarks is a consequence of the same structure.
Those 0.25 seconds are the time to read approximately 1,250 tokens on the GB10 in DGX Spark. Prefill is largely determined by GPU compute, so it should be shorter on datacenter-grade GPUs — my local value is not the upper limit of speed. On the other hand, Jev's 0.28 seconds is the round-trip time from my machine through OpenRouter to TypeSafe and back, with the model's compute time being only part of that. A local configuration where the classifier runs on the same machine as the router doesn't have this round-trip at all. For use cases where the classifier runs on every request, I think it's worth separating whether latency is determined by GPU speed or network round-trip.
The judgment axis is determined by what the model has learned. The 1-token version produces the same judgments as the JSON classifier for requests similar to those in the training data. But for requests in forms not seen in training data — for example, a fix request with a paste of failing tests — the answer flipped between weak and strong across retraining runs. The JSON classifier judges the same request consistently every time.
The difference is whether reasoning is written before the answer. The 1-token version answers with a single character the instant it finishes reading the request, so for unseen forms it has no footing, and the answer is determined by the luck of each training run. The JSON classifier first writes the key points of the request and which rule applies, then produces a probability, so even for unseen forms it can use what it wrote as footing. In the August article I wrote that having reasoning in JSON makes debugging easier, but the act of writing the reasoning itself was stabilizing the judgment. The reasoning being written over 1 second was not wasted — that's the harvest of this experiment.
So where does Jev sit on these two axes? On the speed side, it's the same as the 1-token version — in the non-writing camp. On the judgment side, it's a general-purpose model that hasn't learned the reference tool availability in my Card or the hint that this form of task is one weak tends to fail — so it can't read those signals. Conversely, this means the strength of a specialized classifier lies not in model size but in having trained on labels created from empirical weak model measurements and environment lines. As long as those labels exist, the readout form can be either 1-token or JSON, and the recipe shouldn't be specific to Lightning 30B.
Let me note here why I use an open-weight model as the classifier. It's because I can own the judgment criteria myself. The Card that decides what gets routed to weak, the labels created from empirical weak model measurements, the thresholds, and even the output form of 1 token this time — all of it is in my hands. If I swap out the weak model I can retrain; if I realize the pass/fail line placement was wrong, I can reposition it myself. The cost per judgment is not API metering but local GPU utilization, and request text and code don't leave the machine. General-purpose decision models like Jev have no training overhead in exchange for not being able to change the judgment content yourself, so I see them as complementary rather than competing. The use-case split: specialized open-weight models for use cases where you want to own your judgment criteria, and API decision models where general-purpose judgment suffices.
Summary
I tried to speed up the Nemotron 3.5 Lightning classifier in three directions: speculative decoding, replacement with Jev, and retraining for 1-token readout. With the same weights, speculative decoding took it from 1.18 seconds to 0.95 seconds; reducing output to 1 token took it to 0.25 seconds. Judgment quality also nearly matched the current version. Speed is determined by readout form, and judgment quality is determined by the form of problems in the training data. General-purpose Jev is also in the non-writing camp on the speed side, but reading Card environment sections and discriminating hard problems are qualities only the specialized version trained on empirical labels possessed. Being able to own the judgment criteria themselves is the biggest reason to use open-weight models as classifiers.
I'll also note what remains. The 1-token version has items in my evaluation set that are just one step away, and I plan to do one more round of training data adjustment before deploying to production. The production comparison with Jev was only 14 cases run once. The classifier seconds were measured on a separate machine from production and are not values under production load. Probability calibration was not measured this time either.
Next, I'd like to try transferring this 1-token training recipe to models of 4B parameters or smaller, to see if the same classifier role can run on machines without GPU access.
Reference Links
- NeMo Switchyard v0.3.0 Release (2026-09-22)
- Fine-tuning Nemotron 3.5 Lightning with LoRA Post-Training as an LLM Router Classifier — How this classifier was built (article dated 2026-08-16)
- Reading OpenJev and Thinking About Non-Generative AI Alternatives — Comparing Jev and OpenJev (article dated 2026-09-18)
- Nemotron 3.5 Lightning First-Touch on DGX Spark — DSpark speculative decoding characteristics
- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark — Speculative decoding draft for DGX Spark
- Jev 1.13 - OpenRouter
- Models - TypeSafe AI Docs — Official values for versions, pricing, and response times
- Build your own Jev (100% local) - Avi Chawla — Explanation of reading candidate label logits
- ekzhang/openjev-sglang — Implementation of 1-token readout with SGLang
- vLLM v0.27.1 Release (uses
logprob_token_ids)
