I tried fine-tuning Nemotron 3.5 Lightning with LoRA post-training to make it a judge for an LLM router
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
After the first-touch article I wrote about Nemotron 3.5 Lightning, some of you may have been wondering how to incorporate it into actual work. Beyond its impressive specs of a 1M context 30B MoE, this model has been designed with a strong awareness of being used as a base for post-training (taking a pre-trained model and applying additional training to tailor it for specific purposes). This time, to verify that claim with real measurements, I fine-tuned it using LoRA SFT as the "capability judge" in the LLM router I use daily.
The capability judge is a model that reads requests from a coding agent and determines each time whether "this task is sufficient for a cheap model (weak) or requires a high-end model (strong)." I've been running this configuration with a router called NeMo Switchyard, using deepseek-v4-flash-0731 running on two DGX Sparks as the judge (the two-node configuration is covered in the article below, which is from August 2026). However, this model has 284B parameters. Dedicating two machines to a judgment that runs on every request is clearly too heavy a footprint. I had been thinking for a while that I'd like to replace it with a lighter model dedicated to judging.
To state the conclusion upfront: in an evaluation of 87 real-machine routing cases, it surpassed the distillation source flash-0731 (89.7% → 94.3%), and I was able to replace the judging role that 284B had been performing with a 21GB 30B model. It has also partially become capable of routing difficult problems that the teacher model was structurally unable to judge. However, along the way, there was a puzzling incident where the trained model went haywire only in the production path. The cause was the order of keys in JSON. I believe this story is the most valuable learning to share from this experience.
Since this is a continuation of previous articles, please refer to the first-touch article for Lightning's own characteristics, and the Switchyard article for the router's mechanism (both are from August 2026).
This article covers the entire flow from teacher data design through LoRA SFT, NVFP4 quantization, and real-machine routing evaluation, along with the pitfalls encountered along the way. I hope it resonates with people who are thinking about the next step beyond "measuring" a local LLM — "tailoring" it for business workloads.
Raw Lightning cannot be used as-is as a judge
First, let me cover the overall configuration. It's enough to just grasp where the judge sits within the router.
For each request, the judge returns a 4-field JSON with "the probability that the weak model will complete this task in one shot (p_solve)," and the router compares it against a threshold to determine the destination. Since judgment runs on every request, this is a quietly demanding part that requires speed, cost-efficiency, and format robustness all at once.
What happens when you put raw Lightning directly into this role? Here are the results comparing it against the current judge flash-0731 on a holdout set of 237 cases reserved for validation, not used for training.
| Metric | flash-0731 (current) | Raw Lightning |
|---|---|---|
| JSON validity check passed | 237/237 | 212/237 |
| Judgment category match | — (this is the ground truth) | 17.3% |
| weak/strong judgment match | — | 72.2% |
Even more serious than the numbers was the substance: 234 out of 237 cases were routed to the "unmatched (undeterminable)" category, and all 10 requests in production-shaped format went to strong. If everything flows to the expensive model, the router has no purpose. The state is one where it can roughly follow the format, but the vocabulary and probability intuition as a judge are entirely missing. This is where post-training comes in.
Teacher data is built from empirically measured accuracy
Teacher data refers to input-output example pairs that you want the model to imitate. This time I prepared two streams. The core of the design is using the actual measured accuracy of the weak model directly as the p_solve values (labels) on the example output side.
When I previously retrained the NVIDIA LLM Router v3, I was burned by bugs and biases in labels that had been evaluated by an LLM (article from July 2026). Learning from that experience, this time I had the weak model solve 195 coding benchmark problems twice each, and used the pass frequency as labels. The judgment vocabulary (judgment categories and a one-sentence rationale) is copied from the current judge flash-0731's judgments as examples (this is what's called distillation), while only the critical probabilities are replaced with actual measured values.
| Stream | Count | Source of labels |
|---|---|---|
| Coding benchmark empirical | 195 problems (167 training + 27 validation, excluding 1 with broken teacher response) | Weak model's measured accuracy + difficulty band smoothing |
| Synthetic tasks | 2,085 problems (1,875 training + 210 validation) | Distilled from flash-0731's judgments |
The synthetic side was generated by prompting 8 archetypes such as "brainstorming consultations," "ambiguous formatting requests," and "image-dependent requests," filling in the imbalance in judgment categories that would otherwise all become "supported" with only the empirical side. The generation and distillation calls totaled about 4,000, all fed directly to the current judge flash-0731.
Here's what one completed teacher data item looks like. The input contains the task instruction and a capability card summarizing what the weak model is good and bad at, and a 4-field JSON is provided as the example output.
{
"crux": "Sort each row descending, then for each column take the maximum value across rows and sum those maxima.",
"primary_rule": "SUP-3",
"capability_boundary": "supported",
"p_solve": 0.88
}
crux is a summary of the solution approach, primary_rule is the rule ID for the judgment rationale, capability_boundary is the judgment category, and p_solve is the weak model's completion probability. The model traces thousands of these examples to acquire the judge's vocabulary and probability intuition. And the order of these 4 fields becomes a foreshadowing for the incident in the latter half.
Another subtly important point is freezing the evaluation set. The evaluation data, including "distinguishing 14 problems" where the weak model failed twice in a row but strong would have passed, was excluded from training data with hash verification, leaving not a single case. Since this is the test problem for determining "did we surpass the distillation source," leaking it would be meaningless.
Learning twice with the official cookbook LoRA SFT
Training was based on the official cookbook from the Nemotron repository (the Megatron-Bridge LoRA recipe), with only the dataset portion replaced for judge use. LoRA is a method that adds and trains only small differences rather than rewriting all 30B weights. And SFT refers to supervised fine-tuning that has the model directly imitate the example pairs from the previous chapter. With this combination, even large models can be tailored in a short time and at low cost. The hardware was 2 H100 cards rented from NVIDIA Brev, with settings of rank 8 / alpha 32 / 2 epochs / sequence length 3072, taking 46 minutes of training + 2 minutes for merging the differences back into the original weights per run. GPU costs come to around $10 per run.
With the first training run (v0), the format issues were fixed in one shot. JSON validity checks went from 212/237 to 237/237, and judgment category matching jumped from 17.3% to 73.4%. However, among the benchmark problems set aside for validation, not a single one of the 11 difficult problems that should be routed to strong was being routed to strong. Investigating the cause, the teacher flash-0731 itself had a blind spot where it optimistically judged all problems of this shape as "solvable by weak," and the distillation had faithfully copied that blind spot. Despite having prepared empirical labels, they were losing numerically to the optimistic distillation labels on the synthetic data side.
So for the second run (v1), I adjusted the mixture. I replicated the benchmark problems with empirical labels 6 times and excluded one synthetic category with strong optimistic bias. The results were as follows:
| Metric | Raw Lightning | v0 (initial mixture) | v1 (adjusted mixture) |
|---|---|---|---|
| JSON validity check passed | 212/237 | 237/237 | 237/237 |
| weak/strong judgment match | 72.2% | 78.1% | 80.2% |
| Mean absolute error of p_solve | 0.225 | 0.151 | 0.139 |
| Number of difficult validation problems routed to strong | 0/11 | 0/11 | 2/11 |
The ability to identify difficult problems began moving just from the data mixture alone. The training recipe was not touched at all. This was a firsthand experience of the oft-heard saying that the success or failure of post-training depends on data more than the recipe.
The wall story around the training environment (2 walls of the nemo container)
Wall 1: The container specified by the cookbook cannot be pulled from outside
The nvcr.io/nvidian/nemo:26.08 specified by the cookbook is an NVIDIA internal registry image that cannot be pulled from outside. I switched to the public nvcr.io/nvidia/nemo:26.06, then checked out and inserted the 4 training recipe files for Lightning from an unmerged branch of Megatron-Bridge. It was also necessary to rewrite the internal storage paths hardcoded in the recipes to local checkpoint paths. Even with the official cookbook, "it works as-is" is not guaranteed — checking the sources of the container and recipe comes first is the lesson here.
Wall 2: API differences between 26.06 and the cookbook
The cookbook's train script assumes the 26.08 series API, and in 26.06 about 3 places around the dataset configuration have changed. I made a small modification to skip preprocessing by specifying a dummy dataset to load the pre-built training.jsonl.
JSON key order was causing the judge to go haywire
Here is the main point. When v1 was run through the actual router, about 18% of production-shaped requests produced broken responses, even though direct calls were fine. The breakage was always the same: after outputting to the final field of the JSON, it would continuously emit only whitespace characters up to the limit of 4096 tokens. Judgment counts as failed, and the failsafe routes requests to strong, so the actual harm is quiet — but with judgment taking 52 seconds per call, there's no point in moving the judgment role.
It only happens when going through the router. I diffed the requests arriving at vLLM between the router path and direct calls, and the only difference was the key order in the JSON schema within response_format. The router (Rust implementation) sorts object keys alphabetically when re-serializing the configuration. Meanwhile, xgrammar, which handles structured output (a mechanism that forces output into JSON conforming to a specified schema) on the vLLM side, enforces the order of properties in the schema as the generation order. And all 2,610 training data items in v1 had their label JSON in the same identical key order.
In other words, the model had memorized the judgment JSON as a token sequence "starting with crux and ending with p_solve." When the grammar engine enforces alphabetical order (an ordering where p_solve comes in the middle), the model loses the exit it knows and keeps emitting whitespace. When reproducing the alphabetical schema in direct calls, 4 out of 10 runs went haywire; with the same natural order as during training, it ran cleanly 53 times in a row.
I addressed this on both fronts. On the router side, I submitted a PR upstream with a one-line fix to preserve key order. And on the model side, I retrained by shuffling the key order of the label JSON per sample (v2). This is data augmentation that doesn't change a single bit of semantic content, just thoroughly covering all 24 orderings of the 4 fields. The retraining cost was around $10 per run, same as v1.
The effect was clear: 0 haywire cases in a 200-call probe with 5 different key orderings, and 0 cases in 100 real-machine calls connected directly to the router. Judgment quality was maintained at the same level as v1 (weak/strong match 80.2% → 81.4%).
| Path | v1 | v2 (key order shuffle training) |
|---|---|---|
| Alphabetical schema direct call | 4/10 haywire | 0/40 |
| Real-machine calls via router | ~18% haywire | 0/100 |
When fine-tuning for JSON output that assumes structured output, key order is not "an implementation detail that should be meaningless" — it's part of the contract. Moreover, OpenAI-compatible proxies and routers often re-serialize JSON, and key order can change at any time. As long as you train with a fixed key order in labels, this landmine can be triggered in any configuration, so if you're training JSON output, it's safer to include key order shuffling from the start.
Quantizing to NVFP4 and loading onto DGX Spark
The trained model is 62GB in BF16. It fits in the 128GB memory of DGX Spark, but when judging actual production long contexts, only the prefill (preprocessing for reading the input) takes 6.8 to 89.3 seconds, making it unusable as a judge that runs on every request. So I proceeded to quantization — replacing weights with a coarser numerical representation to compress size and computation. Using the PTQ (post-training quantization) method that doesn't require retraining, I used the official Lightning NVFP4 recipe (4/6 method) bundled with Model-Optimizer. On one GB10 unit, it takes about an hour, compressing 62GB down to 21GB.
My concern was whether quantization would break the judge's probability intuition, but the mean absolute error of p_solve on the holdout 237 cases was 0.131, not worsening from BF16's 0.137, and judgment latency settled at a median of 1.27 seconds. Even for long-context actual production, it's 1.8 to 2.6 seconds. Since the current judge flash-0731 takes a bit over 2 seconds, entrusting judgment to the 30B model delivers equal or better speed.
One point: I hit a problem where vLLM couldn't load the model after quantization. An internal suffix had been mixed into all exported tensor names, and removing it by rewriting the safetensors header resolved it. If you follow the same procedure (Model-Optimizer main branch + hf_ptq.py) and get a load error, try suspecting the tensor names.
How far did it surpass the distillation source flash-0731?
Finally, I compared the three — current judge, raw Lightning, and post-training (v2 NVFP4) — under the same conditions. Evaluation consisted of two parts: a probe of 47 labeled problems, and a real-machine evaluation flowing 87 cases through the actual router. The expected judgments here are not the teacher's output, but ground truth determined independently from whether weak actually solved the problem and from case design. Since flash-0731 also takes the same test as the one being evaluated, we can measure surpassing the distillation source on this axis.
| Metric | flash-0731 (current) | Raw Lightning | Post-training v2-NVFP4 |
|---|---|---|---|
| Match on 47 labeled problems | 38/47 | 32/47 | 45/47 |
| Expected judgment match on 87 real-machine cases | 78/87 (89.7%) | — (not deployable) | 82/87 (94.3%) |
| Number of distinguishing 14 problems routed to strong | 0/14 | — | 5/14 |
| Judgment latency (median) | 2.2 sec | — | 1.27 sec (long-context actual 1.8–2.6 sec) |
| Resources occupied by judgment role | 284B · 2× DGX Spark | — | 21GB · fits on 1 unit |
Surpassing the distillation source was confirmed empirically. In particular, the distinguishing 14 problems are a group that the teacher flash-0731 structurally fails to route a single one of to strong. The effect of mixing in empirical labels shows up here, and I feel the difference between 0 and 5 is larger than it looks. I also re-calibrated the threshold with the post-training probability distribution, updating it from 0.75 to 0.80 (80/87 → 82/87 on real-machine 87 cases).
This judge is a component that plugs into the environment running 2 open-weight models + auto-routing that the team operates (article below). The judgment that required 284B across 2 machines now fits into NVFP4's 21GB on 1 machine, and the freed-up 2 machines can be dedicated to the execution-side models. The configuration of bringing both judgment and execution locally has moved one step further.
And the first production use has begun not on the coding agent side, but on the resident agent side. I set up another system with the NVFP4 judge and Switchyard on DGX Spark, and switched NemoHermes, which runs as a resident agent, to auto-routing. In 16 hours of operation on the first day, there were zero haywire cases, with judgments properly split into 92 weak and 24 strong. The fact that a judge tailored for coding agents plugs directly into a different harness just by reading the request content is quietly satisfying from the standpoint of reusability of post-trained components. NemoHermes itself is introduced in the following article (from June 2026).
Two lineages: predictive routers and generative judges
The judge this time is a model that generates judgment text, but the NVIDIA LLM Router v3 I tried previously was a predictive type that outputs a score directly from the internal state at the point of reading the input, without generating text. Having retrained both myself, the contrast has become quite three-dimensional, so let me summarize it briefly.
| Aspect | Predictive (LLM Router v3) | Generative (this judge) |
|---|---|---|
| Judgment output | Classification head score | JSON with rationale |
| Adaptation cost | 480 problems · $11 · 5 min training | 2,610 items · $10 · 50 min training |
| Critical point in label design | Bias in LLM-judged labels | Distillation copies even the teacher's blind spots |
| Critical point in operation | Retraining whenever the pool changes | Fixing the output contract (including key order) |
Both are the same in that quality is determined by "the source of labels," and "the inference cost of the judgment model itself" is a fixed cost. Personally, the generative type felt easier to debug since judgment rationale remains in JSON, while it carries failure modes unique to generation like haywire behavior. In the long term, there are suggestions that the judgment role will be folded into the prefill of the weak execution layer rather than being a separate model, and if that happens, these two lineages may converge.
Summary
I took the Nemotron 3.5 Lightning measured in the first-touch article all the way through teacher data design → LoRA SFT → NVFP4 quantization → real-machine evaluation, tailoring it as a router judge to the point of surpassing the distillation source. This judge is already routing actual NemoHermes production traffic as the judgment role for Switchyard on DGX Spark. The training itself was straightforward at around $10 and under an hour per run using the official cookbook as a base, and what made the difference was the data. Using empirically measured accuracy as labels, the teacher's blind spots being copied through distillation, and JSON key order being part of the training data contract — these three are the takeaways from this experience.
On what's not yet done: the distinguishing 14 problems are at 5/14, still halfway there, and the quantized model has one instance of deterministic haywire behavior on specific input remaining (accepted for operation since the failsafe falls safely to the safe side). Judgment category vocabulary matching also dropped slightly as the trade-off for taking away the key-order memorization.
Next, I'd like to try QLoRA training on DGX Spark alone using the same dataset. Whether training can be completed entirely on local hardware without renting H100s, and how far quality can be matched, is planned for a follow-up article.
Reference Links
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (Hugging Face)
- NVIDIA-NeMo/Nemotron — The repository containing the LoRA SFT usage-cookbook used this time
- NVIDIA-NeMo/Switchyard
- NVIDIA-NeMo/Megatron-Bridge
- NVIDIA/Model-Optimizer — The NVFP4 4/6 quantization recipe is bundled here
- vLLM v0.27.1 Release (used for serving the post-training model)
