
I Spoke About LLM Routing at NVIDIA Agentic AI Day Japan 2026
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from the Manufacturing Business Technology Department at Classmethod.
On November 10, 2026, I gave a talk titled "From 284B to 30B: LLM Routing for Production Environments that Balances Quality and Cost" at NVIDIA Agentic AI Day Japan 2026, held at The Prince Park Tower Tokyo.
The video and slide archive of the talk has been published in an NVIDIA On-Demand playlist. It also includes all 10 talks from the day besides my own session, so please take a look at them as well.
The "284B to 30B" in the title is not about making the model that produces answers smaller. It is about my practical experience replacing the "judge" that chooses which model to send each request to, swapping the 284B DeepSeek V4 Flash 0731 for the 30B NVIDIA Nemotron 3.5 Lightning. I summarized how I further trained the judge in an article in August (the article is as of 2026-08-16).
The detailed measurement conditions and breakdowns of the numbers are all included in the video and slides in the public archive, so in this article I will briefly introduce "what concerns I started from, what I learned, and where difficulties remain." I hope it helps those who want to reduce API costs by using different models for different tasks, or who are unsure whether to build their own routing judge.

Leaving everything to a high-performance model makes costs pile up
What I used for routing this time is NVIDIA NeMo Switchyard, an open-source routing platform. By placing it between the agent and the model providers, it appears to the agent as a single model while switching the destination for each request behind the scenes.

To the agent it looks like a single model. There are multiple routing approaches, and the one covered in the talk is the "read the request and choose" approach in the red frame.
Following the NeMo Switchyard configuration, I call the high-performance, high-unit-price model strong (Kimi K3 this time) and the low-unit-price model weak (DeepSeek V4 Flash 0731 this time). They are names for roles rather than a ranking of the models.
When you entrust work to an agent, it doesn't end with a single question. In the execution log I still had, 72 tool calls ran over three rounds of implementation, review, and revision, taking about 990 seconds. Comparing output prices per million tokens as confirmed at the time, weak was 0.28 USD against strong's 15.00 USD, a gap of about 54x. If you handled everything with strong every time, costs would pile up in no time.
So how much work can the cheap weak handle? I had both solve 12 tasks prepared for evaluation, and had Claude grade them with the model names hidden.

Answers usable without major rework: weak 10 out of 12, strong 12 out of 12. Total API cost for the 12 tasks: 0.008 USD versus 0.725 USD.
As a result, weak produced answers at a level usable without major rework for 10 out of 12. On the other hand, it fell short on two: inquiry design including expense approval decisions, and system migration procedures.
That said, the goal is not to forcibly increase the proportion of weak. When I ran the three-round example from earlier with weak only, every rework added to the cost of strong reviews. As a result, the cost was almost the same as finishing in a single strong pass from the start. The key to balancing cost and quality is how to identify, at the entrance, those "two tasks that are too much for weak."
Distilling only the judge into the 30B Nemotron 3.5 Lightning
So the approach I chose is one where an LLM judge reads the request text and estimates "whether weak is enough for this." At first, based on the hypothesis that "the model itself should know best whether it can solve something," I left the judgment to the same 284B Flash 0731 as the answering weak.
However, the judge is called once every time before the main answer is produced. Running a 284B model locally just for judging occupies two NVIDIA DGX Spark units, and even via a cloud API, judging alone took a median of about 2.2 seconds. It's a state where the routing gate is too heavy, even though I wanted to save on inference costs.
So I distilled the judgments of the 284B teacher model into Nemotron 3.5 Lightning, which has 30B total parameters (3B weights used per token).

SFT on a mix of the teacher's judgments and measurements of whether weak actually solved the tasks, then NVFP4 quantization to fit on a single DGX Spark.
The judge outputs a score from 0 to 1 (p_solve) for the likelihood that weak can solve the task; if it is at or above the threshold, the request goes to weak, otherwise to strong. Simply copying the teacher's predictions would carry over the teacher's own misconceptions, so I mixed measured results of "whether weak actually solved the task" into part of the 1,775 training examples. The computation from training through NVFP4 quantization finished in about an hour and a half, resulting in a judge that fits on a single DGX Spark.
Routing accuracy did not drop even when shrunk to 30B
The following table shows the results of comparing the 284B teacher model and the further-trained 30B judge under the same conditions.
| Metric | Teacher Flash 0731 (284B) | SFT'd Nemotron 3.5 Lightning 30B |
|---|---|---|
| Routed as expected (87 real-run cases, threshold 0.75) | 78/87 | 80/87 |
| Same, after adjusting threshold to 0.80 | Not measured | 82/87 |
| 14 problems weak failed and strong solved, sent to strong | 0/14 | 5/14 |
| Hardware required to run locally | 2 DGX Spark units | 1 DGX Spark unit |
Comparing at the same threshold of 0.75, the 284B teacher scored 78/87 and the 30B judge scored 80/87. With a sample size of 87, I can't say it "outperformed the teacher," but even when shrinking the model to nearly one-tenth and fitting it on a single DGX Spark, routing accuracy did not collapse.
What I personally found most interesting is the result for the 14 problems in the third row of the table. These are problems that Flash 0731 failed twice in a row and only Kimi K3 could solve, which I set aside by excluding them from the training data. In other words, they are problems the teacher itself can't solve, yet the teacher Flash 0731 sent not a single one of these 14 problems to strong. The initial hypothesis that "the model knows best whether it can solve something" collapsed easily in situations where it can't recognize its own weaknesses. The 30B judge, trained with some measured data mixed in, managed to route 5 of the 14 problems to strong.
Placed in a workflow, a habit of "cautious at first, bold on revisions" appeared
Knowing the accuracy of one-off routing doesn't tell you whether it pays off in real work. So I incorporated NeMo Switchyard and the judge into a workflow of "implementation → review → revision (up to 3 rounds)" and tried it on six development tasks.
The API cost per run came out at 0.41 USD for all-weak, 0.67 USD for all-strong, and 0.54 USD for the router that routes using the judge, landing right in the middle. However, the bigger discovery than the overall totals came when I traced the contents of the judgment log one run at a time.

Distribution of all 24 judgments, including development tasks. All 17 initial ones had a score of 0.70 and went to strong, and all 7 revisions were sent to weak.
When I looked inside, the judge had sent all 17 initial implementations to strong with an estimated score of 0.70, and all 7 post-review revisions to weak with 0.78–0.88. This means that even for tasks that weak alone could have passed, the initial implementation was routed across the board to the safe-side strong. If you only look at the average cost, you'd conclude "it mixed them nicely," but digging into the log reveals this kind of habit in the judgments.
There were also constraints I only ran into by actually running it. When the destination model switches midway through a task, the API provider's input cache discount may stop applying and the input unit price can rise. There was also a case where the conversation history grew during a revision round and exceeded the judge's input limit. In the talk I also shared such operational points beyond accuracy.
Even for the same "please revise this," the judgment changes with how the conversation is shown
In multi-turn exchanges, behavior changes significantly depending on "how much conversation history to show the judge."

The final request text is the same on both sides. When the most recent 8 exchanges are also shown, a notation-only revision is routed to weak (0.98), while a review of the evaluation premises stays with strong (0.35).
For example, even if the last sentence is the same "Please revise this proposal," the model capability required differs depending on whether the preceding discussion was "a notation fix converting USD to US dollars" or "rebuilding from the evaluation premises." If only the final request text is shown, both are judged as strong, but when the most recent 8 exchanges are also shown, only the notation fix is cleanly routed to weak (estimated score 0.98).
However, showing history isn't a cure-all, either. When a heavy design review was requested right after a simple notation fix, there were cases where it was dragged along by the immediately preceding exchange and sent to weak. Furthermore, just rewriting an API redesign request of the same meaning into English moved the weak judgments from 4 out of 30 to 14 out of 15. When I re-ran about 1,860 judgments on a local GPU two days before the talk, the judgments that had moved with a rewrite into bullet points the previous day were swapped to move with a rewrite into English the next day.
Rather than some particular magic phrase pushing toward weak, the reality seems closer to this: near the threshold, judgments waver from surface phrasing alone. I came to realize that the "difficulty" the judge outputs is not a fixed label attached to a task, but an estimate that changes depending on what is shown and how.
When applying it to business, start with "work where there is a way to verify"
In closing the talk, based on these characteristics, I organized how to apply this to practical work, using equipment trouble response in manufacturing as an example (this is a design proposal, at a stage before measurement on real equipment). The axis I used was the perspective of "whether there is a way to verify that the result is correct."

Questions whose answers are in the manual go to weak; cause isolation and decisions spanning multiple pieces of equipment go to strong. Safety-related decisions are not left to the judge.
Inspection procedures and part number checks whose answers are in the manual are candidates for weak, since if the answer is made to cite the relevant passage, it can be verified on the spot in the field. On the other hand, cause isolation spanning multiple pieces of equipment goes to strong. And decisions involving safety or authority are not left to a probabilistic judge; as a rule, they are always fixed to strong plus human confirmation. If you decide in advance "what counts as a pass," you can gradually expand weak's scope of responsibility while measuring quality and cost.
Summary
Stop relying solely on expensive models and move over the work that a cheap model can handle, starting with the easiest. Shrink the judge from 284B to 30B so it fits on a single DGX Spark. And look beyond one-off accuracy to the cost of the entire workflow and the judgment logs. These three points are what I wanted to convey in this talk.
Of course, this verification still centers on synthetic tasks and short conversations, and challenges remain, such as fluctuation in judgment scores. That is exactly why I think there is no need to jump right into building your own router or further training a model. Start by setting a pass line for one familiar task, and compare the pass/fail and cost of the cheap and expensive models. Where a difference becomes visible is where the judge comes into play.
I introduced this at a rapid pace in this article, but the talk itself covers measurement prerequisites and detailed operational points at length with slides. If you're interested, please check out the video and slides in the NVIDIA On-Demand archive.
Reference Links
- Agentic AI Day Japan 2026 On-Demand Playlist (videos and slides)
- NVIDIA Agentic AI Day Japan 2026
- NVIDIA NeMo Switchyard — for routing approaches, see
docs/routing_algorithms/overview.md - NVIDIA Nemotron 3.5 Lightning 30B-A3B
- I turned Nemotron 3.5 Lightning into an LLM router judge with LoRA post-training (2026-08-16)
- I made the Nemotron 3.5 Lightning judge a one-token non-generating judgment, like Jev (2026-09-23)
- I tried the non-generating diffusion model Nemotron Deciffusion 8B on DGX Spark (2026-10-15)
- I tried NVIDIA's new LLM routing platform NeMo Switchyard (2026-07-03)
