I tried retraining the NVIDIA LLM Router to match my own persona (Training Edition)

Introduction

Hello, I'm Morishige from the Classmethod Manufacturing Business Technology Department.

After going through the basics and getting NVIDIA LLM Router v3 up and running, I spent a while using the default checkpoint for my own work. The default pool ships with a 9-model general-purpose pool covering Nemotron / GPT-OSS / Qwen / GPT-5 / Opus variants (v1-9models-qwen08b.yaml), and routing decisions work perfectly well with just that.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3/

After tinkering with it for a while, what started to bother me was that this default pool was "slightly off from my own preferences." The latest-generation Opus 4.8 / Sonnet 4.6 / Gemini 3.5 Flash weren't included yet, and the lineup didn't mesh well with the newer models I tend to want to call (DeepSeek V4 / Qwen 3.7 / Kimi K2.6 / GLM 4.7, etc.). The MLP training data was also built with general-purpose use in mind, so it wasn't tuned to my particular usage patterns (which tend to split sharply between heavy design discussions and light brainstorming sessions).

Fortunately, NVIDIA LLM Router v3 comes with official resources to rebuild your own checkpoint from scratch using the three steps: collect → train → evaluate. Rearrange the pool YAML to your liking, convert the questions you typically ask into training data, and you can build routing tailored to you from the ground up. Since the infrastructure was already in place, the starting point for this article was: let me try rebuilding it once, tailored to my own persona.

Specifically, I distilled my persona (the way I typically use Claude Code / Codex, blog posts I'm writing, design discussions around Hermes Agent and NemoClaw, etc.) into 480 training questions, and rebuilt a checkpoint from scratch for a new 9-model pool (including the latest Opus 4.8 / Sonnet 4.6 / Gemini 3.5 Flash). Designing around quality-preserving routing (heavy questions go to Opus, light questions go to cheaper models), I arrived at a configuration that pushes Opus 4.8 adoption to 43.1% while demonstrating cost savings of 98–99% (up to 99.3%) on light and medium-weight questions.

For LLM cost optimization, there are also approaches like task-type-based rule routing, as described in MindStudio's "Run Local AI Models with Claude Code" writeup. The reason I chose NVIDIA LLM Router v3 here was that I wanted to teach the quality × price tradeoff within the same task type (e.g., code generation where deeply-coupled work goes to Opus, but light one-liners go to gpt-oss-120b) to a Qwen3.5-0.8B encoder + PCA + MLP. If you just want to offload auxiliary tasks locally, rule-based routing is perfectly sufficient, so using them by purpose seems practical. This article focuses specifically on the use case of "someone who has been using Claude Code with Opus as the default and wants to move to routing without sacrificing quality."

Pool Redesign Philosophy

Lessons from the Failed 5-Model Configuration

In the first 5-model pool I built (Local + gpt-oss-120b + Kimi K2.7 + GLM 5.2 + Opus 4.6), feeding in 5 use cases resulted in 4 out of 5 non-trivial queries all collapsing into gpt-oss-120b. Looking at the cause, the pricing ladder was lopsided—there was a 14× cost cliff between gpt-oss-120b ($0.05 / $0.21) and the next most expensive option kimi ($0.74 / $3.50). The AUC difference from Opus was only about 5 points, and the MLP learned honestly that "the middle tier is good enough with gpt-oss-120b" when factoring in economics.

This is a subtly impactful trap when building an LLM Router: when the cost difference between adjacent models is too large, the MLP tends to favor the cheaper side. Having continuity with roughly 1.5–3× increments between adjacent models is the basic practice for building quality-preserving routing—this is something I learned through firsthand experience here.

The 9-Model Ladder

The final configuration I arrived at after the redesign is as follows.

⁽*⁾ Slot 1 is running on the OpenRouter Nano paid tier ($0.05 / $0.20) to improve reproducibility across the series, but the pool YAML's cost_per_m_*_tokens is kept at 0. If you have an environment where you can run models locally, replacing litellm_model with openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 + api_base: http://localhost:8000/v1 will switch Slot 1 to local operation. The MLP label local-nemotron-3-nano stays as-is, and routing decisions remain directly usable without retraining.

Note that slug notation varies by provider (Anthropic / Google / Moonshot / Z.AI use dot format, DeepSeek / NVIDIA use hyphen format)—this is per OpenRouter's specifications. The slugs in this table are taken directly as verified from OpenRouter's /api/v1/models endpoint. In the basics article's default pool table, the hyphen-format Anthropic slug (anthropic/claude-opus-4-6) was listed because that was the notation as written in the default config v1-9models-qwen08b.yaml at the time Part 1 was published.

The key design point is keeping the adjacent multiplier to a maximum of 2.73×. This is roughly a 5× improvement from the old 5-model's 14× jump, aimed at preventing the MLP from "gravitating toward a lower-tier model because of a cost cliff."

※ Values shown inside Mermaid are output unit prices only. Refer to the preceding table for input unit prices.

The Reality of 2026-Generation Benchmarks

Before finalizing the configuration, I wanted to get a clear picture of the capabilities of each of the 9 models—but it turned out that 2026-generation frontier models seem to have dropped the traditional MMLU / HumanEval / MBPP / Aider polyglot from their official benchmarks. Anthropic / DeepMind / Z.AI / Moonshot / Qwen have almost universally removed them from their official disclosures. The metrics appear to have consolidated around 7 axes: GPQA Diamond / AIME / HLE / SWE-bench Verified / MMMU-Pro / LiveCodeBench v6 / Arena Elo.

Here is a reorganized view focusing on GPQA Diamond and Arena Elo for practical verification (as of writing, June 2026; sourced from each company's June 2026 official releases and lmarena.ai).

※ (vendor) = vendor self-reported values; (parent) = equivalent value from parent model (Gemini 3 Pro family) used as proxy; (internal) = vendor's own internal evaluation.

What surprised me when laying this out was that the top 3 emerging models (V4-Pro / Qwen 3.7-Plus / Kimi K2.6) are virtually tied with Sonnet 4.6 on GPQA / Arena. The price difference is 17.2× (V4-Pro $0.87 vs. Sonnet $15), so there was certainly a case for "removing Sonnet from the pool"—but I kept it in due to the operational convenience of running 9 slots and because I wanted to capture any edge cases where Anthropic's strengths in long-form coherence and safety nuances make a difference. As described later, Sonnet 4.6 ended up holding a 23.1% adoption rate and reliably filling the "medium-heavy quality middle" role.

Gemini 3.5 Flash's Role and Securing the Multimodal Path

Another tricky decision in the pool design was how to handle Gemini 3.5 Flash. While its multimodal capabilities stand out (MMMU-Pro 83.6), the NVIDIA LLM Router v3's Prefill Router is text-only by design (it takes hidden states from the prompt using the Qwen3.5-0.8B encoder), meaning image / audio / video blocks have no effect on routing decisions. With short prompts like "Please analyze the following image," the moment routing selects DeepSeek or Local Nemotron, OpenRouter returns 404 No endpoints found that support input image.

The approach here was to cover Gemini's text-based roles (Google grounding / diagram structuring / Arena Elo 1480 general quality) through training data, while handling the multimodal path by adding a thin shim at the litellm adapter layer to handle it separately. It runs on a static capability lookup for the 9-model pool (image → {gemini-3.5-flash, sonnet-4.6, opus-4.8} / audio · video → {gemini-3.5-flash}). In hands-on testing, a 10×10 PNG / 1-second WAV / 1-second MP4 were all routed to Gemini 3.5 Flash and returned image, audio, and video analysis responses. I've submitted this shim as a proposal upstream as PR #34, so feel free to check that out if you're interested. If you want to build full-fledged multimodal routing based on CLIP (as in NVIDIA AI Blueprint v2's Auto-Router), that's a different story—but for the practical need of "properly routing image / audio / video requests to capable upstreams," this thin shim should cover it adequately.

Collect: Question Data Design

Overall Structure

The training data consists of 480 questions total, broken down as follows.

The 150 Opus-favored questions are the core of this configuration, designed with the intent to "heavily embed heavy-weight signals where only Opus can answer correctly, so the MLP doesn't get outweighed by the economic rationale of emerging models." The 30 Gemini-favored questions reinforce the learning of Gemini's text-based distinctiveness (grounding / diagrams / structuring). Sonnet-favored questions were intentionally excluded (since Sonnet ties with the top emerging models on GPQA, making differentiated questions hard to calibrate—so I let the natural distribution handle it). The breakdown ends up as 210 heavy-leaning questions (44%) + 30 Gemini-leaning questions (6%).

Building the 100 Personal Persona Questions

Articulating the patterns of questions you typically send to Claude Code first makes it easier to balance against the Opus-favored questions.

The latter 60 questions (refactoring / infrastructure / PR / issue) are clearly heavy-weight, so they act as strong signals for "heavy = Opus."

Building the 150 Opus-Favored Questions

I organized the scenarios where Opus 4.8 clearly outperforms the top emerging models by characteristic, then structured the 150-question breakdown accordingly.

Detailed question examples will be placed in nvidia-llm-router-v3-training/data/questions-opus-favor-150.txt on GitHub. One thing I was careful about when writing the questions was "not making them too abstract." Questions like "Please explain ○○" don't produce much differentiation from the top emerging models, but framing them as "Please discuss the X aspect of ○○ in N steps within the constraints of Y and Z, in 600 characters" with explicit constraints tends to bring out Opus's true value.

Building the 30 Gemini-Favored Questions (Text-Only)

This is the part where I wanted to teach the MLP Gemini 3.5 Flash's distinctiveness. Since the multimodal path itself was separated into a different layer via the adapter shim described earlier, I prepared 30 questions focused on text-based signals I wanted the MLP to learn (grounding / diagrams / structuring).

The key is explicitly including grounding-required keywords like "as of June 2026 or later" and "the latest." While the Prefill Router can only make decisions based on the pre-call text, prompts carrying these keywords have characteristics of "text requiring Google Search grounding," which serves as a signal that indirectly nudges the MLP toward selecting Gemini. The actual grounding then runs on the Gemini side once it's called.

Sampling the 100 Public Dataset Questions

For bias avoidance, I sampled 100 questions from a combination of MMLU 30 + HumanEval 15 + GSM8K 15 + DollyJA 40. These were deterministically extracted (with a fixed seed) from a pre-built pool of 300 questions, making the process reproducible.

Running the Training Pipeline

Step 1. Environment Prerequisites and Connectivity Check

The setup is DGX Spark (aarch64, GB10, 128GB UMA) + vLLM Nemotron 3 Nano 30B-A3B NVFP4 (local) + OpenRouter (remaining 8 models). Use probe-models.py to send Reply with the single word 'pong' to each model and confirm HTTP 200 / latency / cost.

# Please adapt the verification scripts from this article to your own working directory
cd workspace/blog/scripts/nvidia-llm-router-v3-training
uv run --with httpx --with pyyaml scripts/probe-models.py \
  --config configs/my-pool-9models.yaml

Once all 9/9 show OK, proceed to the next step.

Step 2. Dry-Run (10 Questions × 9 Models)

Before the full collect run, run a dry-run with 90 calls to estimate cost and latency. I hit about 2 thinking-related traps here, but after 3 attempts I arrived at a fully working 9/9 OK configuration using the approach of disabling thinking via extra_body.reasoning.enabled: false with an exception for Gemini 3.5 Flash using max_tokens: 2048. The actual cost came out to $11.27 for the full 480-question run—about 1/4 of the pre-estimate of $45–55.

Step 3. The judge=vote Trap and Switching to judge=llm

After completing the dry-run, submitting the full collect with judge=vote produced per-model accuracy numbers that made me do a double take: "Local Nemotron 99.8% / all other models 0–0.2%." The cause was in the _judge_vote logic in collect.py: when each model returns a different natural language response, the Counter ends up with all equal counts, and insertion order causes the pool's first entry (Local Nemotron) to always be treated as "correct." This works for MMLU-style option matching, but it's unusable for generative tasks.

Switching to judge=llm + Sonnet 4.6 as judge and re-running collect (7h 15m), a reasonable distribution emerged—Opus at the top, the top 3 emerging models clustered together, and Local / GLM at the bottom—making this usable as a quality signal.

⁽*⁾ Sonnet 4.6 also serves as judge, so self-scoring bias applies

def _normalize(text: str) -> str:
    return " ".join(re.split(r"\s+", text.strip().lower()))

def _judge_vote(outputs: list[str]) -> str:
    normalized = [_normalize(o) for o in outputs]
    counts = Counter(normalized)
    return counts.most_common(1)[0][0]

model-router collect \
  --config configs/my-pool-9models.yaml \
  --questions data/questions-9models.txt \
  --judge llm \
  --judge-model openrouter/anthropic/claude-sonnet-4.6 \
  --output data/collected-my-pool-9models.csv

Step 4. Train (Completes in 5 Minutes)

model-router train \
  --config configs/my-pool-9models.yaml \
  --data data/collected-my-pool-9models.csv \
  --output-dir checkpoints/my-router-9models/

Training runs with accelerate enabled, a 9-dim MLP, and the Qwen/Qwen3.5-0.8B encoder. Stages 1–6 (Load → Extract → PCA → MLP ensemble → Calibrate → Save) completed in 5 minutes and 1 second.

The per-model AUC for the completed checkpoint is as follows.

All models achieved AUC above 0.85, exceeding the target value I had in mind (per-model AUC 0.5–0.9). Compared to the default checkpoint where p_max was 0.07–0.10, it's striking how much tighter the MLP's predictions become just by preparing proper training data.

Step 5. evaluate (Quality Improvement in Numbers)

model-router evaluate \
  --config configs/my-pool-9models.yaml \
  --checkpoint checkpoints/my-router-9models/prefill_router.pt \
  --data data/collected-my-pool-9models.csv \
  --output results/eval-my-router-9models.json

The quantitatively visible figure here is: "Compared to 69.17% accuracy when Opus alone solved 480 questions, the router achieves 79.37%, an improvement of over 10 percentage points." The router captures 43.7% of the gap between that and the Oracle accuracy (the upper bound when the best model is always chosen) of 92.5%.

Let's also look at the argmax distribution from the evaluation.

Opus 4.8 leads with 43.1% adoption, followed by Sonnet 4.6 at 23.1% with a remarkable 99.1% accuracy. Gemini 3.5 Flash also comes in at 12.3% with 98.3%. During the design phase, I had said "Sonnet would be selected only occasionally," but when the lid was opened, it turned out to be surprisingly stable as the main player in the mid-to-heavy quality middle tier.

It may seem counterintuitive that Opus's 64.7% is lower than Sonnet's 99.1%, but this is simply because "the difficulty of the sets selected by argmax differs." The 207 questions for which Opus is chosen via argmax are hard question clusters where the MLP has determined that "P(correct) won't be high unless it's Opus," so even Opus faces a higher bar to be judged correct, dropping to 64.7%. Sonnet / Gemini's 99.1% / 98.3% are results of comparatively easier questions being routed to them, as the MLP determined "these can be solved without Opus." Just a note that the previously mentioned 69.17% for Opus alone is aggregated across all 480 questions, so it comes from a different set than the 64.7% here (a 207-question subset).

Conversely, Local Nemotron and GLM 4.7-Flash end up at 0% with argmax. Their accuracy itself is around 38–39%, but with tolerance=0 argmax judgment, they always lose because "other models produce higher P(correct) for the same question." In this configuration where training data is skewed 44% toward heavy models, they have no opportunity to be selected. If tolerance is raised to 0.20, they will properly be picked for lightweight questions.

Behavior of the Persona-Optimized Checkpoint

Routing Demonstration by Tolerance (bench_persona_3tol)

bench_persona_3tol.py is a script that sends 5 representative question types (casual chat / code generation / technical explanation / math proof / philosophy) at 3 tolerance levels (0.05 / 0.10 / 0.20), comparing the routing distribution and cost at each level. I brought up the new checkpoint with model-router serve --port 8204 and ran it.

The (All-Opus) row uses the bench script's comparison baseline model fixed to anthropic/claude-opus-4-6 as-is, so it's the old Opus 4.6 rather than Slot 9 (Opus 4.8) in the new pool. The output price per token is similar, so it's fine as a comparison baseline, but if you want to match the figures precisely, rewrite OPUS_MODEL in the bench script to claude-opus-4.8.

The 5 bench questions are all "lightweight to medium," so Opus not appearing is as expected. Even at tol=0.05, deepseek-v4-flash dominates the lightweight tier — this looks like a gap compared to V4-Flash being selected only 4.8% in the evaluate (480 questions / argmax), but that's because the 5 bench questions are skewed toward the left tail (lightweight side) of the evaluate difficulty distribution. When tolerance is raised to 0.20, Local Nemotron also gets picked up, making lightweight questions virtually free.

Interpreting the Routing Distribution and Sonnet's Role

Let me organize the point that routing behavior looks completely different between evaluate (480 questions / argmax) and bench (5 lightweight-medium questions / with tolerance).

This is the ideal form of the design — routing automatically switches based on question difficulty and tolerance — realized as intended. There is a clear contrast: heavy questions are handled by Opus / Sonnet to preserve quality, while lightweight questions are handled by lightweight models to cut costs.

At the design stage, my stance was "Sonnet ties with the emerging top tier (V4-Pro / Qwen 3.7-Plus / Kimi K2.6) on GPQA, so I won't add differentiation questions in training data — it'll be selected only occasionally," but evaluate showed 23.1% adoption / 99.1% accuracy. Since argmax (tol=0) doesn't factor in cost, "it was chosen because it's cheap" doesn't explain it. The plausible explanations are: (1) using Sonnet 4.6 as the judge introduced self-scoring bias, boosting Sonnet's correct labels and causing the MLP to learn higher P(correct) for Sonnet; or (2) for the 111 selected questions, the MLP's Sonnet predictions genuinely surpassed Opus. A clean separation requires re-collecting with a different judge (GPT-5.4 or Gemini 3.5 Pro), but the fact that Sonnet is functioning stably as a mid-to-heavy quality middle tier doesn't change — the ideal form of "maintaining quality while also reducing costs" is working quantitatively.

How to Increase Local Nemotron's Adoption Rate

The evaluate result showed Local Nemotron at 0% with argmax, and only 2/5 adoption with bench tol=0.20. If you want to "route more brainstorming-type queries to Local," you can tune this in 3 stages: (1) add extra_body.routing.tolerance: 0.20 to the request body to raise tolerance per request (easiest, no restart needed); (2) spin up a separate server on a different port with --models restricted to only the lightweight tier; (3) add 50–100 questions of casual chat / short translation / simple memo organization to the training data and re-collect / re-train. In practice, (1) is the most convenient, and using it per Hermes Agent profile — "tolerance=0.20 for casual chat, 0.05 for code generation via Claude Code" — lets you handle multiple use cases with a single routing service. The topic of integrating with flows running on the Hermes side is planned to be covered in the practical edition.

The Cost Cliff Visible Through Comparison with the Old 5-Model Checkpoint

For reference, I'll also show the routing distribution of the old 5-model pool (Local Nemotron / gpt-oss-120b / Kimi K2.7 Code / GLM 5.2 / Opus 4.6) mentioned at the beginning of the pool redesign. When 5 use cases were submitted, all 4 non-lightweight cases were concentrated on gpt-oss-120b, with p_max 0.79–0.94 showing high confidence, but Opus / Kimi / GLM were never called — a structure where they'd never get a turn.

When placed side by side with the new 9-model ladder, the root cause — the 14x jump — is clearly visible even visually.

※ All prices within Mermaid are unified as output prices per token. For input prices, please refer to the respective tables shown earlier.

As a basic practice when building your own routing, being conscious from the start that "the cost multiplier between adjacent models should be within 3x, ideally around 2x" makes it easier to avoid cost cliffs like this.

Summary

Starting from a feeling of "this might be a little different from my preferences" after thoroughly trying out the NVIDIA LLM Router v3 default checkpoint, I went all the way through rebuilding a 9-model pool from scratch and running persona training on 480 questions. The result: with Opus 4.8 adoption at 43.1% preserving quality, while achieving 98–99%+ cost reduction (up to 99.3%) with tolerance 0.05–0.20 for light-to-medium questions, the effectiveness of quality-preserving routing became tangible. The pitfalls encountered along the way (thinking on consuming all max_tokens, the insertion order bug in judge=vote, Gemini's mandatory reasoning, the 14x cost cliff jump) are written down in detail in hopes that they'll serve as reference for others building their own routing for different use cases. In particular, the pool design principle of "cost multiplier between adjacent models within 3x, ideally around 2x" is a basic practice worth being aware of from the start.

To avoid "just building and stopping," observation after putting the checkpoint into production also becomes important. The routing service used in this article has Langfuse callbacks and thin stdout logging of routing decisions, making it possible to retrospectively confirm how many times each model was called, what the costs were, and how the distribution shifts when tolerance is changed. I'm thinking about covering this observability topic more leisurely in a separate article.

Even with "reference implementation only" written on it, v3 is this much fun to play with — the subtle enjoyment of watching it train and smoothly conform to your own persona is genuinely satisfying.

Reference Links

NVIDIA LLM Router

• NVIDIA-AI-Blueprints/llm-router (GitHub) — The repository including the v3 branch covered in this article
• NVIDIA LLM Router で LLM の用途別使い分け環境を構築してみた（基礎編） — Series Part 1, basics of default checkpoint behavior and the OpenAI-compatible endpoint

Upstream PRs Submitted During Verification for This Article

• PR #32: forward all OpenAI-compatible fields to upstream — Proposed fix for the issue where OpenAI-compatible fields such as tools were being dropped
• PR #33: translate upstream errors to OpenAI-compatible responses — Proposed fix for the issue where upstream 4xx errors were being returned as 500
• PR #34: route image/audio/video requests to capability-matched models — Proposed adapter shim for routing multimodal content to capability-matched models

Related Topics

• OpenRouter — API gateway used as upstream for this article's pool
• Langfuse — Integrated via LiteLLM callback for observability
• lmarena.ai — Source of Arena Elo referenced in the benchmark tables in this article