I Tried a Non-Generative Decision Model, CLM, as the Judge for Switchyard on DGX Spark

Introduction

Hello, this is Mori Shigeru from the Manufacturing Business Technology Department at Classmethod.

A model called CLM (Contrastive Language Model) has been released under Apache 2.0. It has Language Model in its name, but it doesn't write text. All it does is embed the current state and the candidate actions separately, then pick the candidate that fits best. The README sets TypeSafe's Jev as the comparison target, and also advertises that it can be called with the same API shape as Jev.

https://github.com/Contrastive-LM/CLM

Just recently I covered non-generative judgment in two articles. One compared implementations of OpenJev (an article as of 2026-09-18), and the other rebuilt the judge of Nemotron 3.5 Lightning, which routes requests to weak and strong in NeMo Switchyard, into a 1-token judge (an article as of 2026-09-23). In the latter I had Jev make the same judgment too, so the evaluation sets and the comparison method are still on hand. Now that a general-purpose decision model with published weights has appeared, there's no reason not to put it on the same yardstick.

https://dev.classmethod.jp/articles/openjev-non-generative-ai-alternatives/

https://dev.classmethod.jp/articles/dgx-spark-lightning-judge-single-token-jev-inspired/

To state the conclusion first: zero-shot CLM did not reach Jev as a judge for Switchyard, but retraining only the head got it to Jev-level accuracy in about 8 minutes, including embedding. When I gave it the same question as Jev zero-shot, it answered weak for almost every request, including those that need strong, and on the unseen 237 cases it had 42 misroutes against Jev's 10. On the same 237 cases, the AUC of the retrained head was 0.892, above Jev's 0.856. However, it also learned the biases that were present in the training data. The speed is as designed: if the candidate embeddings are in the cache, one judgment takes about 110 ms whether there are 2 candidates or 1000.

In this article I organize how CLM works and its strengths from primary sources, then run it on DGX Spark and show the results of putting it next to Jev and the current judge on the same 528 cases as the 9/23 article.

CLM embeds the state and the actions separately and compares them

The inside of CLM is simple. It turns text into a 4096-dimensional vector with a frozen Qwen3-8B, and puts two small MLPs on top. One is the state head, which maps the state (the current conversation or question) into 512 dimensions; the other is the action head, which maps a candidate action into the same space. The score is the cosine similarity of the two vectors multiplied by a learned scale, and a softmax over the candidates turns it into probabilities.

Training uses bidirectional InfoNCE, training both the direction of predicting the correct action from the state and the direction of predicting the correct state from the action. According to the README there are three stages: about 60 million question-answer pairs from Nemotron DQA build the foundation, about 30 million confusing wrong answers generated by Gemini 2.5 Flash-Lite teach fine-grained discrimination, and about 1 million agent trajectories tune it toward choosing actions. The README also shows that the training loss falls as a power law in compute, model size, and data size, and scaling to larger backbones is placed at the top of the roadmap.

CLM's strengths are reusing candidates and being able to retrain locally

Now that we understand the mechanism, let's look at what CLM advertises.

The first is that the state and the candidates are embedded separately. The candidate-side embeddings can be reused once computed, so even if candidates increase, all you pay for at each judgment is one encode of the state. The README explains that this property lets the same call be used for tool selection by a running agent, selecting one from multiple answers, and narrowing search results. The model card's "13x faster than Jev at about 1000 candidates" also comes from this property, and I verified it on DGX Spark in the second half.

The second is that the way of calling it matches Jev. clm-serve returns the TypeSafe-compatible POST /v1/systemone and accepts as-is the three types: Noul, which returns yes/no probabilities; Choice, which picks one from candidates; and Score, which scores in ordered steps. The bundled T-Rex comparison script changes only the base URL and API key between CLM and Jev. There are also POST /v1/rank, which freely reorders candidates, and a playground where you can write a state and question in the browser and try it out.

The third is that both the weights and the code are Apache 2.0, so you can run it on your own GPU and retrain it. What you train is not Qwen3-8B but only the heads on top, and train/finetune.py is published. The head retrained for DeepSWE is on Hugging Face, and the procedure to reproduce that result is in the README.

Placed next to Jev, the differences are as follows. The CLM figures are claims by the CLM side, written in the README and model card.

Let's also look at the figures published by the CLM side. First, the 4 zero-shot tasks without retraining. I extracted the numbers from the figures in the README.

The README sums this up as "on par with Jev, with up to 9x lower latency." Jev is slightly ahead in success rate on BFCL and WikiRacing, and the speed comparison has caveats. In the bundled T-Rex instructions, CLM runs on a single RTX 4090, while Jev is the jev-latest hosted by TypeSafe, called over the network, with time measured on the client side.

The other headline is the result of retraining the head and using it as a verifier, that is, a role that picks the likely correct one from multiple candidate answers.

The candidate answers were generated by Opus 5 for DeepSWE and Fable 5 for Terminal-Bench 2.1, and the latency is on an H100. Jev falls below the one-attempt solve rate on both tasks, and the README says it did not work as a verifier for long tasks. That a small retrained head pushes the accuracy of the selecting role above the one-attempt solve rate is quite interesting as a direction. However, these are values for the retrained head, and the evaluations are only 38 and 30 cases. The model card's limitations section also says these values are not for the zero-shot checkpoint.

Running CLM on DGX Spark

The setup is two processes. Serve Qwen3-8B for embeddings (pooling) with vLLM, and put clm-serve in front of it.

There were two easy places to get stuck on DGX Spark. The first is that pip install contrastive-lm tries to resolve the dependency vllm and, on aarch64, falls back to a source build of an old vllm and fails. The package itself is pure Python, so installing it with --no-deps inside the vLLM aarch64 image works. The second is the input limit. States exceeding the default 2048 tokens are truncated, and this time the inputs were up to 2,883 tokens, so I raised both vLLM and clm-serve to 4096.

# embedding server (Qwen3-8B, last-token pooling)
docker run -d --name clm-emb --gpus all --ipc host --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface --entrypoint vllm \
  vllm/vllm-openai:v0.25.0-aarch64 serve Qwen/Qwen3-8B --served-model-name qwen3-8b \
  --runner pooling --enable-prefix-caching --max-model-len 4096 --gpu-memory-utilization 0.30 --port 8090

# clm-serve (heads are downloaded to ~/.cache/clm on first start)
docker run -d --name clm-serve --gpus all --network host \
  -v ~/works/clm/wheels:/wheels:ro -v ~/.cache/clm:/root/.cache/clm \
  --entrypoint bash vllm/vllm-openai:v0.25.0-aarch64 \
  -c "pip install -q --no-deps /wheels/contrastive_lm-0.1.0-py3-none-any.whl && exec clm-serve --port 8700 --max-tokens 4096"

/v1/models lists two models, clm-latest and clm-raw. clm-raw is a comparison model that skips the heads and chooses by the cosine similarity of the raw embeddings.

This article's yardstick

The evaluation used the same 528 inputs as the 9/23 article. The Switchyard judge receives a system prompt together with the request's conversation. The system prompt contains the judgment procedure, the rules on which work can and cannot be left to the weak-side agent, and a runtime-environment section listing the tools the requesting agent can currently use. The runtime-environment section is the part that gets inserted to match each request's environment. For CLM and Jev, I pass these rules and the runtime-environment section together with the conversation as the state, and ask the binary question "is weak enough for this request?"

A misroute is sending a request that needs strong to weak, and the safe side is sending a request where weak is enough to strong. The only errors counted are misroutes. The decision rule was also set by the same procedure as Jev. I looked only at the development split (a separate 40 cases and half of the production 87) to choose "at what confidence or higher do we take the choice, and below that send to strong," and froze it before looking at the remaining sets. CLM returns the same answer for the same input so it is 1 iteration, while Jev is a majority vote of 3.

With the same question as Jev, almost everything became weak

First, I asked CLM with exactly the same question text and option descriptions as Jev. The result is that almost everything is weak.

On the 10 design-consultation cases that should need strong, the median P(weak) was 0.967. On the 14 discernment problems the median is 0.975. The probability bands almost overlap with those of the implementation tasks where weak suffices. The rule was frozen at confidence 0.9 or higher, but CLM's confidence almost always exceeds 0.9, so passing through the rule doesn't change the conclusion. On the production 87, it sent 23 of the 25 cases that need strong, 9 of the 10 design consultations, and all of the 14 discernment problems to weak.

Maybe the probability band is just narrow and the ordering still carries information. With that thought, I also computed AUC, which does not depend on a threshold (the probability that a request needing weak is given a higher P(weak)). 0.5 is a coin flip and 1.0 is a perfect separation.

The overall values exclude the 40 cases used to freeze the rule. CLM stays close to 0.5, and on the production 87 it is ranked in the opposite direction at 0.41. Looking at the 17 environment pairs where the correct destination differs between E0 and E1, it gave the higher P(weak) to the weak side in only 3 pairs. In the range I tried preliminarily on the production 87, passing the state as a JSON string (0.424) or removing the head (0.112) does not improve it. Making the question English raised it to 0.86, but it dropped on other sets, so changing the language does not seem to solve it.

Rewriting the candidates into descriptions of the destinations

What I wonder here is what CLM is comparing against what. Reading the implementation, what gets embedded as a choice candidate is the description text of the option itself. The description I wrote for Jev is a long criterion like "A request an efficient agent can handle. The spec is in the request text or...", and the difference between the weak and strong descriptions comes down almost entirely to the presence of a negation. If pre-training is about the correspondence of questions and answers, and post-training is about the correspondence of a state and the next action, then writing the destination itself might mesh better than an abstract criterion sentence. With that in mind, I tried three ways of writing the candidates.

What I looked at for selection was only the production 87 and the development split used to freeze the rule. The unseen sets were run after settling on one way of writing.

With the last way of writing, I also changed the question text to "which model should handle the latest request?" and made the options descriptions of the destination models. The contents are as follows.

{
  "question": "Which model should handle the latest request?",
  "criteria": {
    "weak": "Nemotron 3.5 Lightning 30B-A3B, a small efficient coding agent with read_file, apply_patch, run_tests and lookup_docs",
    "strong": "Claude Opus, a frontier model for design, ambiguous specs, investigation and high-stakes work"
  }
}

The AUC on the production 87 rose from 0.414 to 0.936. With the same system prompt and conversation, the way of writing the candidates alone moving it this far is a bigger difference than I expected. However, since I chose this way of writing by looking at the production 87, 0.936 is a number on the side I chose on. What verifies it is the unseen sets in the next chapter.

Putting it next to Jev and the current judge on unseen sets

I fixed the candidates written as destination descriptions and froze the rule again on the development split (adopt at confidence 0.9 or higher, below that strong). Then I ran the sets not used for selection, and the results are in the table below. Jev is with the same question as Jev, and the current judge is the value of the JSON judge running in production.

CLM can now read differences in the runtime environment too. Of the 17 environment pairs where the correct destination splits, it gave the higher P(weak) to the weak side in 16, close to Jev's 17. However, when the runtime-environment section is included, P(weak) drops across the board, and the median for implementation tasks (with environment) was 0.805. Since the rule is frozen at confidence 0.9, all 30 of these cases fell to strong, and on the environment pairs the safe side also swells to 33. The direction is read, but the level of the probability moves with the presence of the section.

The 0 misroutes on the 14 discernment problems look at first glance like a win over both Jev (12) and the current judge (3). But looking at the contents, in all 14 the confidence did not reach 0.9, and they simply fell to strong by the rule. The median P(weak) was 0.494, and it is not that CLM itself chose strong. The 10 design-consultation cases fall the same way.

The difference showed clearly on holdout, where misroutes were 42 for CLM and 10 for Jev. The overall AUC is also 0.681 for CLM against 0.901 for Jev. In the part that reads the rules of the system prompt and estimates how hard it is to say whether weak is enough for the request, CLM did not reach Jev zero-shot.

Judgment time is determined not by the number of candidates but by encoding the state

Apart from the quality of the judgment, the speed was as CLM designed. First, the time of one judgment this time. Called from a Mac over Tailscale, p50 was 0.15 seconds. Jev, measured the same day, was 0.24–0.26 seconds including the network via OpenRouter, and the 1-token-judge Lightning from the 9/23 article is 0.25 seconds.

To remove the network, I also measured the server-side value by calling directly from on node2. On the first pass with the embedding cache empty, p50 was 117.5 ms, and when the same state is sent a second time it is 1.1 ms. On the second time the state's embedding is also in the cache, so it is only the head's computation and comparison.

When increasing the number of candidates, I passed N synthesized tool descriptions to /v1/rank and measured. From the second time on, I sent a new state with a unique marker at the start. This is a condition that hits neither the embedding cache nor vLLM's prefix cache.

The first time with 1000 candidates takes 6.5 seconds, but once they are loaded, you can choose from 1000 in about 110 ms, the same as with 2. Most of the time is the state-side encode, and the number of candidates barely matters. Seeing this shape, the README's "the more candidates, the wider the speed gap" makes sense.

Retraining only the head matched Jev's discrimination

Is the zero-shot weakness because Qwen3-8B is not reading the state, or because the head simply doesn't know how to estimate this job? Since CLM lets you retrain only the head, I tried it.

The training data is the same 4,511 cases that I used to train the 1-token Lightning in the 9/23 article. Each one has the system prompt and conversation that the judge receives, and weak / strong labels made from measurements. The state is assembled exactly the same way as in evaluation, and I confirmed there is no overlap with the 6 sets used for evaluation. I set aside 10% by task as validation, and trained with the defaults of train/finetune.py, starting from the published head.

python3 train/finetune.py --task choice --data /w/data/choice-mn --workflow all --out-dir /w/runs/mn-infonce-hard \
  --embed-url http://127.0.0.1:8090/v1/embeddings --served-model-name qwen3-8b --max-len 4096 \
  --init-ckpt /root/.cache/clm/CLM_v0.1-8B.pt --targets hard

The time taken, including the time to embed the 3,718 distinct training texts with Qwen3-8B, was 7.85 minutes. The second run, which reused the embeddings and changed only the seed, took 0.13 minutes, so the head training itself took less than 10 seconds. When I trained the 1-token Lightning with LoRA on the same data, the whole thing took about 3 hours 50 minutes with a peak memory of 65.1 GiB. The weight of a single trial differs by orders of magnitude.

I loaded the retrained head into clm-serve under a separate name, froze the rule by the same procedure as zero-shot, and measured; the results are in the table below. For the question text, I trained with two variants: the destination descriptions, and the same question as Jev.

On the 14 discernment problems, unlike with zero-shot, the head itself now chooses strong. The head trained with the same question as Jev has a holdout AUC of 0.892, above Jev, and its misroutes are 6, fewer than Jev's 10. There was also a reversal: zero-shot, the destination descriptions were overwhelmingly better, but after training the criterion sentence identical to Jev's gives higher discrimination. Even retraining with a different seed, holdout AUC goes from 0.854 to 0.86 and misroutes from 17 to 15, staying at about the same level.

Shortcomings clearly remained too. The safe side on the production 87 is 47 under every condition. The 1-token Lightning, on the same training data, had 4–5. To investigate the cause, I inserted, at inference time, a section saying the tools are usable into inputs that did not contain a runtime-environment section. For the head trained with destination descriptions, the safe side dropped from 47 to 2, but holdout misroutes jumped from 17 to 61. This head had learned the presence of the section as a strong cue toward weak, separate from the difficulty of the request. In the training data, all the fix requests of a form close to the production 87 had the section, so it picked up that bias. For the head trained with the same question as Jev, adding the section barely moved the production 87. I suspect this one reads the production requests themselves, whose phrasing differs from the training data, as leaning toward strong. A frozen encoder and a small head may generalize less well outside the training data's phrasing than a 30B LoRA. However, separating these two causes was done after seeing the production 87 results, and I have not yet verified it.

What of Switchyard to leave to a general-purpose decision model

Lining up the results this time, it becomes clear that the work CLM is good at and the work required of Switchyard's judge have different shapes.

Where CLM is fast and works straightforwardly is on work where the candidates are concrete actions, numerous, and rarely change. Examples are narrowing down what to call next when there are hundreds of MCP tools, reordering search result candidates, and picking one from multiple answers. Since candidate embeddings can be reused, judgment time barely increases even if candidates increase. That discrimination came back when I rewrote the candidates as destination descriptions is, I take it, the flip side of CLM having been trained to choose concrete actions.

On the other hand, Switchyard's capability judgment has only two candidates, weak and strong, and instead needs to read the system prompt's rules and the runtime environment and estimate the difficulty of the request. With 2 candidates there's almost no role for an action cache, and what makes the difference is how deeply the state side can be read. CLM could read the direction of runtime-environment differences too, but the difficulty estimation lay with the Lightning judge trained on measured labels in the 9/23 article, and with Jev.

That said, this difficulty estimation was recovered quite a lot just by retraining the head. The Qwen3-8B embeddings contain material for the judgment, and what was lacking was how to read it. The prefill router I looked into earlier was also built by putting a predictor trained on my own measured labels onto the internal state of a frozen encoder (an article as of 2026-08-22). Rather than putting a general-purpose decision model in as-is, using published weights as a base and tailoring a head with my own labels looks better suited to a router's judge. Even in that case, picking up the training data's biases is something to watch even more than with a LoRA judge.

https://dev.classmethod.jp/articles/dgx-spark-switchyard-prefill-router-vs-judge/

Summary

CLM is a general-purpose decision model that puts two small heads on a frozen Qwen3-8B and compares the state and candidates by embedding them separately. The weights and code are Apache 2.0, it can be called with the same API shape as Jev, and only the head can be retrained locally. On DGX Spark too, if the candidate embeddings are in the cache it can choose from 1000 candidates in about 110 ms, confirming that the design suits selections with many candidates.

Measuring Switchyard's weak / strong judgment on the same 528 cases as the 9/23 article, zero-shot with the same question as Jev, P(weak) stuck around 0.94–0.99 and the AUC was 0.581. Even after rewriting the candidates as descriptions of the destination models, holdout misroutes were 42 against Jev's 10. On the other hand, retraining only the head on the same 4,511 cases as the 1-token Lightning brought the holdout AUC to 0.892 in about 8 minutes including embedding, above Jev's 0.856. In exchange, it also learned training-data biases such as the presence or absence of the runtime-environment section, and the safe side on the production 87 rose to 47. Please bear in mind that I'm asking an English-centric model with a Japanese system prompt, and that the evaluation is only one kind of my own workload.

Next, I'd like to add requests without the section and phrasing close to production to the training data and check whether the retrained head's high safe-side count gets fixed.

References

• Contrastive-LM/CLM — Code, training procedure, API reference
• Contrastive-LM/CLM-v0.1-8B — Model card and limitations
• contrastive-lm (PyPI)
• Jev 1.13 (OpenRouter)
• NVIDIA-NeMo/Switchyard