
I read OpenJev and thought of an alternative solution using non-generative AI
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
Jev, announced by TypeSafe AI on September 15, 2026, has caught the attention of many people. It doesn't do chat or text generation. It's a model provided in a callable form from programs that does only these three things: select from a set of options provided in advance, score according to criteria, and return the probability of yes or no.
Within about 3 days of the announcement, several implementations calling themselves OpenJev appeared on GitHub and Hugging Face. Even though they share the same name, reading through them reveals completely different contents. Some simply change how an existing model is used without adding anything, while others retrained a model specifically for judgment. I wanted to examine this difference more carefully.
Recently, I compared two types of judges for LLM routers: a judge that generates decisions as text, and NeMo Switchyard's prefill router, which determines the routing destination from the internal state at the time the request is read. At that time, I wrote that all the weaknesses of the judge stem from a single point: making it write text (this was an article from August 22, 2026). It turns out I had already been observing the idea of stopping the writing in the router world.
In this article, I'll keep the introduction to Jev brief, and instead cover what OpenJev reproduces from Jev and what it doesn't, based on the published code, numbers, and some light Japanese-language measurements on the DGX Spark. I hope this resonates with people who want to carve out the judgment portion of agents or routers into something small.
Reviewing the Question Jev Raised
Jev is the first model of the kind TypeSafe AI calls a System One Model. Its usage differs slightly from a normal LLM: you pass text that serves as judgment material (state), a question, and either options or scoring criteria, and it returns answers and probabilities as typed values. There are three types of questions:
| Type | What it returns | Example use case |
|---|---|---|
| Choice | Probability per candidate, the selected candidate, and confidence | Which department handles this inquiry: technical, billing, or sales |
| Score | Scoring along verbally defined ordered levels | Is this defect minor, moderate, or severe |
| Noul | Probability of yes from 0 to 1 | Is this message requesting a refund |
The price is $0.042 per 1 million input tokens (the unit models use to process text), with output being free. The current jev-1.13.0 accepts text only, and there is no per-customer fine-tuning. The official response time is 70–500 milliseconds.
The training method is called RLCD (Reinforcement Learning for Calibrated Decisions). The goal, in addition to selecting the correct candidate, is to bring the returned probabilities close to actual frequencies. It might be easier to understand with a weather forecast analogy. If you collect all the days when the rain probability was stated as 80%, about 80% of those days actually had rain. That kind of forecast lets you trust the numbers directly when deciding whether to carry an umbrella. This is called probability calibration.
There are two points that are easy to confuse here. One is confidence. This is a value representing how concentrated the probability distribution is on a single candidate, not the probability that the answer is correct. It's the kind of number that becomes low when the distribution is split, like 0.34, 0.33, and 0.33 across three options.
The other is how to read the zero-hallucination selling point. What Jev guarantees is that the returned values never deviate from the specified type. If you set the options to keep and delete, a third operation will never be returned. However, that doesn't mean the possibility of choosing delete for something that should be kept disappears. The type is deterministic, but the judgment itself is probabilistic. The official constraints page is also honest: it explicitly states that Jev is not a calculator, that it struggles with enumeration and date comparisons, that it can be swayed by instructions mixed into the state, and that threshold values tuned for Noul cannot be carried over directly to Choice.
One note for Japanese usage: the official model list states that while CJK including Japanese is supported, accuracy drops compared to English, and that you should verify with your own data before adoption.
That covers the overview of Jev. What I want to follow in this article is the question Jev raised: if all a program needs is whether the answer is A or B, is there really a need to have an LLM write out text or JSON?
Different Implementations Are Published Under the Name OpenJev
The first thing to establish is that OpenJev is not the official open-source version of Jev. TypeSafe has not released the model weights or details of the training method. OpenJev is closer to a collective term for independent projects trying out Jev's ideas and input/output format using available open models. Here are the main implementations I could confirm as of September 18, 2026.
| Implementation | Approach | Base model and runtime | Additional training |
|---|---|---|---|
| TheoLeeCJ/openjev | Reads scores of tokens corresponding to candidates directly. Reuses a shared state | Qwen3.5-4B, RTX 3090. Browser version also available | None |
| AlexWortega/openjev | Retrains as a model that classifies the relationship between premise and hypothesis into three classes | Qwen3.5-4B | Yes |
| ekzhang/openjev-sglang | API server with the same form as Jev. Reads only 1 token worth of data per question | Qwen3.6-35B-A3B in NVFP4, SGLang, B200 | None |
| daseinlabs/open-jev | Scores the plausibility of candidate strings collectively. Includes Jev-compatible API | Gemma 3 4B, Apple Silicon MLX | None |
| rorshopping/jev-on-a-laptop | Reads context once, reads scores per item directly, and assembles JSON | Qwen2.5 1.5B and 7B, Qwen3-8B | None |
| vinnylarouge/jevlike | Trains a small model to score candidates. Text reading part uses a custom or frozen existing model | Custom small model, or Qwen2.5-0.5B | Yes |
| zhihz/openjev | Reads the next token distribution once per question. English and Chinese interface | Qwen3-4B-Instruct-2507 | None |
The authors who explicitly mention the relationship with Jev are both cautious. The TheoLeeCJ version's README states that what was reproduced is the interface pattern—that is, the input/output types—not the non-public model or training. The zhihz version goes so far as to clarify that this name does not mean an official open-source version of Jev.
Looking at the table, the approaches divide broadly into two camps. Those that change only the readout method without modifying the existing model, and those that train a model specifically for judgment. In this article, I'll look at the TheoLeeCJ version as a representative of the former and the AlexWortega version as a representative of the latter.
First, Running It in a Browser to See Judgment Without Generation
Before getting into the mechanics, let's look at something that actually runs. The TheoLeeCJ version has a demo that runs in the browser alone, with no registration or waitlist required.
You select and load a model, enter the state, question, and options, and run it. This lets you solve the same problem in two different ways with the same model. One is Direct readout, which reads only the probabilities of the options without generating a single character. The other is Generation, which writes out the probability distribution as JSON one token at a time.

The screen for entering state, question, and options. Here I entered a self-made Japanese inquiry. Clicking RUN BOTH METHODS in the top right runs both paths in sequence.
The desktop default is a quantized version of MiniCPM5 2B, with a download size of 1.56 GB. Here are the results from running it with a Japanese inquiry:

Left is the result of reading scores directly, right is the result of having it write JSON. In this run, reading directly took 0.925 seconds, writing took 2.843 seconds, and the site displayed a ratio of 3.07x (September 18, 2026, Chrome 153, single measurement).
The same model, but one answers without writing while the other answers by writing. The question Jev raised is now visible.
Since the inquiry is about not being able to log in, I labeled the correct answer as technical support. The direct readout side assigned 0.734 to billing, and the JSON generation side assigned 0.4 to sales. Both are wrong, and the two paths disagree with each other. This is a single result from running a quantized 2-billion-parameter model on Japanese text, so I have no intention of drawing conclusions from it. The site itself notes that the displayed scores are calculated only from the candidate tokens listed and are not calibrated confidence values, that it makes no claim that any model is comparable to Jev, and that quantization changes both quality and speed. Note that when I submitted the same question to Jev using the method described in a later section, it returned technical support with probability 1 all three times.
The Mechanism for Extracting Judgment from a Generative Model Without Generating
What is the TheoLeeCJ version doing? It hasn't created a new model. It uses Qwen3.5-4B as-is, with no additional training. Only the readout method has changed.
When an LLM writes text, internally it assigns scores to all tokens in its vocabulary at every step. Tokens that are more likely to come next receive higher scores. These scores are called logits. Normally, one token is selected from this score table, appended, then the score table is regenerated, and so on, until text is produced.
Let me use a multiple-choice exam analogy. The approach of having it write JSON is like asking a test-taker to write out the answers to each question in order. The TheoLeeCJ version peeks at the internal inclination—that moment when the test-taker has just finished reading the question and is pondering whether to fill in A, B, or C. For example: A gets 14.2 points, B gets 11.8 points, C gets 7.1 points. Once you know that, there's no need to have the answer written out. These inclinations correspond to what is called the KV cache.
In the implementation, options are mapped to single-character labels like A, B, C, and only the scores for those labels are extracted at the end of the prompt. What's careful here is the verification that a label fits within a single token and that the tokenization boundary doesn't shift at the join with the prompt. The extracted scores are converted to probabilities using the following formula:
Here
There's a second reason for the speed increase. When you want to ask 21 questions about the same state, doing it naively means having it read the same long text 21 times. The TheoLeeCJ version has it read the shared state only once, saves that intermediate result, and processes only the short continuations for each question in parallel. It's like reading a long document once, sticking bookmarks in it, and then answering 21 questions by looking at those bookmarks. The thing serving as those bookmarks is the KV cache.
In the published measurements, with the same model, same state, 21 binary questions, and an RTX 3090, the following results were obtained:
| Path | Time | Output tokens | What is returned |
|---|---|---|---|
| Reading scores directly | 1.023 s | 0 | 21 pairs of probabilities |
| Generating a yes/no JSON array | 5.332 s | 111 | Array of 21 values in sequence |
The difference is 5.21x. What I found honest in the author's writing here is that they themselves wrote that the two paths agreed on 18 out of 21 questions. Since it's a different readout method rather than a faster version of the same computation, the answers change slightly too. It's explicitly stated that this is a comparison of output path costs and not a claim that the meaning is the same. The shared state reuse is similar: when processing questions in batches, 777 judgments shrank from 333.1 seconds to 38.8 seconds, while 6 selected candidates switched.
One more property of this approach worth keeping in mind: the returned probabilities represent relative strength among the provided candidates. If only retry and cancel are provided, whichever action is actually needed—say, gather more information—will still have its probability split between those two. The ability to quantify relative preference among candidates and whether the set of candidates itself is appropriate are separate matters.
Reading Out Japanese Judgments on the DGX Spark
The published numbers were measured on an RTX 3090 with English material. I verified on my DGX Spark unit whether the same pattern holds for Japanese. The model is the same Qwen3.5-4B as in the TheoLeeCJ version, with weights kept at 16-bit BF16. The measurement scripts are also the unmodified ones from the TheoLeeCJ version repository—I'll call this the upstream going forward. The English instruction templates are also unchanged; only the state, questions, and candidate descriptions were made Japanese. flash-linear-attention is not installed, so some layers run on PyTorch's standard implementation. Both paths being compared are under the same conditions.
I created two sets of material. The first is 21 binary questions about a single filling machine shutdown report. Questions include things like whether cooling water was flowing and whether restart approval has been issued, with 4 unrelated background records mixed in. The second is 36 questions that each ask you to select one from four candidates, which corresponds to Jev's Choice type. Half of the 18 questions involve inquiry routing: reading an inquiry and choosing which department handles it—technical support, billing, sales, or not applicable. The other 18 involve the next action for equipment alerts, choosing from restart, parts replacement, wait and see, or insufficient information. Seven of the 36 questions were designed so that the correct answer is that there's insufficient information to decide. The correct labels were assigned by me manually.
The four-option questions look like this, for example:
Record: The previous month's invoice includes a charge for an option that was supposedly canceled. Please check.
Question: Which department should handle this inquiry?
Options: Technical support, Billing, Sales, Not applicable
Correct answer: Billing
One point to note for those who try this on a DGX Spark themselves: the torch==2.10.0 pinned by upstream, when installed from PyPI, becomes a CPU-only version on aarch64, and the GPU is not visible. Reinstalling the same version from PyTorch's CUDA 13.0 index fixes this.
uv pip install --index-url https://download.pytorch.org/whl/cu130 'torch==2.10.0'
First, here are the results comparing direct score readout against JSON generation. The English row uses the upstream material run as-is on the DGX Spark, with medians of 3 runs each.
| Material | State length | Direct readout | JSON generation | Ratio | Agreement between paths | Correct answers (direct readout) |
|---|---|---|---|---|---|---|
| English (21 questions from upstream) | 1,812 tokens | 1.608 s | 6.984 s | 4.34x | 18/21 | — |
| Japanese (21 self-made questions) | 426 tokens | 0.961 s | 6.092 s (invalid at 20 items) | 6.34x | Not determinable | 21/21 |
The 18/21 for English exactly matched the upstream published value. The seconds can't be compared directly since the hardware and kernel differ, but the pattern—direct score readout finishing in a fraction of the generation time—holds on the DGX Spark as well. The higher ratio for Japanese is because the state is shorter, so the time spent reading takes up a smaller proportion, and the weight of generation time increases.
For Japanese, the direct score readout method got all 21 questions correct. With English instruction text framing Japanese records, and a bearing wear story mixed into the background records, the results matched all 21 of my manually assigned correct labels.
What I found more interesting than expected was the generation side. The agreement between paths is marked as not determinable because the generated array had only 20 items in all three runs. I instructed it to return a JSON array of yes/no answers in order for 21 questions, and it came back one question short. The JSON itself was valid, and all 20 items in the array matched the correct sequence. Even so, the receiving side cannot trust which of the 20 values corresponds to which question. The upstream script also marks this output as invalid. It writes, so it miscounts. The direct score readout approach doesn't have this failure mode at all, since it simply reads scores at a fixed location per question.
Next, here are the results for the 36 four-option questions.
| Metric | Value |
|---|---|
| Correct answers (original order) | 33/36 |
| Correct answers (options in reverse order) | 35/36 |
| Questions where selection changed with reversed order | 2/36 |
| Of the 7 questions where "insufficient information" is correct, those scoring another candidate above 0.8 | 0 |
| Average probability of selected candidate | 0.916 |
Getting 33 out of 36 correct with a 4-billion-parameter model and no training is not bad. For inquiries that only say something like "Best regards" or alerts that only say "Error F-02," it correctly selected not applicable or insufficient information.
The 3 wrong answers revealed the character of this approach. Two of them selected insufficient information rather than the correct answer, with low probabilities of 0.472 and 0.587, and both switched to the correct answer when the option order was reversed. Questions with low probability are susceptible to meaningless changes like order. Conversely, this suggests that a design of routing low-probability questions to humans rather than automated processing could be effective.
The remaining one question routed an inquiry about where to change a credit card expiration date in the settings screen to technical support with a probability of 0.900. My label was billing. It was probably pulled by the phrasing "where in the settings screen." The answer doesn't change even with reversed order. This isn't necessarily a question where my label is the only correct answer, but for a company with separate departments, it's an example that consistently and confidently falls on the unexpected side. The point that high probability and being operationally correct are different things surfaced even in this small 36-question set.
I also looked at the effect of shared state reuse in a separate measurement. The 0.961 seconds in the first table was the median of 3 runs after warming up the shared-state path with one dry run. For a cold measurement with one run each: asking 21 questions individually took a total of 4.159 seconds, while reading the state once and batching 21 questions took 1.642 seconds. No questions switched their selected candidate (0/21).
Submitting the Same Material to Jev
Having come this far, I was curious what would happen if I submitted the same material to the original. Since Jev became available via OpenRouter on September 18, 2026, I submitted the same material directly.
The chat completions endpoint rejected the request and directed me to a dedicated decisions endpoint. The submission format was the same state and questions as the TypeSafe API. The candidate keys, to keep conditions aligned with Qwen3.5-4B, were assigned the characters A through D based on order.
| Material | Jev (via OpenRouter) | Qwen3.5-4B (direct score readout) |
|---|---|---|
| 21 questions on equipment shutdown report | 21/21 | 21/21 |
| 36 four-option questions, original order | 34/36 | 33/36 |
| 36 four-option questions, reversed order | 34/36 | 35/36 |
| Questions where selection changed with reversed order | 0/36 | 2/36 |
| Of questions answered incorrectly in original order, those with probability ≥ 0.7 | 0 | 1 |
The correct answer counts were nearly tied. The official documentation states that accuracy drops for CJK including Japanese, but no breakdown was seen in this small set of material. The differences showed up in stability and how it failed. Jev didn't switch a single answer when the candidates were reversed.
Let me also look at the 2 wrong answers. One was the same credit card expiration date inquiry that Qwen3.5-4B got wrong—Jev also routed it to technical support. However, the probability was 0.66, much more modest than Qwen3.5-4B's 0.900. The other was an inquiry about wanting to switch to a higher plan and use SSO, which was routed to technical support with a probability of 0.47. In the original order, both of Jev's 2 wrong answers had probabilities below 0.7, while only 1 correct answer had a probability below 0.7. However, this is just an observation of scores on incorrect answers. Whether the probabilities correspond to actual correct-answer frequencies—that is, calibration—was not measured on this material.
Moreover, reversing the order raises the probability on the card inquiry to 0.76. You can't resolve this by drawing a single threshold line from 36 questions. The fact that both models got the same question wrong on the same side is also material for doubting my own labels.
Let me also include the speed. When batching 21 questions as 21 Noul calls in a single request, the round trip from my machine to OpenRouter was 0.245–0.327 seconds. Since this includes network and relay time, it measures something different from the DGX Spark's 0.961 seconds. The Jev seconds are via OpenRouter and not from calling the TypeSafe API directly. The total cost across three sets of material and repetitions was approximately $0.002.
Since this is a small set of material, I'm not claiming this works as a Japanese substitute for Jev. What I found is that the direct score readout approach works with Japanese records, and that the properties shown in the published values—namely the speed, the disagreement with generation, and the instability of low-probability questions—appeared in the same form on my own machine.
An Alternative: Retraining for Judgment
The AlexWortega version, despite sharing the OpenJev name, takes a different direction. It retrains Qwen3.5-4B into a model specifically for judgment.
The framework used is NLI (Natural Language Inference). Two sentences—a premise and a hypothesis—are fed in, and the model classifies the relationship into three categories: the premise supports the hypothesis (entailment), contradicts it (contradiction), or the premise alone isn't enough to determine (neutral). Think of it as a three-way problem: given this premise, can this hypothesis be concluded?
Premise: The monitoring log records a connection timeout to an external API.
Hypothesis: A problem occurred in communication with the external API.
→ entailment (supports)
The idea behind the AlexWortega version is to use this three-way classification as a universal judgment engine. If you want to rank candidates for an answer, create a hypothesis for each candidate and select the one with the highest probability of being supported. If you want to score an answer, put the correct answer in the premise and see whether the answer is supported. The model card even includes examples of playing Doom or Flappy Bird in real time.
The training details can be confirmed from the published code. It uses 120,000 examples from the English NLI datasets SNLI and MNLI, trains for 1 epoch, and is a full fine-tune that updates the entire model body, with a loss of standard three-class cross-entropy. The word zero-shot in the model card is best read as meaning that this trained NLI model is applied to downstream tasks like Doom without additional training. It doesn't mean the model was trained on nothing. Nor is it a reproduction of Jev's RLCD. The training is for getting the correct label right, not for calibrating probabilities.
The meaning of the returned probabilities also differs from the TheoLeeCJ version. The TheoLeeCJ version's probabilities are the scores assigned to candidate labels, redistributed so that they sum to 1 across all provided candidates. Even if A gets a high value, it doesn't guarantee that A is the most operationally appropriate choice. The AlexWortega version outputs, independently for each candidate, whether the candidate is supported, contradicted, or undeterminable.
The AlexWortega version's values are independent per candidate and don't share a sum of 1 across candidates. This means multiple candidates can be simultaneously supported, and none might be supported at all. On the other hand, a statement being supported doesn't mean it's the best next action to take.
There's a number in the published evaluation that captures this property well. It's two measurements using GPQA-diamond, a collection of graduate-level four-option questions.
| Method | What is given to the model | Accuracy |
|---|---|---|
| Have it select the correct answer from the candidates | Question and 4 candidates | 27.3% |
| Show the correct answer, then have it score whether each candidate is correct | Question, correct answer, and 1 candidate | 96.8% |
The 27.3% when selecting without the correct answer is near random-chance level for four-option questions. The scoring side with the correct answer provided was 96.8%. The former is counted per question and the latter per candidate, so they're not on the same scale. The author writes about these two separately in the report. The ability to derive an answer and the ability to evaluate candidates against provided evidence or a correct answer are different things, and what this model possesses is the latter.
I took this as a designation of where to use it rather than a weakness. It seems well-suited for judgments where the material is already on hand and you're just cross-referencing—like checking whether a retrieved RAG passage and a response contradict each other, or verifying whether an agent's output meets a specification. However, the fact that it scored well on GPQA evaluation is separate from whether it works for RAG evidence verification, so you'd need to measure accuracy for that use case yourself. Whether what you need from a judgment model is deriving answers or cross-referencing: settling that before adoption will change which implementation you choose.
What Is the Same as Jev, and What Is Different?
With the above in mind, let me compare with the original. Rather than collapsing this into a single word of whether it was reproduced or not, I'll break it down by input/output, inference method, training, calibration, and judgment quality.
| Comparison item | Jev | TheoLeeCJ version | AlexWortega version |
|---|---|---|---|
| Input/output form | Pass question and candidates at runtime, receive typed probabilities | Almost the same form | Converted into premise-hypothesis form |
| Non-generative inference | Dedicated architecture and parallel sampler | Reads scores directly from existing model | Reads output of classification head |
| Training | Non-public model with RLCD | No additional training | Supervised NLI training |
| Probability calibration | Listed as a training objective | Explicitly states not calibrated | No discussion of calibration |
| Judgment quality | 88.3% agreement with reference on 102 judgments from public data | 84.5% on the same 102 judgments | High score on GPQA with correct answer provided for scoring |
The 84.5% and 88.3% in the bottom row are numbers that are tempting to headline. It looks like a 4-billion-parameter open model is within 3.8 percentage points of the original. However, I want to read this carefully. The comparison is only on 102 judgments that could be matched out of 711 judgments in TypeSafe's public data. The Jev value was read from a published record, not from the author calling the API directly. The author themselves writes that this does not demonstrate capability close to Jev's. On a separate metric measuring how far the probability distribution deviates from the reference, the TheoLeeCJ version scored 0.177 versus Jev's 0.127. Since smaller means closer to the reference, there is still a gap.
Furthermore, the correct answers in this evaluation were not confirmed by humans. The reference labels in TypeSafe's public evaluation were created from the average of responses from GPT-6 Astra and Claude Fable 5.1, and the official documentation itself notes that it measures against the most intelligent large models currently available, assuming the code is correct. In other words, the agreement rate here is not accuracy against real-world correct answers but rather how closely it resembles the judgments of large models.
Robustness data is also published. In the TheoLeeCJ version, simply reversing the order of options without changing meaning caused 10 out of 36 judgments to change. Cases were also reported where it answered with a score above 0.8 on questions that should have had insufficient information. That's why these scores cannot be treated as operationally calibrated values like Jev's, according to the author's conclusion.
Does that mean the original can be trusted to take probabilities at face value? That's not straightforward either. As mentioned earlier, the official constraints page acknowledges that Noul thresholds cannot be carried over to Choice and that probabilities from separately asked questions don't necessarily sum to 1. The official documentation also explains that thresholds are not a single number but should be varied per operation based on the severity of being wrong.
To summarize: the original claims calibrated judgment as a product capability. In the OpenJev materials I read, I could not confirm operational calibration equivalent to that. In exchange, you can examine and fix the internals yourself. And regardless of which you choose, you can't skip evaluating with your own business data. My takeaway from reading through all of this is that it doesn't reduce to a binary of the original being trustworthy and OpenJev being untrustworthy.
Among articles written about Jev, there was an assessment that by reading just 1 token per question, you can get most of Jev's speed, consistency, and parallelism—suggesting the technical moat may not be very deep.
OpenJev's published measurements showed, within 3 days of the announcement, that judgment can be extracted in a fraction of the time needed for JSON generation. Whether the same speed or throughput as Jev was reproduced has not been measured under matched conditions. If there is a moat, I think it's more on the side of how far the probabilities can be trusted than on the speed side.
Where to Put the Power to Judge
Looking at all three side by side, they start to look less like competitors and more like different answers to the same problem: where in the system to place the power to judge.

The lower you go, the more purpose-built for judgment it becomes. The TheoLeeCJ version corresponds to layer 2, AlexWortega version and prefill router to layer 3, Jev to layer 4. As of September 18, 2026, no open reproduction of layer 4 was confirmed.
Layer 1 is having a general-purpose LLM write JSON and reading that—the most common approach today. Layer 2 is the TheoLeeCJ version: keep the model as-is, only change the readout method. Layer 3 is the layer where something is trained for judgment: this includes approaches like the AlexWortega version that retrain the entire model, as well as approaches that freeze the model and train only a small head. Layer 4 is Jev, where calibration of probabilities is included as a training objective. I couldn't find any implementation that credibly claims to have openly reproduced RLCD within my search.
The prefill router mentioned at the start belongs to layer 3 of this map. It's a router that uses the internal state of an LLM after reading a request as features, and uses a small neural network to output the probability that a given model can handle the request. When I built one and compared it with a generative judge in August, it achieved accuracy comparable to the judge on requests similar to those in the training data. However, when applied to 47 standalone requests flowing through day-to-day, the ability to rank by difficulty remained, but the probabilities shrank into a narrow band from 0.4 to 0.6, and with the threshold I had set at 0.80, all questions were routed to the higher-performance model. I took this to mean that even if you get good accuracy on requests similar to the training data, you can't take the same threshold to a different type of request. I had hit the same wall a month earlier that the OpenJev authors keep disclaiming about not being calibrated.
I think the way to use this map is to try from the top down. First, at layer 2, measure how far judgment can be extracted without any additional training. Only advance to layer 3 and train where accuracy or stability falls short. When you want to use the probabilities themselves as thresholds for automated processing, that's when to consider for the first time whether you need layer 4 capability—meaning whether to use Jev, or take on calibration work yourself. There's no need to train everything into a model, nor to stuff everything into a prompt.
For a concrete example of what cutting out judgment looks like in practice, TypeSafe's own example is useful. It's the scene of selecting which of Hermes agent's 182 skills is needed for the current request. The first call ranks all skills while simultaneously determining whether the request requires a skill at all, and the second call re-reads the detailed descriptions of the top 3 to narrow it down to one. A "not applicable" answer can also be returned. The published example compares a Claude Haiku 4.5 agent with and without Jev 1.12 skill suggestion across 488 requests. The rate of loading the wrong skill dropped from 16.8% to 7.3%, and unnecessary loading dropped from 9.8% to 4.0%. What I found likeable was the note that there were also cases where things that had been working were broken.
On the Hermes Agent side as well, in the Issue considering Jev integration, a line was drawn: don't make Jev the chat model for sessions, limit it to advisory judgment. The judgment model is treated not as something holding final authority, but as a component used while the code retains control over conditions, permissions, and thresholds.
For a while now, it's felt like the development of models has only been heading in the direction of larger and more complex. What Jev and OpenJev showed us was the opposite perspective: AI as a small component whose output type is fixed and can be plugged directly into program if-statements. This pairs well with the idea of placing only the needed capability where it's needed, and there may be more judgments than we think that don't require a massive generative model to write JSON every single time.
Summary
OpenJev was not a copy of Jev. In response to Jev's question of whether generation is necessary if you only want judgments, the TheoLeeCJ version answered that you just need to read the score tables of existing models, while the AlexWortega version answered that you should retrain for the purpose of judgment. The name is the same, but the approaches differ. That was the most interesting part to read.
However, just because you can extract an answer without having it write anything does not mean that answer can be used directly in automated processing. Qwen3.5-4B misclassified with a high probability of 0.900, and the official Jev documentation also states that the threshold is not a single number but should be determined per operation based on the impact of mistakes. What options to list, how much to trust the returned probabilities, and how well it performs on your own business data — these three questions remain the user's responsibility regardless of which OpenJev you choose or whether you use the original. What I found valuable about OpenJev is that both the code and the evaluation methods are publicly available, allowing you to verify these things with your own hands.
Reference Links
- Introducing System One Models & Jev - TypeSafe AI Blog (announced 2026-09-15)
- System One - TypeSafe AI Docs
- Models - TypeSafe AI Docs — notes on versions, pricing, and languages
- Confidence - TypeSafe AI Docs — definition of confidence and approach to thresholds
- Jev 1.13 jaggedness - TypeSafe AI Docs — official list of constraints
- TypeSafe Workflow Evals — public evaluations and how reference labels are created
- Skill suggestion - TypeSafe AI Cookbook
- Jev 1.13 - OpenRouter (listed 2026-09-18)
- TheoLeeCJ/openjev — implementation that reads scores directly.
docs/RESULTS.mdcontains a list of what could and could not be reproduced - OpenJev in your browser
- AlexWortega/openjev - Hugging Face — implementation retrained on NLI
- ekzhang/openjev-sglang
- daseinlabs/open-jev
- rorshopping/jev-on-a-laptop
- vinnylarouge/jevlike
- zhihz/openjev
- Jev means structured output is interesting again - Sean Goedecke (2026-09-16)
- Implement Jev (TypeSafe) in Hermes - NousResearch/hermes-agent Issue #113837
- LLM Router: Rethinking Routing with Prefill Activations — paper on prefill router

