"I Spoke at DevelopersIO 2026 Tokyo with the Title 'You Don't Need an LLM for That Judgment'" #devio2026

Hello, I'm Keima.

I gave a presentation at DevelopersIO 2026 Tokyo, held on October 24, 2026.

In this article, I'll summarize the content I talked about on the day, following along with the slides.

I'd be happy if you read this whether you couldn't make it to the event, or as a review if you were there in person.

1. Session Overview

That judgment doesn't need to be LLM ― Designing "when in doubt, ask a human" with Decision Model
Keima (Keima Kawada) (AI Business Division / Engineer) | 20 minutes

LLMs are excellent, but the high usage cost has become a challenge for many companies. Moreover, when you ask an LLM "how confident are you in that answer?", it will give you a number, but that confidence score is completely unreliable. Against this backdrop, TypeSafe's announcement of Jev on September 15, 2026 sparked widespread discussion about utilizing "Decision Models" that return type-safe decisions with probabilities. Early use cases have been notably seen in browser automation and developer-facing agent validation, but what's needed in real business settings is application to tasks where misclassification must be avoided as much as possible. This session covers threshold design that escalates to humans only when uncertain, and discusses a practical approach to incorporating Decision Models into real-world business tasks like document judgment to improve accuracy while reducing costs.

Source: DevelopersIO 2026 Tokyo <AIxCloud Technology Festival> #devio2026 - connpass

2. Presentation Materials

https://speakerdeck.com/kawadakeima/sono-hantei-llm-ja-naku-te-ii-developersio-2026

This presentation is aimed at those who call a large LLM for every classification or judgment, or those who are unsure how much to trust LLM answers.
I listed three key takeaways:

• The difference between Decision Models and regular LLMs

• A design that routes "when in doubt" to humans using probabilities

• The procedure for determining thresholds using your own data

3. The Problem: Reviewers Become the Bottleneck

In document processing, it's common practice to divide roles among programs, AI, and humans based on the nature of the task.
Parts that can be handled by fixed procedures go to programs, parts where some errors are acceptable go to AI, and parts where mistakes are absolutely not allowed go to humans (Human in the Loop).
And among these, human checking takes the most time.

AI has become fast and cheap.
Even so, when it comes to pressing approval on a coding agent, answering "what should we do here?", or checking AI results, we're ultimately still waiting on humans.
To route only uncertain cases to humans, we need "probabilities" in AI answers.

4. What Is a Decision Model

4.1 A Model That Returns a Probability of Yes

A Decision Model is a model that answers questions with Yes/No and returns the "probability of Yes" (0 to 1).
Unlike general LLMs, it does not generate text.

In code, this can be written in roughly this amount.
Here is an example of asking TypeSafe's Jev whether two values are the same.

import os, requests
KEY = os.environ["TYPESAFE_API_KEY"]

def ask(a, b):  # Returns the probability that 2 values are "the same"
    res = requests.post(
        "https://api.typesafe.ai/v1/systemone",
        headers={"Authorization": f"Bearer {KEY}"},
        json={"model": "jev-1.14.0",                  # Pin the version
              "state": {"value_a": a, "value_b": b},  # The 2 values to compare
              "questions": {"same": {"type": "noul",  # Answer with probability of Yes
                  "instructions": "Is value_a the same value as value_b?"}}})
    return res.json()["answers"]["same"]["noul"]

print(ask("2026/09/12", "2026年9月12日"))  # → 0.98 (probability they are the same value)

4.2 Fall 2026: Multiple Companies Released Decision Models

Triggered by TypeSafe's Jev, various companies released Decision Models one after another.

4.3 Features and Capabilities

The output cost is $0, and it is characterized by being fast and cheap.
The answer types it can return include three options: Yes/No (noul), choice (choice), and scale (score).

On the day, I also introduced which services the four models (Jev, Decisions API, Clef, and Decision-1) are available from and which regions they are processed in.
There is no officially documented route that processes data only within Japan.
Please refer to slide 13 of the Speaker Deck for details.

5. Concrete Example: Asking Jev Whether Two Values Have the Same Meaning

5.1 Looking at the Probabilities

I prepared 107 pairs of values extracted from two documents and asked Jev whether they were "the same value even if written differently."

Differences in date formatting and upper/lower case in company names are clearly answered as "same," while one-day differences or differences in units are clearly answered as "different."
On the other hand, a one-character misread (the orange row) scored 0.76, leaning toward "same" but uncertain.

5.2 Sorting by Threshold

If the probability is 0.8 or above, it is "same"; if 0.2 or below, it is "different"; and only the uncertain ones in between are judged by a human.

Only 14 cases need to be judged by a human.
The 27 cases determined to be "different" prompt a human to go and correct the documents.

6. What Happens When You Have a Regular LLM Do the Same Thing

6.1 Cost and Speed

I also had Gemini 3.8 Flash solve the same 107 cases.
Jev's cost was about 1/19th, and response time was less than half.

Correct answers were 93 and 94, so for simple judgments, they are about equally capable.
Looking at this table alone, the difference appears to be only in "cost" and "speed."

6.2 How Probabilities Are Distributed

The real difference lies in how probabilities are distributed.
When asked for "the probability that the values are the same," Gemini only answers with 0 or 1.
In contrast, Jev returns intermediate values like 0.3 or 0.7.

If only 0 or 1 is returned, there is no way to place a threshold for routing "when in doubt" to humans.

7. How to Determine Thresholds

7.1 Probabilities Are Reasonably Reliable

Of the 81 cases that answered clearly (0.9 or above or 0.1 or below), 80 were correct.
Of the 14 uncertain cases (0.2 to 0.8), only 7 were correct.

7.2 Raising the Upper Threshold Trades Off Between Human Review and Slip-throughs

The higher the threshold, the fewer mistakes, but the more human reviews increase.
At 0.8, there were 14 reviews and 3 mistakes; at 0.9, there were 19 reviews and 1 mistake.

7.3 Probabilities Shift When the Version Changes

When I re-queried with the same version four weeks later, the probabilities barely changed.
However, when upgrading Jev from 1.13 to 1.14, the routing of 6 cases changed.

7.4 How to Determine Thresholds

Determine thresholds using your own data, and re-measure whenever you upgrade the version.

1. Collect data: Prepare pairs of values with correct labels (107 cases in this case)

2. Pin the version: Write the version in the model name like jev-1.14.0, and avoid specifications like jev-latest

3. Count by threshold: For each upper threshold value, count the number of human reviews and slip-throughs

4. Choose acceptable range: The less mistakes are tolerated in the task, the higher the threshold (if up to 1 slip-through is acceptable, use 0.9)

8. Summary

Here is a summary of the session:

• LLMs return answers. Decision Models return "the probability of Yes"

• Could probabilities be used to distinguish between "leave it to AI" and "have a human check"?

• Determine thresholds using your own data. Raise the threshold if mistakes cannot be tolerated

• Re-measure whenever you change the model version

The final message I wanted to convey was "stop reviewing all AI results, and route only the uncertain ones to humans."

The slide appendix includes comparisons by product and impressions from trying them out.
Details of the validation are summarized in my list of DevelopersIO articles.