[TypeSafe] Can LLMs Get Faster by Limiting Output? Comparing Speed and Cost of Gemini 3.8 Flash and Jev on the Same 107 Cases

Hello, I'm Keema.

The other day, I wrote an article about trying cross-matching of document fields with Jev in TypeSafe.

【TypeSafe】How far can Jev go with document field cross-matching? Measuring accuracy, speed, and cost | DevelopersIO

Jev is a model that doesn't generate text and only returns probability values in response to questions.
After seeing the results of throwing 107 cases with a median of about 0.7 seconds and a total of about 0.35 yen, I suddenly thought:
"Isn't it fast and cheap simply because it doesn't generate text?"
If that's the case, even a regular generative LLM should run at roughly the same speed and cost if the output is limited to a few tokens.

So this time, I had Gemini 3.8 Flash solve the exact same 107 cases as before, and compared accuracy, speed, and cost (the verification date is September 20, 2026).

To put the conclusion first, my prediction was completely wrong.
I hope this serves as a reference for those thinking "if it only returns probabilities, a regular LLM should be sufficient."

1. Is "it's fast because it doesn't generate" really true?

The response time of a generative LLM is largely determined by two factors.
The time until the first token is returned after reading the input (TTFT), and the time to generate the output one token at a time from there.
When you have it write long text, the latter time accumulates.

In the previous verification, Jev's output was only 36 tokens per case.
In that case, if Gemini is made to return only a dozen or so tokens like {"exact": 0.95, "semantic": 0.2}, the time spent on output generation should be nearly zero.
The remaining factor is just "the time to read the input," so the hypothesis this time is that it would be a close match with Jev.

The reason I chose Gemini 3.8 Flash as the comparison target is that it was cited as the fastest by Artificial Analysis.
https://artificialanalysis.ai/

2. How conditions were aligned

2.1 What was aligned with the previous time

To allow a fair comparison, the following conditions were aligned with the previous article.

The contents of the test data and the concept behind the two judgment axes are all included in the previous article.
For the Jev side numbers, the results from the English version of the previous article are used as-is.

2.2 What was devised on the Gemini side

To limit the output, structured output was used for Gemini (a feature that, when you specify the JSON schema you want returned, outputs only in that format).
This fixes the output to just the two items exact and semantic.

Probabilities are output as numerical values from 0 to 1 by the model itself. The probabilities that Gemini returns this time are "self-reported values output by the model as text."

Another point to note is the handling of thinking (internal reasoning before answering).
Even if the output is kept short, speed won't be achieved if the model thinks deeply behind the scenes.
However, according to the official documentation, the thinking levels accepted by Gemini 3.8 Flash are only three stages—low, medium, and high—and specifying the minimum value of minimal results in an error.

Supported (low, medium, high)
Note: minimal is not supported and returns an error.

Source: Model information: Gemini 3.8 Flash | Google AI for Developers

In other words, thinking cannot be completely turned off.
This time, it was set to low, the minimum that can be specified, and the actual number of tokens thought behind the scenes was recorded.

2.3 Environment

Note that the previous Jev re-established a connection for each request, but the Gemini side was measured with a setting that reuses HTTP connections (--keepalive).
This creates measurement conditions that are advantageous to the Gemini side, as TLS handshakes per connection can be skipped.

3. Script and execution

3.1 File placement

Add run_eval_gemini.py for this time to the directory containing run_eval.py and cases.jsonl from the previous article.
Since the question text is imported from the previous script, please make sure to place it in the same directory.

<working directory>/
├── cases.jsonl          # Test data from the previous article
├── run_eval.py          # Script from the previous article
└── run_eval_gemini.py   # Script added this time

3.2 Script

As before, it runs on Python's standard library alone without using any third-party libraries.

run_eval_gemini.py:

#!/usr/bin/env python3
"""Measure the same cross-matching cases as JEV using a generative LLM (Gemini).

What we want to verify is the hypothesis: "JEV is fast and cheap because it doesn't generate text,
so if we limit the output of a generative LLM to the bare minimum, it should achieve the same speed and cost."

The following steps are taken to align conditions:
- Cases are read directly from JEV's cases.jsonl (not copied)
- Question text is also imported from JEV's run_eval.py QUESTIONS to use the same wording
- Latency is measured with sequential execution like the JEV side. Connections are new per request by default, reused with --keepalive
- Output is limited to just 2 items using structured output (forced JSON schema)

API: POST https://generativelanguage.googleapis.com/v1beta/models/<model>:generateContent
     https://ai.google.dev/api/generate-content

Runs on standard library only.
"""

from __future__ import annotations

import argparse
import http.client
import json
import os
import statistics
import sys
import threading
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

# Find run_eval.py and cases.jsonl from the JEV side. Use them if they're in the same directory,
# otherwise use the neighboring jev-cross-match-validation/.
HERE = Path(__file__).resolve().parent
JEV_DIR = HERE if (HERE / "run_eval.py").exists() else HERE.parent / "jev-cross-match-validation"
sys.path.insert(0, str(JEV_DIR))
from run_eval import QUESTIONS, STATE_KEYS  # noqa: E402  Use the same question text as JEV

HOST = "generativelanguage.googleapis.com"
ENDPOINT_PATH = "/v1beta/models/{model}:generateContent"
ENDPOINT = f"https://{HOST}{ENDPOINT_PATH}"
DEFAULT_MODEL = "gemini-3.8-flash"

# USD / 1M tokens (Standard, published pricing as of 2026-09). Output unit price also applies to thinking tokens.
# https://ai.google.dev/gemini-api/docs/pricing
PRICES = {
    "gemini-3.8-flash": (0.75, 3.75),       # From 2027-01-01: 1.50 / 7.50
    "gemini-3.5-flash-lite": (0.30, 2.50),
}

# Minimum thinking level that can be specified per model.
# gemini-3.8-flash doesn't support minimal (400 error), so thinking cannot be completely turned off.
MIN_THINKING = {
    "gemini-3.8-flash": "low",
    "gemini-3.5-flash-lite": "minimal",
    # Gemma 4's thinking is binary on/off, and minimal in the API corresponds to off.
    "gemma-4-26b-a4b-it": "minimal",
    "gemma-4-31b-it": "minimal",
}

RETRY_STATUS = {429, 500, 503}

def build_prompt(case: dict, lang: str) -> tuple[str, str]:
    """Convert the same information as JEV's questions/state into two strings: system and user."""
    src = QUESTIONS[lang]
    key_a, key_b = STATE_KEYS[lang]
    lines = []
    for key in ("exact", "semantic"):
        q = src[key]
        lines.append(f"[{key}] {q['instructions']}")
        lines.append(f"  true: {q['criteria']['true']}")
        lines.append(f"  false: {q['criteria']['false']}")
    state = json.dumps({key_a: case["a"], key_b: case["b"]}, ensure_ascii=False)
    return "\n".join(lines), state

def build_schema(mode: str) -> dict:
    """bool returns only true/false; prob returns a probability from 0 to 1, aligned with JEV's noul."""
    if mode == "bool":
        prop = {"type": "BOOLEAN"}
    else:
        prop = {"type": "NUMBER", "minimum": 0, "maximum": 1}
    return {
        "type": "OBJECT",
        "properties": {"exact": prop, "semantic": prop},
        "required": ["exact", "semantic"],
        "propertyOrdering": ["exact", "semantic"],
    }

_local = threading.local()

def post_fresh(body: bytes, api_key: str, args) -> tuple[int, str]:
    """Same measurement approach as the JEV side. TLS handshake occurs every time since a new connection is made per request."""
    req = urllib.request.Request(
        ENDPOINT.format(model=args.model), data=body, method="POST",
        headers={"x-goog-api-key": api_key, "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=args.timeout) as resp:
            return resp.status, resp.read().decode("utf-8")
    except urllib.error.HTTPError as exc:
        return exc.code, exc.read().decode("utf-8", errors="replace")

def post_keepalive(body: bytes, api_key: str, args) -> tuple[int, str]:
    """Reuses connections. This is closer to actual production servers, and faster by the handshake amount."""
    for retry in (False, True):
        conn = getattr(_local, "conn", None)
        if conn is None:
            conn = _local.conn = http.client.HTTPSConnection(HOST, timeout=args.timeout)
        try:
            conn.request("POST", ENDPOINT_PATH.format(model=args.model), body=body,
                         headers={"x-goog-api-key": api_key,
                                  "Content-Type": "application/json"})
            resp = conn.getresponse()
            return resp.status, resp.read().decode("utf-8", errors="replace")
        except (http.client.HTTPException, ConnectionError):
            # If the server has closed an idle connection, reconnect only once
            conn.close()
            _local.conn = None
            if retry:
                raise
    raise AssertionError("unreachable")

def call_gemini(case: dict, api_key: str, args) -> dict:
    """Send one case and return the response and latency. Latency is only for successful attempts."""
    system, state = build_prompt(case, args.lang)
    if args.mode == "prob":
        system += ("\nAnswer each question with the probability (0 to 1) that it is true."
                   if args.lang == "en" else
                   "\nAnswer each question with a probability from 0 to 1 of it being true.")

    body = json.dumps({
        "systemInstruction": {"parts": [{"text": system}]},
        "contents": [{"role": "user", "parts": [{"text": state}]}],
        "generationConfig": {
            "responseMimeType": "application/json",
            "responseSchema": build_schema(args.mode),
            "thinkingConfig": {"thinkingLevel": args.thinking_level},
        },
    }, ensure_ascii=False).encode("utf-8")

    for attempt in range(4):
        started = time.perf_counter()
        try:
            status, raw = (post_keepalive if args.keepalive else post_fresh)(body, api_key, args)
        except Exception as exc:  # Network disconnection / timeout
            return {"error": f"{type(exc).__name__}: {exc}",
                    "latency_ms": (time.perf_counter() - started) * 1000}
        latency_ms = (time.perf_counter() - started) * 1000
        if status == 200:
            return {"payload": json.loads(raw), "retries": attempt,
                    "latency_ms": latency_ms}
        if status in RETRY_STATUS and attempt < 3:
            time.sleep(2 ** attempt)
            continue
        return {"error": f"HTTP {status}: {raw[:500]}", "latency_ms": latency_ms}
    raise AssertionError("unreachable")

def judge(case: dict, result: dict, args) -> dict:
    """Cross-check the response against expected values. Row format is aligned with JEV's results_*.jsonl."""
    row = {
        "id": case["id"],
        "category": case["category"],
        "note": case["note"],
        "latency_ms": round(result["latency_ms"], 1),
    }
    if "error" in result:
        row["error"] = result["error"]
        return row

    payload = result["payload"]
    usage = payload.get("usageMetadata") or {}
    row["model"] = payload.get("modelVersion") or args.model
    row["retries"] = result["retries"]
    row["input_tokens"] = usage.get("promptTokenCount")
    row["output_tokens"] = usage.get("candidatesTokenCount")
    row["thinking_tokens"] = usage.get("thoughtsTokenCount") or 0

    try:
        parts = payload["candidates"][0]["content"]["parts"]
        text = "".join(p.get("text", "") for p in parts if not p.get("thought"))
        answers = json.loads(text)
    except (KeyError, IndexError, json.JSONDecodeError) as exc:
        row["error"] = f"Cannot interpret response: {type(exc).__name__}: {json.dumps(payload, ensure_ascii=False)[:300]}"
        return row

    thresholds = {"exact": args.threshold_exact, "semantic": args.threshold_semantic}
    for key in ("exact", "semantic"):
        value = answers.get(key)
        expected = case[f"expect_{key}"]
        if args.mode == "bool":
            prob, got = None, value
        else:
            prob = value
            got = None if value is None else value >= thresholds[key]
        row[f"{key}_prob"] = prob
        row[f"{key}_threshold"] = thresholds[key] if args.mode == "prob" else None
        row[f"{key}_got"] = got
        row[f"{key}_expected"] = expected
        row[f"{key}_ok"] = None if expected is None else (got == expected)
    return row

def summarize(rows: list[dict], args) -> None:
    errors = [r for r in rows if "error" in r]
    ok_rows = [r for r in rows if "error" not in r]

    print("\n=== Results Summary ===")
    print(f"Model: {args.model} / thinking {args.thinking_level} / output {args.mode}")
    print(f"Cases: {len(rows)}  Success: {len(ok_rows)}  Failure: {len(errors)}")

    for key in ("exact", "semantic"):
        scored = [r for r in ok_rows if r.get(f"{key}_ok") is not None]
        hit = sum(1 for r in scored if r[f"{key}_ok"])
        rate = f"{hit / len(scored):.1%}" if scored else "-"
        print(f"{key:9s} correct {hit}/{len(scored)} ({rate})")

    print("\n--- By category (semantic) ---")
    for cat in sorted({r["category"] for r in ok_rows}):
        scored = [r for r in ok_rows
                  if r["category"] == cat and r.get("semantic_ok") is not None]
        if scored:
            hit = sum(1 for r in scored if r["semantic_ok"])
            print(f"{cat:12s} {hit}/{len(scored)} ({hit / len(scored):.0%})")

    misses = [r for r in ok_rows if r.get("semantic_ok") is False]
    if misses:
        print("\n--- Cases where semantic was missed ---")
        for r in misses:
            print(f"{r['id']:5s} {r['note']}  expected={r['semantic_expected']} "
                  f"actual={r['semantic_got']} (p={r['semantic_prob']})")

    if ok_rows:
        lat = sorted(r["latency_ms"] for r in ok_rows)
        p = lambda q: lat[min(int(len(lat) * q), len(lat) - 1)]  # noqa: E731
        print("\n--- Latency (ms) ---")
        print(f"Mean {statistics.mean(lat):.0f} / Median {p(0.5):.0f} / "
              f"p95 {p(0.95):.0f} / Min {lat[0]:.0f} / Max {lat[-1]:.0f}")

        n = len(ok_rows)
        tin = sum(r.get("input_tokens") or 0 for r in ok_rows)
        tout = sum(r.get("output_tokens") or 0 for r in ok_rows)
        tthink = sum(r.get("thinking_tokens") or 0 for r in ok_rows)
        print("\n--- Tokens ---")
        print(f"Input total {tin} (avg {tin / n:.0f}/case) / Output total {tout} (avg {tout / n:.1f}/case) / "
              f"Thinking total {tthink} (avg {tthink / n:.1f}/case)")
        price_in = args.price_in if args.price_in is not None else PRICES.get(args.model, (None, None))[0]
        price_out = args.price_out if args.price_out is not None else PRICES.get(args.model, (None, None))[1]
        if price_in is None or price_out is None:
            print("Estimated cost: Unit price not registered. Specify with --price-in / --price-out")
        else:
            cost = tin / 1e6 * price_in + (tout + tthink) / 1e6 * price_out
            print(f"Estimated cost: ${cost:.6f} (input ${price_in} / output+thinking ${price_out} per 1Mtok)")

    for r in errors:
        print(f"[ERROR] {r['id']}: {r['error']}", file=sys.stderr)

def main() -> int:
    ap = argparse.ArgumentParser(description="Measure the same cross-matching cases as JEV using Gemini")
    ap.add_argument("--cases", default=str(JEV_DIR / "cases.jsonl"))
    ap.add_argument("--out", help="Default is results_<model>_<mode>_<lang>.jsonl")
    ap.add_argument("--lang", choices=("ja", "en"), default="en")
    ap.add_argument("--model", default=DEFAULT_MODEL)
    ap.add_argument("--mode", choices=("bool", "prob"), default="bool",
                    help="bool returns only true/false. prob returns a probability from 0 to 1 and judges by threshold")
    ap.add_argument("--thinking-level", choices=("minimal", "low", "medium", "high"),
                    help="Default is the minimum value the model accepts")
    ap.add_argument("--threshold-exact", type=float, default=0.8)
    ap.add_argument("--threshold-semantic", type=float, default=0.8)
    ap.add_argument("--price-in", type=float, help="USD / 1M input tokens")
    ap.add_argument("--price-out", type=float, help="USD / 1M output tokens")
    ap.add_argument("--concurrency", type=int, default=1,
                    help="Parallelism. Leave at 1 for accurate latency measurement")
    ap.add_argument("--keepalive", action="store_true",
                    help="Reuse HTTP connections. Conditions differ from the JEV measurement approach (new connection each time)")
    ap.add_argument("--limit", type=int, help="Run only the first N cases (for connectivity check)")
    ap.add_argument("--timeout", type=float, default=60.0)
    args = ap.parse_args()
    if args.thinking_level is None:
        args.thinking_level = MIN_THINKING.get(args.model, "low")
    suffix = "_keepalive" if args.keepalive else ""
    out_path = args.out or f"results_{args.model}_{args.mode}_{args.lang}{suffix}.jsonl"

    with open(args.cases, encoding="utf-8") as f:
        cases = [json.loads(line) for line in f if line.strip()]
    if args.limit:
        cases = cases[:args.limit]

    api_key = os.environ.get("GEMINI_API_KEY")
    if not api_key:
        print("GEMINI_API_KEY is not set. Please export it.", file=sys.stderr)
        return 1

    print(f"Sending {len(cases)} cases to {args.model} (question language {args.lang} / "
          f"thinking {args.thinking_level} / output {args.mode} / concurrency {args.concurrency})...",
          file=sys.stderr)

    run = lambda c: judge(c, call_gemini(c, api_key, args), args)  # noqa: E731

    wall = time.perf_counter()
    if args.concurrency > 1:
        with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
            rows = list(pool.map(run, cases))
    else:
        rows = [run(c) for c in cases]
    wall = time.perf_counter() - wall

    with open(out_path, "w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

    summarize(rows, args)
    print(f"\nTotal time: {wall:.1f}s ({wall / len(rows):.2f}s/case)")
    print(f"Details: {out_path}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The prompt passed to Gemini consists only of a system prompt that lists the question text from last time, and a JSON containing the two values to be compared.

# System prompt (probability mode)
[exact] Are value_a and value_b identical as strings? Any difference in whitespace, punctuation, letter case, or full-width versus half-width characters counts as not identical.
  true: Identical character for character.
  false: They differ by at least one character.
[semantic] Do value_a and value_b refer to the same underlying value, even when they are written differently? This is a cross-check of the same field on two business documents. When one side lists itemized values and the other states a total, add the items up and compare that sum against the stated total.
  true: Only the format, language, unit, or abbreviation differs and the underlying value is the same, or the itemized values add up to the stated total.
  false: The underlying value itself differs, one side has no value, or the itemized values do not add up to the stated total.
Answer each question with the probability (0 to 1) that it is true.

# User message
{"value_a": "INV-2026-0912", "value_b": "INV-2026-0912"}

3.3 Execution

Issue an API key in Google AI Studio and set it as an environment variable.
If you use a key from a project without a billing account linked, it will run within the free tier as-is.

export GEMINI_API_KEY="<YOUR_API_KEY>"

Run with the mode set to return probabilities and the option to reuse connections.

python3 run_eval_gemini.py --mode prob --keepalive

# Example output
Sending 107 cases to gemini-3.8-flash (question language en / thinking low / output prob / concurrency 1)...

=== Results Summary ===
Model: gemini-3.8-flash / thinking low / output prob
Cases: 107  Success: 107  Failure: 0
exact     correct 107/107 (100.0%)
semantic  correct 101/106 (95.3%)

--- By category (semantic) ---
address      10/11 (91%)
aggregate    7/8 (88%)
baseline     9/9 (100%)
date         12/12 (100%)
description  10/10 (100%)
missing      7/7 (100%)
number       17/18 (94%)
ocr          7/8 (88%)
party        14/15 (93%)
whitespace   8/8 (100%)

--- Cases where semantic was missed ---
3-6   One side has no currency notation, cannot definitively say it's the same amount  expected=False actual=True (p=1.0)
4-4   Pre/post corporate designation (policy-dependent)  expected=True actual=False (p=0)
5-3   Granularity difference. City name only vs. with prefecture cannot be said to be the same address  expected=False actual=True (p=0.95)
7-3   O misread in number. Different characters means different value  expected=False actual=True (p=1)
9-4   Quantity matches  expected=True actual=False (p=0.2)

--- Latency (ms) ---
Mean 1877 / Median 1635 / p95 3380 / Min 1147 / Max 5607

--- Tokens ---
Input total 26330 (avg 246/case) / Output total 1653 (avg 15.4/case) / Thinking total 4532 (avg 42.4/case)
Estimated cost: $0.042941 (input $0.75 / output+thinking $3.75 per 1Mtok)

Total time: 200.8s (1.88s/case)
Details: results_gemini-3.8-flash_prob_en_keepalive.jsonl

4. Results: Gemini for accuracy, Jev for speed and cost

4.1 Overall numbers

Cost is converted at 150 yen per dollar.
The unit price for Gemini 3.8 Flash was calculated using the introductory pricing valid through December 31, 2026 ($0.75 per million input tokens, $3.75 for output).
The output unit price also applies to thinking tokens.
Note that both unit prices are scheduled to double from January 1, 2027.

Source: Pricing: Gemini Developer API pricing | Google AI for Developers

4.2 Even with limited output, it was more than twice as slow

The median latency was approximately 2.3 times that of Jev.
Thinking "could it be slow because of thinking?", I extracted only the 57 cases where thinking was 0 tokens and aggregated them, but even then the median was 1616ms.
In these 57 cases, the output Gemini returned was only about a dozen tokens.

In other words, the main cause of the slowness is not the amount of token generation, but rather the processing time to read the input and return the first token.
The reasoning that "it's fast because it doesn't generate text" alone cannot explain Jev's overwhelming speed.
The aim of cutting output to the bare minimum to achieve Jev-level speed could not be realized with Gemini 3.8 Flash.

Additionally, the large spike of p95 exceeding 3 seconds is due to thinking.
Even with low specified, thinking occurred in 50 out of 107 cases, and in those 50 cases, the model was reasoning behind the scenes at an average of about 91 tokens.
It seems that when asked "what is the probability?", the model tends to think carefully.

4.3 The price difference was determined by the unit price of input and output in the first place

In terms of cost, there was approximately a 19x difference.
What's interesting is that the average number of input tokens itself is less than half for Gemini.
Even with the same question text, Jev tends to use more tokens because typed questions need to be passed as a JSON schema, while Gemini can accept them as plain text, keeping the count lower.

There are two reasons why the cost doesn't reverse even so.
The first is the unit price of input tokens, which has a gap of about 18x ($0.042 vs $0.75).
Even comparing just the input costs, Gemini was about 8.6x more expensive than Jev.
The second is the output-side cost.
Jev's output tokens are free, but Gemini charges $3.75 per unit for both output and thinking.
The cost breakdown for Gemini was 46% input, 14% output, and 40% thinking, with the output side alone exceeding half.

Even if you cut the output down to a dozen or so tokens, the thinking that you can't cut yourself uses nearly 3x as many tokens and is charged at 5x the unit price of input.

4.4 Gemini's probabilities were stuck at either 0 or 1

This was personally my biggest discovery.
I divided the distribution of semantic probabilities into three bands.

The cases where Gemini "was uncertain" numbered just 1 out of 107.
For semantic, 64 cases were exactly 1.0 and 29 cases were exactly 0.0.
For exact, all 107 cases swung to either 0.0 or 1.0.

This isn't a problem as long as the accuracy rate is high.
However, the trouble comes when it makes a mistake.
The output probabilities for the 5 cases where Gemini was incorrect were 1.0, 1.0, 0.95, 0.0, and 0.2 — all wrong with full confidence.
Even if you set up a flow like "send anything uncertain at the 0.8 threshold to human visual inspection," these 5 cases would either pass through with high probability or be unconditionally rejected.

As introduced in the previous article, Jev returns an exquisitely calibrated probability of around 0.6 to 0.7 for cases where it can't make a judgment.
The difference between whether a probability value is "a self-reported value output as text" or "the direct result of the model's internal calculation" is clearly shown here.
If you want to use probability values as operational thresholds or gates, Jev's probabilities are far more practical.

The probabilities for each case are listed in full in Chapter 5 below.

5. Test data and probabilities (all 107 cases)

For each of the 107 cases, we list the probabilities returned by the two models side by side.
The exact and semantic columns in the table are the correct labels.
The Jev and Gemini columns list the exact match probability (exact) / semantic match probability (semantic) in that order.
The Jev values are the same as in the English version of the previous article.
Values shown in bold are results where the judgment at threshold 0.8 did not match the correct answer.

Invisible characters in strings are represented the same as in the previous article: ␣ for half-width space, ␠ for full-width space, ⏎ for newline, and → for tab.
The method for determining correct labels is as described in Chapter 4 of the previous article.

5.1 Identical (9 cases)

5.2 Dates (13 cases)

The combination of 03/04/2026 and 2026-03-04 is excluded from the accuracy count, as before, because the correct answer cannot be uniquely determined.

For Sep 12, 2026 and 12/09/2026, Jev returned 0.71 and Gemini returned 0.95.
Since 12/09 can be read as either December 9th or September 12th, Jev lowered its probability.
Gemini thought for 184 tokens before judging them as a match.

5.3 Numeric values / amounts (18 cases)

For 1,250.00 and 1,250.00 USD, both Jev at 0.96 and Gemini at 1.00 incorrectly identified them as a match.
The weakness noted in the previous article regarding cases where only one side includes a currency notation applies equally to Gemini.

5.4 Company names (15 cases)

Both models were incorrect for the pre-position vs. post-position company name (株式会社サンプル商事 vs. サンプル商事株式会社), but they were wrong in different ways.
Jev returned 0.75, meaning "probably the same but not confident," while Gemini thought for 244 tokens and declared 0.00, asserting "they are different legal entities."
As noted in the previous article, this is a gray zone that can go either way depending on business rules.
I feel that having uncertain cases returned as uncertain makes it easier to incorporate into real-world operations.

5.5 Addresses (11 cases)

5.6 Product names (10 cases)

5.7 OCR misreads (8 cases)

For 1,25O.00 (the letter O in place of zero) and 1,250.00, Gemini thought for 96 tokens and answered with a probability of 1.00, judging them as a match.
It helpfully reinterpreted the OCR misread.
Since the purpose of matching is "to detect differences in the values written on documents," this leads to a missed discrepancy.
Jev also returned a high value of 0.76, but since it fell below the 0.8 threshold, it can be sent for human review.

5.8 Missing values (7 cases)

Looking only at the number of correct answers, Jev scored 3/7 and Gemini scored 7/7, making it look like a landslide for Gemini.
However, this category touches on "whether it is acceptable to treat two values with no content as the same value," which is a judgment call that varies by person.
This time the correct label was set to "Match," but depending on the business context, "Mismatch" or "Requires review" may be more natural.
Gemini thought for 304 tokens when comparing two empty fields and still declared 1.00.
Jev returned low probabilities of 0.11 to 0.43 for the 4 cases it got wrong, so they can be sent for human review under the 0.8 threshold operation.
In terms of being able to defer uncertain judgments to humans rather than making definitive calls, Jev's behavior in this category is easier to work with.
Note that the difference in the semantic correct answer count is 96/106 for Jev and 101/106 for Gemini, but excluding the 7 missing value cases, it becomes 93/99 vs. 94/99, which is nearly identical.

5.9 Aggregations (8 cases)

For the case where the line item total is 450 KG but 500 KG is written, Jev incorrectly identified them as a match at 0.89, while Gemini correctly judged them as a mismatch at 0.00.
Gemini used 100 tokens of thinking in this case, suggesting it performed addition in the background.
On the other hand, for 1 箱 and 管理番号 ABCD1234567 (1箱), Jev got it right at 0.91 while Gemini missed it at 0.20, though I'm actually inclined to think Gemini may be correct since it's debatable whether these are truly semantically equivalent.

5.10 Invisible differences (8 cases)

6. Summary

My hypothesis that "Jev is fast because it doesn't generate text, and an LLM should be equivalent if you limit its output" was wrong.
Even with Gemini 3.8 Flash with output trimmed to a dozen or so tokens, the median latency was about 2.3x slower and the cost in paid tier terms was about 19x higher.
The latency difference arises at the stage of reading the initial input, and the cost difference widens due to differences in input token unit prices and whether output and thinking are billed.
The semantic accuracy rate came out higher for Gemini, but most of the difference came from the contentious missing value category, and the results were nearly equivalent in all other categories.

The distinction between the two can be summarized as follows:

1. For high-volume matching where speed and cost are critical, use Jev (use the calculated probabilities directly as the criterion for deciding whether to send a case for human review)

2. For items requiring reasoning such as calculating line item totals, use a generative LLM, or pre-calculate on the upstream program side as mentioned in the previous article

3. Even if you have a generative LLM output a probability, the values tend to be biased toward 0 or 1, so don't over-rely on them as a confidence judgment

Supplement: Efforts to use DiffusionGemma like Jev

This time we tested with Gemini 3.8 Flash, but models that can be used in a similar way to Jev seem likely to increase going forward.
The official Google Gemma account also introduced an attempt to run DiffusionGemma like Jev.
DiffusionGemma is a diffusion model that generates the whole output at once, rather than outputting tokens one by one in sequence.
According to the post, option judgment can be completed in a single parallel pass, and on DGX Spark results are returned in approximately 0.2 seconds.
This mechanism has been published as a pull request to vLLM.
As of September 21, 2026, it has not yet been merged and is still at the prototype stage.

Reference: Post: "DiffusionGemma as Jev" | Google Gemma (X)

Reference: Pull request: structured generation mode for DiffusionGemma model (Jev-like) | vllm-project/vllm

While Jev only supports text input, DiffusionGemma inherits Gemma 4's image recognition capabilities and can apparently be used for judgments that include images.

Inherits Gemma 4's spatial vision capabilities for complex visual and text decisions.

I haven't been able to try it myself yet, but I'm very much looking forward to being able to run it.

References

• Guide: Structured outputs | Google AI for Developers

• Guide: Gemini thinking | Google AI for Developers

• Guide: Using Gemini API keys | Google AI for Developers

• Introducing System One Models & Jev | TypeSafe AI Blog