[TypeSafe] How Far Can Jev Go for Document Item Matching? Measuring Accuracy, Speed, and Cost

Hello, I'm Kema.

For a certain project, I developed a feature that checks whether the same items recorded in two documents are semantically consistent.
The tricky part here is variations in notation.
For example, for a date field, one document might say 2026/09/12 while the other says 2026年9月12日, or a company name might differ in case like ACME CO., LTD. versus Acme Co., Ltd..
A person can tell they're the same, but a simple string comparison would flag all of them as mismatches.
On the other hand, once you start writing regex or custom normalization rules, there are endless variations—Japanese era names, European decimal points, abbreviations for corporate entities, and so on—making it practically impossible to cover everything.

For that reason, we had traditionally been calling Claude models via Amazon Bedrock to determine semantic consistency.
However, running large volumes of matching operations inevitably drives up costs.
So I thought TypeSafe's Jev might work as an alternative to Claude models.

Jev is a model that doesn't generate text—it only returns probability values in response to typed questions.
Input is limited to text, but since we're dealing with string-to-string comparisons here, this falls squarely within Jev's area of strength.

In this article, I prepared 107 fictional document field cases and measured on September 20, 2026, how accurately Jev could perform the matching.
In addition to accuracy, I also recorded latency and token consumption.
I also ran both versions—questions written in Japanese and questions written in English—and compared the differences in results.

1. Prerequisites for Matching

1.1 Two Axes for Judgment

In real-world document matching, depending on the document and field, you may want to judge based on whether values are "strictly identical without a single character difference (Exact)," or evaluate whether "two values refer to the same meaning even if written differently (Semantic)."

So I split the questions to Jev into these two:

• exact: Whether the strings are identical character for character

• semantic: Whether two values refer to the same thing even if written differently

For 2026/09/12 and 2026年9月12日, the expected result is that exact is a mismatch and semantic is a match.

Of course, for exact matching (exact) alone, a simple string comparison on the program side would be faster and more reliable.
However, building out individual logic to decide "check exact match programmatically here, check semantic match with the model there" for each document and field would make the code and rules complex to manage.
That's why I wanted to verify what happens when you let Jev handle both Exact and Semantic judgment on the model side—that's the reason I set up these two axes.

1.2 Threshold of 0.8

Jev's noul type returns a probability value between 0 and 1.
The threshold for classifying this probability value into a binary match/mismatch was set to 0.8 for both exact and semantic.

In practice, it's common to see operational designs where anything with a confidence score below 0.8 gets flagged for review and passed to a human for verification.
Classifying notation variations as "the same value" carries the risk of letting incorrect matches slip through.
Therefore, in this evaluation, keeping real-world operations in mind, 0.8 was set as the threshold for erring on the side of caution.

2. Environment

The API specification follows the official reference.
There are only three request parameters: state (data serving as the basis for judgment), model, and questions (a map of typed questions).
The response returns judgment results under the same keys as the questions, and token counts are recorded in usage.

Source: API reference | TypeSafe Docs

Parallel execution was not used in order to accurately measure latency.

3. Execution Script and How to Run

3.1 Test Data Format

The test data was prepared in JSONL format with one case per line.

cases.jsonl:

{"id": "2-1", "category": "date", "a": "2026/09/12", "b": "2026年9月12日", "expect_exact": false, "expect_semantic": true, "note": "Japanese notation"}
{"id": "3-9", "category": "number", "a": "500 KG", "b": "1102.31 LBS", "expect_exact": false, "expect_semantic": true, "note": "Unit conversion required"}
{"id": "4-6", "category": "party", "a": "ACME TRADING CO.", "b": "ACNE TRADING CO.", "expect_exact": false, "expect_semantic": false, "note": "Different company with one character difference"}

a and b are the two values to be matched, and expect_exact and expect_semantic contain the correct labels for each.

The specific contents of all 107 cases are the data itself listed in the table in Section 4.

3.2 Script

There are no external library dependencies; it runs with Python's standard library only.

run_eval.py:

#!/usr/bin/env python3
"""JEV(TypeSafe AI System One)で横突合の精度・レイテンシ・トークンを測る。

各ケースについて値A・値Bを state として渡し、noul(yes/no確率)で
「完全一致か」「意味として同じ値か」を同時に聞く。1リクエストで両方answerが返る。

API: POST https://api.typesafe.ai/v1/systemone
     https://docs.typesafe.ai/api.md
"""

from __future__ import annotations

import argparse
import json
import os
import statistics
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

ENDPOINT = "https://api.typesafe.ai/v1/systemone"
DEFAULT_MODEL = "jev-latest"

# 入力トークンのみ課金。出力トークンは無料
USD_PER_INPUT_MTOK = 0.042

# state のキー名。質問文と同じ言語に揃える。
STATE_KEYS = {"ja": ("値A", "値B"), "en": ("value_a", "value_b")}

# 質問文の言語だけを変えた2セット。中身の意味は揃えてある。
QUESTIONS = {
    "ja": {
        "exact": {
            "type": "noul",
            "instructions": (
                "値Aと値Bは、文字列として完全に同一か。"
                "空白・記号・大文字小文字・全角半角の違いも「同一ではない」と扱う。"
            ),
            "criteria": {
                "true": "1文字も違わず完全に同じ",
                "false": "1文字でも違う",
            },
        },
        "semantic": {
            "type": "noul",
            "instructions": (
                "値Aと値Bは、表記が違っていても同じ値を指しているか。"
                "これは2つの書類の同じ項目を突合している場面での判断である。"
                "片方が明細の羅列で、もう片方が合計値の場合は、"
                "明細をすべて足し合わせてから合計値と比べること。"
            ),
            "criteria": {
                "true": ("書式・言語・単位・略記の違いだけで、指している値は同じ。"
                         "または明細の合計が合計値と一致する"),
                "false": ("指している値そのものが違う、または片方にしか値がない。"
                          "または明細の合計が合計値と一致しない"),
            },
        },
    },
    "en": {
        "exact": {
            "type": "noul",
            "instructions": (
                "Are value_a and value_b identical as strings? "
                "Any difference in whitespace, punctuation, letter case, or "
                "full-width versus half-width characters counts as not identical."
            ),
            "criteria": {
                "true": "Identical character for character.",
                "false": "They differ by at least one character.",
            },
        },
        "semantic": {
            "type": "noul",
            "instructions": (
                "Do value_a and value_b refer to the same underlying value, "
                "even when they are written differently? This is a cross-check "
                "of the same field on two business documents. When one side "
                "lists itemized values and the other states a total, add the "
                "items up and compare that sum against the stated total."
            ),
            "criteria": {
                "true": (
                    "Only the format, language, unit, or abbreviation differs "
                    "and the underlying value is the same, or the itemized "
                    "values add up to the stated total."
                ),
                "false": (
                    "The underlying value itself differs, one side has no "
                    "value, or the itemized values do not add up to the "
                    "stated total."
                ),
            },
        },
    },
}

def build_questions(lang: str) -> dict:
    """1リクエストで聞く質問セットを組み立てる。"""
    src = QUESTIONS[lang]
    return {"exact": src["exact"], "semantic": src["semantic"]}

def call_jev(case: dict, questions: dict, api_key: str, model: str,
             timeout: float, lang: str) -> dict:
    """1ケースをJEVに投げ、応答とレイテンシを返す。"""
    key_a, key_b = STATE_KEYS[lang]
    body = json.dumps({
        "state": {key_a: case["a"], key_b: case["b"]},
        "model": model,
        "questions": questions,
    }, ensure_ascii=False).encode("utf-8")

    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        method="POST",
    )

    started = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            payload = json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        detail = exc.read().decode("utf-8", errors="replace")[:500]
        return {"error": f"HTTP {exc.code}: {detail}",
                "latency_ms": (time.perf_counter() - started) * 1000}
    except Exception as exc:  # ネットワーク断・タイムアウト
        return {"error": f"{type(exc).__name__}: {exc}",
                "latency_ms": (time.perf_counter() - started) * 1000}

    return {"payload": payload,
            "latency_ms": (time.perf_counter() - started) * 1000}

def judge(case: dict, result: dict, thresholds: dict[str, float]) -> dict:
    """応答を期待値と突き合わせて1行分のレコードにする。"""
    row = {
        "id": case["id"],
        "category": case["category"],
        "note": case["note"],
        "latency_ms": round(result["latency_ms"], 1),
    }
    if "error" in result:
        row["error"] = result["error"]
        return row

    payload = result["payload"]
    answers = payload.get("answers", {})
    usage = payload.get("usage") or {}

    row["model"] = payload.get("model")
    row["input_tokens"] = usage.get("input_tokens")
    row["output_tokens"] = usage.get("output_tokens")

    for key in ("exact", "semantic"):
        prob = answers.get(key, {}).get("noul")
        expected = case[f"expect_{key}"]
        got = None if prob is None else prob >= thresholds[key]
        row[f"{key}_prob"] = prob
        row[f"{key}_threshold"] = thresholds[key]
        row[f"{key}_got"] = got
        row[f"{key}_expected"] = expected
        # 期待値 None は正解が決まらないケースなので正答率から外す
        row[f"{key}_ok"] = None if expected is None else (got == expected)

    return row

def summarize(rows: list[dict]) -> None:
    errors = [r for r in rows if "error" in r]
    ok_rows = [r for r in rows if "error" not in r]

    print("\n=== 結果サマリ ===")
    print(f"ケース数: {len(rows)}  成功: {len(ok_rows)}  失敗: {len(errors)}")

    for key in ("exact", "semantic"):
        scored = [r for r in ok_rows if r.get(f"{key}_ok") is not None]
        hit = sum(1 for r in scored if r[f"{key}_ok"])
        rate = f"{hit / len(scored):.1%}" if scored else "-"
        th = ok_rows[0].get(f"{key}_threshold") if ok_rows else None
        print(f"{key:9s} 正答 {hit}/{len(scored)} ({rate})  [閾値 {th}]")

    print("\n--- カテゴリ別(semantic) ---")
    for cat in sorted({r["category"] for r in ok_rows}):
        scored = [r for r in ok_rows
                  if r["category"] == cat and r.get("semantic_ok") is not None]
        if not scored:
            continue
        hit = sum(1 for r in scored if r["semantic_ok"])
        print(f"{cat:12s} {hit}/{len(scored)} ({hit / len(scored):.0%})")

    for key in ("exact", "semantic"):
        misses = [r for r in ok_rows if r.get(f"{key}_ok") is False]
        if not misses:
            continue
        print(f"\n--- {key} を外したケース ---")
        for r in misses:
            print(f"{r['id']:5s} {r['note']}  期待={r[f'{key}_expected']} "
                  f"(p={r[f'{key}_prob']:.3f})")

    if ok_rows:
        lat = sorted(r["latency_ms"] for r in ok_rows)
        p = lambda q: lat[min(int(len(lat) * q), len(lat) - 1)]  # noqa: E731
        print("\n--- レイテンシ(ms) ---")
        print(f"平均 {statistics.mean(lat):.0f} / 中央値 {p(0.5):.0f} / "
              f"p95 {p(0.95):.0f} / 最小 {lat[0]:.0f} / 最大 {lat[-1]:.0f}")

        tin = sum(r.get("input_tokens") or 0 for r in ok_rows)
        tout = sum(r.get("output_tokens") or 0 for r in ok_rows)
        print("\n--- トークン ---")
        print(f"入力 合計 {tin} (平均 {tin / len(ok_rows):.0f}/件) / 出力 合計 {tout}")
        print(f"概算コスト: ${tin / 1_000_000 * USD_PER_INPUT_MTOK:.6f} "
              f"(入力 ${USD_PER_INPUT_MTOK}/1Mtok、出力は課金対象外)")

    for r in errors:
        print(f"[ERROR] {r['id']}: {r['error']}", file=sys.stderr)

def main() -> int:
    ap = argparse.ArgumentParser(description="JEVで横突合の精度・速度・トークンを測る")
    ap.add_argument("--cases", default="cases.jsonl")
    ap.add_argument("--out", help="デフォルトは results_<lang>.jsonl")
    ap.add_argument("--lang", choices=("ja", "en"), default="en",
                    help="質問文とstateキーの言語")
    ap.add_argument("--model", default=DEFAULT_MODEL)
    ap.add_argument("--threshold-exact", type=float, default=0.8,
                    help="exactのnoul確率を一致とみなす閾値")
    ap.add_argument("--threshold-semantic", type=float, default=0.8,
                    help="semanticのnoul確率を一致とみなす閾値")
    ap.add_argument("--concurrency", type=int, default=1,
                    help="並列数。レイテンシを正しく測るなら1のまま")
    ap.add_argument("--timeout", type=float, default=30.0)
    args = ap.parse_args()
    out_path = args.out or f"results_{args.lang}.jsonl"

    with open(args.cases, encoding="utf-8") as f:
        cases = [json.loads(line) for line in f if line.strip()]

    questions = build_questions(args.lang)

    api_key = os.environ.get("TYPESAFE_API_KEY")
    if not api_key:
        print("TYPESAFE_API_KEY が未設定です。export してください。", file=sys.stderr)
        return 1

    print(f"{len(cases)} ケースを {args.model} に投げます "
          f"(質問文 {args.lang} / 並列 {args.concurrency})...", file=sys.stderr)

    thresholds = {"exact": args.threshold_exact,
                  "semantic": args.threshold_semantic}

    run = lambda c: judge(  # noqa: E731
        c, call_jev(c, questions, api_key, args.model, args.timeout, args.lang),
        thresholds)

    wall = time.perf_counter()
    if args.concurrency > 1:
        with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
            rows = list(pool.map(run, cases))
    else:
        rows = [run(c) for c in cases]
    wall = time.perf_counter() - wall

    with open(out_path, "w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

    summarize(rows)
    print(f"\n全体所要: {wall:.1f}s ({wall / len(rows):.2f}s/件)")
    print(f"明細: {out_path}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

3.3 Running the Script

Set the API key as an environment variable.

export TYPESAFE_API_KEY="<YOUR_API_KEY>"

Navigate to the directory containing the script and test data, then run the script with the question language set to English.

python3 run_eval.py --lang en

When running the Japanese version, the output destination automatically switches to results_ja.jsonl, so there's no risk of overwriting the English results file.

python3 run_eval.py --lang ja

When run, the following summary is output.

# Example output
107 ケースを jev-latest に投げます (質問文 en / 並列 1)...

=== 結果サマリ ===
ケース数: 107  成功: 107  失敗: 0
exact     正答 106/107 (99.1%)  [閾値 0.8]
semantic  正答 96/106 (90.6%)  [閾値 0.8]

--- カテゴリ別(semantic) ---
address      10/11 (91%)
aggregate    7/8 (88%)
baseline     9/9 (100%)
date         10/12 (83%)
description  10/10 (100%)
missing      3/7 (43%)
number       17/18 (94%)
ocr          8/8 (100%)
party        14/15 (93%)
whitespace   8/8 (100%)

--- レイテンシ(ms) ---
平均 702 / 中央値 709 / p95 872 / 最小 537 / 最大 1001

--- トークン ---
入力 合計 54508 (平均 509/件) / 出力 合計 3852
概算コスト: $0.002289 (入力 $0.042/1Mtok、出力は課金対象外)

全体所要: 75.1s (0.70s/件)
明細: results_en.jsonl

The output results_*.jsonl files record the probability values returned by Jev as-is.

4. Test Data and Results (All 107 Cases)

The exact and semantic columns in the table show the correct labels.
The ja probability (exact / semantic) and en probability (exact / semantic) columns contain the values returned by Jev, listed in order of exact match probability (exact) / semantic match probability (semantic).
Values shown in bold are results where the judgment at threshold 0.8 did not match the correct answer.

For invisible characters contained within strings, ␣ represents a half-width space, ␠ represents a full-width space, ⏎ represents a newline, and → represents a tab.

4.1 Exact Matches (9 cases)

4.2 Dates (13 cases)

Note that the combination of 03/04/2026 and 2026-03-04 is a match if interpreted as month/day format (MM/DD), but a mismatch if interpreted as day/month format (DD/MM), making it impossible to determine a unique correct answer without context.
Therefore, this case is excluded from accuracy calculations.

4.3 Numbers and Amounts (18 cases)

The comparison of 500 KG and 1102.31 LBS has its correct label set with a policy of considering them identical after unit conversion.
If the system requirements do not allow unit conversion, the expected value of the correct label would be reversed.

4.4 Counterparty Names (15 cases)

4.5 Address (11 cases)

4.6 Item Description (10 cases)

4.7 OCR Misread (8 cases)

For OCR misread patterns, the correct labels are set based on the policy that "if even one character differs, it is a different value."

4.8 Missing Values (7 cases)

4.9 Aggregation (8 cases)

4.10 Invisible Differences (8 cases)

5. Accuracy, Latency, and Tokens

5.1 Overall Numbers

※ Latency p95 refers to the response time at the 95th percentile when all requests are sorted by response time from shortest to longest — the point that excludes the slowest top 5%. It indicates a stability benchmark meaning "95% of requests completed within this number of seconds."

The cost is an estimated value calculated by multiplying the measured input token count by a unit price of $0.042 per 1 million tokens (at a rate of 150 JPY per dollar).
Under Jev's billing structure, output tokens are not charged.

Reference: Jev | Vercel AI Gateway

5.2 semantic Correct Answers by Category

Only the missing values category showed a notably low accuracy rate; all other categories maintained an accuracy rate of roughly 90% or higher.

6. Key Findings from the Validation Results

6.1 Absorbing Notation Variations and Identifying Differences Are in the Practical Range

Conversions from the Japanese era system to the Gregorian calendar (令和8年9月12日 and 2026-09-12), European-style decimal notation (1.250,00 and 1,250.00), corporate entity abbreviations (GLOBEX CORPORATION and GLOBEX CORP.), and matching English addresses against Japanese addresses were all consistently judged as matches with high confidence. Compared to the effort of building complex normalization rules from scratch, delegating this domain to Jev is sufficiently practical.

The accuracy in detecting different data as "different items" was also stable, with 29 out of 37 mismatched data points returning probabilities of 0.32 or below. Since clearly different data returns clearly low probabilities, this behavior is easy to incorporate into real-world operations.

6.2 Rate of Misidentifying Mismatches as Matches, and Gray Zone Behavior

The most problematic scenario in real-world matching systems is false positives — cases where data is actually a mismatch but gets automatically passed through because the model falsely reports a high-probability match.

Of the 37 cases defined as "mismatch" in this validation, only the following 2 cases were misidentified as matches with a high probability of 0.8 or above:

• Slight calculation error: Total 500 KG and Details: 200 KG / 200 KG / 50 KG (the detail sum is 450 KG, yet the model returned 0.89–0.91 probability of match)

• Presence or absence of currency unit: 1,250.00 and 1,250.00 USD (the model treated the unspecified currency as USD and returned 0.88–0.96 probability of match)

Excluding these 2 cases, mismatched data was consistently rejected with low probabilities (mostly 0.32 or below).

On the other hand, cases that were expected to match but fell below the 0.8 threshold (name order of corporate prefix/suffix, date formats, missing values) were gray zones where human judgment would also be divided, rather than model errors.

• Counterparty name (corporate prefix vs. suffix): 株式会社サンプル商事 and サンプル商事株式会社 stayed at a probability of 0.75–0.78. In practice, companies with the prefix and suffix positions differing are sometimes registered as separate legal entities, so returning a cautiously low probability rather than a confidently high one can be considered appropriate behavior.

• Date order (MM/DD vs. DD/MM): Sep 12, 2026 and 12/09/2026 (probability 0.69–0.71), and 03/04/2026 and 2026-04-03 (probability 0.77) fell below 0.8. Without context, this can be interpreted as either March 4th or April 3rd, so it is natural that confidence does not increase.

• Handling of missing values: Comparisons between N/A entries (probability 0.22) and between empty fields (probability 0.20) resulted in extremely low semantic probabilities. This represents a dividing line in interpretation — "can two items with no actual value be considered the same entity?" — and can also be viewed as the model reliably flagging ambiguous cases.

In this way, Jev returns scores of 0.9 or above for items it is confident about, and scores in the 0.6–0.7 range for gray zones where judgment is uncertain.
Since it honestly lowers the probability for things it cannot determine, using 0.8 as the threshold makes it very easy to design a workflow where uncertain items are reliably surfaced and routed for human review.

6.3 Weaknesses That Truly Require Attention in Practice

On the other hand, points requiring clear attention when integrating into real-world operations also came to light.

• Missing slight calculation errors: In a case where the total was 500 KG but the detail sum was 450 KG (200+200+50 KG), even with calculation instructions provided in the prompt, the model returned a probability of 0.89 and misidentified it as a match.

• Probabilities differ depending on language: Cases were observed where the probability values for the same data differed between Japanese and English prompts (e.g., 該当なし vs. 該当なし returned 0.88 in English and 0.61 in Japanese). While English prompts consume approximately 13% fewer tokens, the judgment output needs to be verified for each language.

• Treating the presence or absence of units as equivalent without consideration: When comparing 1,250.00 and 1,250.00 USD, the model considered them a match with a probability of 0.88–0.96. When only one side has a unit or currency symbol and the other does not, as in this case, the model helpfully fills in the gap and treats them as equivalent. It would have been better to include prerequisite rules in the prompt, such as "do not treat as identical if only one side has a unit" and "be cautious when units differ completely."

Note that cases where a trailing newline was included (1,250.00⏎ vs. 1,250.00) resulted in an exact match determination with match probability (0.92–0.94), but the presence or absence of a trailing newline often carries the same meaning in practice, so the operational impact is minor.

7. Countermeasures for Real-World Implementation

Based on the validation results and weaknesses above, it is safe to incorporate the following two points into the design when integrating into actual business operations.

1. Route items requiring calculation to human review or solve them externally: For matching line item sums against totals, either pass pre-computed confirmed values to Jev on the program side, or design the operation so that when the system determines "does this comparison require calculation?" and the answer is yes, it bypasses automatic processing and routes to human review. Jev's documentation also notes that it is weak at calculation.

2. Constrain unit presence and differences in the prompt: Explicitly state unit-related rules in the prompt in advance, such as being careful when units differ completely, and lowering confidence when only one side has a unit and the other does not.

8. Summary

From validation of 107 cases using fictional document fields, Jev demonstrated practical utility — stably absorbing notation variations in dates, numbers, company names, and addresses with high probabilities of 0.89 or above, and reliably rejecting differing items at 0.32 or below. Processing speed had a median of approximately 0.7 seconds, and the total cost for all 107 cases was approximately 0.35–0.41 JPY (approx. $0.002), making it extremely inexpensive.

Jev's strength lies in its ability to honestly return low probabilities for ambiguous data it is not confident about, naturally surfacing items that should be reviewed by a human. On the other hand, it has a weakness in that it can miss processing that requires calculation, so rather than delegating everything to it, division of responsibilities is important.

Handle addition and missing value judgments with upstream programs, and entrust Jev with "automatic resolution of notation variations" and "surfacing gray-zone items that require judgment." Designing the system with this division of responsibilities appears to be the safest and most effective approach.

References

• Primitives (Questions) | TypeSafe Docs

• Answers and responses | TypeSafe Docs

• Introducing System One Models & Jev | TypeSafe AI Blog

• テキスト専用の Jev に PDF や PNG を Base64 で渡したら読めるのか試してみた | DevelopersIO