Trying Out Document Value Reconciliation with OpenAI's Decisions API and Comparing It to Jev and Clef

Hello, I'm Keima.

When matching documents such as invoices against purchase orders, notation variations often occur where values that look the same to a person do not match as strings, such as 2026/09/12 and 2026年9月12日, or 株式会社サンプル商事 and サンプル商事株式会社.
In Trying Cloudflare's Clef for document value matching and comparing it with Jev | DevelopersIO, I tested this judgment of "do they refer to the same value?" with TypeSafe's Jev and Cloudflare's Clef. Jev judged 96 of 107 cases correctly, while clef's correct answers dropped to 76 when the prompt was written in Japanese.

In this article, I passed the same 107 cases and the same prompts to OpenAI's Decisions API (model: gpt-6-luna) and compared it with Jev and Clef.

1. Test Conditions

• Data: the same 107 cases as the article above (10 categories including dates, amounts, counterparty names, addresses, OCR misreads, and missing values). All values are fictional, and Japanese and English notations are mixed

• Judgment axes: two questions asked as predicate (a question that returns the probability of yes): exact (are they completely identical as strings?) and semantic (do they refer to the same value even if written differently?)

• Prompt language: two patterns, with the instructions given to the model in English and in Japanese. The values A and B to be matched are the same in both

• Threshold: 0.8. If the probability is 0.8 or higher, it is judged as a "match"

• Comparison targets: results from Jev (September 20, 2026, jev-1.13.0) and Clef (October 4, 2026, clef and clef-flash)

The request to the Decisions API looks like this (only the semantic question is shown).
The comments are for explanation and are not included in the JSON actually sent.

{
  "model": "gpt-6-luna",
  // the two values to match (the invoice value and the purchase order value)
  "input": "{\"value_a\": \"2026/09/12\", \"value_b\": \"2026年9月12日\"}",
  "questions": [{
    "type": "predicate", // a yes/no question. Returns the probability of yes
    "name": "semantic",
    // Do value_a and value_b refer to the same value even if written differently? (Matching the same field in two business documents. If one is line items and the other is a total, add up the line items and compare)
    // True when: only the format, language, unit, or abbreviation differs and the value is the same, or the sum of the line items matches
    // False when: the value itself differs, one side has no value, or the sum of the line items does not match
    "instructions": "Do value_a and value_b refer to the same underlying value, even when they are written differently? ... \nTrue when: Only the format, language, unit, or abbreviation differs ...\nFalse when: The underlying value itself differs, one side has no value, ..."
  }]
}

With Jev and Clef, the true and false conditions were passed separately in criteria, but the Decisions API predicate has no corresponding field, so I appended them to the end of instructions (Decisions API | OpenAI API).

2. Results

2.1 Overall

The table below shows the results tallied by judging a case as a "match" when the probability output by each model is 0.8 or higher.
The semantic correct count is tallied over the 106 cases for which the correct answer is determined.

The Decisions API judged 106 cases correctly on exact, the same as Jev, and on semantic it got 90 (English) and 91 (Japanese) correct, roughly on par with clef (English).
Unlike clef, the Decisions API did not lose accuracy when the prompt was written in Japanese.

The cases the Decisions API missed at the 0.8 threshold showed the following two tendencies.

• Missing values: for cases where one or both values were empty, the Decisions API also got the judgment wrong, just like Jev and Clef

• OCR-garbled strings: even for completely identical strings such as DOC-O0O12345 on both sides (1-7) and INVO1CE on both sides (7-7), the probability stayed at 0.66 to 0.77. For clean identical strings (such as 1-1) it gave 0.91 or higher

2.2 Cost

The Decisions API cost about 60% of clef and about 2.3 times Jev.

2.3 Probabilities for all cases

3. Summary

The Decisions API matched Jev on exact, and on semantic it also delivered a stable result with 91 correct answers using the Japanese prompt.
Unlike Clef, its accuracy did not drop when the prompt was written in Japanese, so I felt it is quite strong with Japanese as well.

The Decisions API had the shortest execution time, but since this also varies with how busy the servers are at the time, I consider the difference of about 0.3 to 0.7 seconds in this test to be within the margin of error.
When embedding it in an AI system, you are rarely required to respond in under a second, so speed does not seem to be a major concern.

On the other hand, Jev costs less than half as much as the Decisions API.
For workloads that process a large number of items, this difference of more than 2x will matter.

Also, the appropriate threshold differs for each model.
When embedding this in a system, please check how the probabilities are distributed on your actual data before deciding on a threshold.