I tried showing document images (Japanese) to OpenAI's Responses API to have it find errors in OCR results

Hello, I'm Keema.

In I had Cloudflare's Clef look at document images (Japanese) and find OCR errors | DevelopersIO, I passed Japanese document images and OCR extraction results to Cloudflare's Clef and had it determine whether the values matched what was printed in the images.
Clef was able to find 32 out of 36 injected errors, but it only caught 7 out of 19 actual misreads made by the OCR (Claude Haiku 4.5).

OpenAI's Decisions API (model gpt-6-luna) can also receive images and answer questions that return a yes probability.
So in this article, I passed the same three Japanese documents and the same questions as the Clef verification to the Decisions API, and compared whether it could catch Haiku's misreads against Clef.

1. Verification Conditions

The data is the same as the Clef verification.
When I had Haiku extract from three Japanese documents — an invoice, a quote, and a receipt (3 documents, 149 items total) — 19 misreads occurred naturally, so these are also included as detection targets.
Additionally, I injected 4 patterns of errors (36 cases total).

• Digit changes (12 cases): Change a single digit in amounts or numbers (51,750 → 51,790, MQ-2026-0918 → MQ-2026-0916)

• Similar character substitutions (9 cases): Replace with visually similar characters (カ → 力, 口 → ロ, ー → 一)

• Full-width and half-width swaps (3 cases): ２０食 → 20食, MN-2407 → ＭＮ－２４０７

• Replacement with other values (12 cases): Replace with other values that could plausibly appear in a document (普通 → 当座, 株式会社ひかり物流 → 株式会社ひかり運輸)

For the extraction results, I prepared the following 4 versions per document.

• Haiku output: The raw results extracted by Haiku (including Haiku's own 19 misreads)

• Error set A / Error set B: Haiku's output with 6 verification errors each injected

• Correct data: A version with only correct values (control data to observe behavior when there are zero errors)

I pass one document image and extraction results, and for each item ask via predicate (a question that returns a yes probability) whether "this value is exactly as printed in the image."
A probability below 0.5 is judged as an "error," and the comparison targets are clef and clef-flash (actual measurements from October 4, 2026).
Images were passed as original PNGs (1700×2200, approximately 200–300KB).
With Clef, the estimated token count exceeded the limit, so images were color-reduced to 32 colors; the image conditions differ from Clef's in this respect only.

Requests to the Decisions API take the following form (only one question is shown).
Comments are for explanatory purposes and are not included in the actual JSON sent.

{
  "model": "gpt-6-luna",
  "input": [{
    "role": "user",
    "content": [
      // Task premise (an OCR system read a Japanese invoice image and extracted keys and values; verify against the image) and extraction results
      {"type": "input_text", "text": "{\"task\": \"An OCR system read the attached Japanese invoice (請求書) image ...\", \"extracted\": {\"issuer_name\": \"株式会社さくらフーズ\", ...}}"},
      {"type": "input_image", "image_url": "data:image/png;base64,..."}
    ]
  }],
  "questions": [{
    "type": "predicate", // yes/no question; returns the probability of yes
    "name": "field__issuer_name",
    // Does the extracted value of issuer_name exactly match the string printed in the document image, character for character?
    "instructions": "Is the extracted value of \"issuer_name\" (\"株式会社さくらフーズ\") exactly what is printed in the document image, character for character?"
  }]
}

The invoice used in the verification. A stamp seal overlaps the issuer name and registration number

2. Results

2.1 Were Errors Detected?

Each model returns the probability that a value matches the image for each item.
Items where this probability is below 0.5 are judged as "errors," and counts were tallied from the following three perspectives.

• Detection of injected errors: How many of the 36 injected errors were judged as "errors"

• Detection of Haiku's errors: How many of the 19 misreads in Haiku's output were judged as "errors"

• False positives: How many items that were actually correct were incorrectly judged as "errors"

The Decisions API detected the most cases of both injected errors and Haiku's misreads among the three models.
On the other hand, false positives numbered 16, more than double Clef's 7.

The Decisions API detected all digit, similar character, and replacement errors, but like Clef, caught only 1 out of 3 full-width/half-width swap cases.
The 27 cases where the Decisions API made incorrect judgments (9 missed Haiku misreads, 2 missed injected errors, 16 false positives) are as follows.

9 of the Decisions API's 16 false positives occurred in the correct data version.
In particular, for the receipt quantity items where Haiku misread full-width as half-width, the model assigned 0.04 to the correct ３日 and 0.98 to Haiku's misread 3日, making the same inverted judgment as Clef regarding full-width and half-width.

2.2 Distribution of Probabilities

The spread of probabilities assigned to injected errors (36 cases), Haiku misreads (19 cases), and correct values (503 cases) is as follows.
The bottom 5% and top 5% are the values at the 5th and 95th percentile positions when probabilities are sorted in ascending order.

With one item per point, arranged by model as follows (points are scattered vertically to avoid overlap).
Note that the following three graphs were created by Claude based on the verification result data.

Decisions API: Injected errors cluster near 0.0, but Haiku misreads split toward both ends at 0.0 and 1.0

clef: Injected errors are scattered in the 0.0–0.3 range, and most Haiku misreads exceed 0.5

clef-flash: Distribution pattern similar to clef, but some correct values and some injected errors overlap across 0.5

With the Decisions API, the median probability for injected errors was 0.0 and for correct values was 0.99, with probabilities skewed toward the extremes just as in English.
The median for Haiku misreads was also 0.34, lower than Clef (0.691), but some missed items were assigned probabilities above 0.9.

Results when varying the threshold are as follows.

With the Decisions API at a threshold of 0.1, it still caught 33 injected errors with only 1 false positive.

2.3 Time and Cost

Requests were made once per document, sending one image and all questions for that document's items (30–63 questions) together.
With 3 documents × 4 extraction versions, 12 calls were made per model.
Time is measured from actual REST API calls from within Japan, and costs are converted at 150 yen per dollar.

Processing time for the Decisions API was about half that of clef, and cost was about three-quarters of clef's.
Input tokens were about 1.8 times that of Clef, but since the image conditions differ (the Decisions API used non-color-reduced images), a direct comparison is not straightforward.

3. Summary

Even with Japanese documents, the Decisions API detected 34 out of 36 injected errors and 10 out of 19 Haiku misreads, surpassing both Clef models on both counts.
Since avoiding missed errors is paramount when checking OCR results, I found it to be relatively strong even with Japanese documents.
Also, unlike Clef, there is no need to color-reduce images, and being able to pass images at their original quality is likely advantageous for reading fine characters.

On the other hand, false positives numbered 16, more than double Clef's 7, so it falls short of Clef in terms of false positive rate.
Full-width/half-width differences were also easy to miss, just as with Clef, and about half of Haiku's actual misreads passed through with a probability of 0.5 or higher.
The distinction between full-width and half-width is a concept that doesn't exist in English-speaking contexts, so models developed by American companies — not just the Decisions API or Clef — may tend to struggle with this difference regardless.