I Showed a Document Image (in English) to OpenAI's Decisions API and Had It Find Errors in the OCR Results

Hello, I'm Keima.

When extracting values from documents with a VLM, even if you ask the VLM itself "Is this value correct?", it sometimes returns high confidence for a misread value.
In I showed Cloudflare's Clef an English document image and had it find OCR errors | DevelopersIO, I passed a document image and the extraction results to Cloudflare's Clef and had it judge "Is this value as shown in the image?", and it detected all 36 planted errors.

OpenAI's Decisions API (model: gpt-6-luna) can also accept an image as input and return the probability of yes for a question.
So this time, I passed the same three English documents and the same questions used in the Clef verification to the Decisions API, and compared the number of detected errors and the tendency of false positives with Clef.

1. Verification conditions

I used the same data as the Clef verification.
I injected three patterns of errors (12 each, 36 in total) into the results of extracting an English quote, invoice, and receipt (3 documents, 91 fields in total) with Claude Haiku 4.5.

• Digit change (12): change just one digit of an amount or date ($1,150.00 → $1,160.00, 09/18/2026 → 09/19/2026)

• Change to similar characters (12): replace with characters easily confused in OCR (0 → O, 1 → l, m → rn)

• Replacement with another value (12): replace with another word that is not unnatural in context (SAMPLE MART → FRESH MARKET, Service → Hardware)

The judgment method is also the same as in the Clef verification.
I pass one document image and the extraction results, and for each field ask with a predicate (a question that returns the probability of yes) whether "this value is exactly as printed in the image."
A probability below 0.5 is judged as an "error," and the comparison targets were clef and clef-flash (measured values from October 4, 2026).

The request to the Decisions API looks like this (only one question is excerpted).
The comments are for explanation and are not included in the JSON actually sent.

{
  "model": "gpt-6-luna",
  "input": [{
    "role": "user",
    "content": [
      // Premise for the judgment (an OCR system read the invoice image and extracted keys and values; verify against the image) and the extraction results
      {"type": "input_text", "text": "{\"task\": \"An OCR system read the attached invoice image and produced these key-value pairs. Verify them against the image.\", \"extracted\": {\"invoice_no\": \"INV-2026-0042\", ...}}"},
      {"type": "input_image", "image_url": "data:image/png;base64,..."}
    ]
  }],
  "questions": [{
    "type": "predicate", // A yes/no question. Returns the probability of yes
    "name": "field__invoice_no",
    // Does the extracted value of invoice_no match the string printed in the document image exactly, character for character?
    "instructions": "Is the extracted value of \"invoice_no\" (\"INV-2026-0042\") exactly what is printed in the document image, character for character?"
  }]
}

Clef's noul corresponds to predicate in the Decisions API.

The invoice used in the verification

2. Results

2.1 Could it find the errors?

Each model returns, for each field, the "probability that the value matches the image."
Fields whose probability was below 0.5 were judged as "errors," and I counted them from the following two perspectives.

• Error detection: the number of the 36 injected errors that were correctly judged as "errors"

• False positives: the number of values that were actually correct but were judged as "errors"

The Decisions API also detected all 36 injected errors.
On the other hand, there were 5 false positives where correct values were judged as errors, which is more than clef (0) and clef-flash (3).

The bill_to_attn on the invoice is printed as "Attn: Accounts Payable" in the image.
Since the extracted value did not include "Attn:", the Decisions API seems to have interpreted the instruction "exactly matches character for character" strictly and judged it as a mismatch.

The match probabilities for the 36 injected errors are as follows.
The lower the value, the more strongly the model judges that it "does not match the image."

2.2 How the probabilities separate

The spread of the probabilities output for the erroneous fields (36) and the correct fields (237) is as follows.
The bottom 5% and top 5% are the values at the lowest 5% and 95% when the probabilities are sorted in ascending order.

Treating each field as one point, plotting them per model gives the following.
Note that the following three charts were created by Claude based on the verification result data.

Decisions API: errors cluster around 0.0, and correct values cluster around 1.0

clef: errors are scattered in the 0.0–0.2 range, and correct values in the 0.6–0.9 range

clef-flash: the separation is close to clef, but some correct values fall below 0.5

The probabilities output by the Decisions API were heavily skewed toward the extremes.
Of the 36 errors, 31 were exactly 0.0, and of the 237 correct values, 123 were exactly 1.0, so it did not show the gradual spread into the 0.9 and 0.0 ranges seen with Clef.
On the other hand, the bottom 5% of correct values was 0.718, so some low values comparable to Clef also appeared.

The results when changing the threshold are as follows.

Since all the Decisions API's probabilities for errors were 0.08 or below, it could detect all 36 even when the threshold was lowered to 0.1.

2.3 Time and cost

I made one request per document, sending one image together with the questions for all fields in that document (21 to 38 questions).
The time is the measured value when calling the REST API from within Japan, and the cost is calculated at 150 yen per dollar.

The Decisions API had the fastest response among the three models tested, finishing in about one third of the time of clef.
The number of input tokens increased to about 1.9 times even with the same content as Clef, but because the per-token price is cheaper than clef ($0.24), the total cost came to about three quarters of clef's.

3. Summary

It looks like the Decisions API can take English document images as they are and check whether the OCR extraction results are correct.

However, the tendency for the output probabilities to stick to extremes near 0.0 and 1.0 is a bit of a concern for operation.