I Showed Document Images to Gemini Decision 1 and Had It Look for OCR Errors, Then Compared It with the Jev Decisions API

Hello, I'm Keima.

For OCR processing of documents on AWS, one approach is to host an OCR model on Amazon SageMaker to extract the text, then use a downstream LLM to structure keys and values while making use of the character recognition confidence scores.
However, keeping a model hosted on SageMaker around the clock tends to get expensive, and combining multiple services tends to make the architecture heavy.

Recently, the character recognition accuracy of VLMs themselves has improved, so a setup where a VLM alone handles the whole extraction has become practical.
However, the confidence values an LLM outputs tend to be self-reported, and it can give high scores even to misread values, so they have a weakness: they can't be fully trusted as a basis for judgment.
So if you pass the document image and the extraction results to a decision model and have it judge, per field, with a probability whether "this value matches the image," you can route only the suspicious fields to human review.

Previously, I tried the same thing with Cloudflare's Clef.
For details, please see I showed Cloudflare's Clef document images (Japanese) and had it find errors in OCR results | DevelopersIO.

In this article, I passed document images and OCR values to Gemini Decision 1, which Google released on November 11, 2026, and had it look for errors. I compared it under the same conditions with TypeSafe's Jev, which also supports image input, and OpenAI's Decisions API (the verification date was November 13, 2026). I'm curious how well Google's model, said to be strong at OCR, does.

1. Verification method

1.1 Documents

I used five real documents that can be published in an article.
There are two Japanese and three English documents.

From the left: receipts from AEON MaxValu Matsuyama store, CamelMart Akita Hachimantai store, and a Malaysian bookstore

From the left: the open access publishing fee invoice, and page 2 of the Form 1040 test sample entry

The source and license of each image are as shown in the table above. All were reduced in size on this blog and combined side by side into a single image.

1.2 OCR values and deliberately inserted errors

From each document I selected 5 to 6 fields and transcribed the correct values (for example, for the AEON receipt: store name, date, 8% tax, total, change, and so on).
I limited it to some fields rather than the whole document so that the amount of ground-truth data could be reliably checked by hand.

I prepared the following three patterns as verification data.

• No errors: the transcribed correct values as they are

• Error A and Error B: versions in which the values of two fields each were deliberately changed

There were three types of error insertion, and I created a total of 20 errors across the 5 documents.

• Swapped or missing digits (¥246 → ¥264, 36,852 → 36,582)

• Replacement with similar-looking characters (マックスバリュ → マックスバリユ, UCSFD00005 → UCSFDO0005)

• Replacement with a different value (the deposit amount ¥500 in the change field)

1.3 Questions

I passed the image together with the list of values to the model and asked yes/no questions like the following for each field.

{
  "field__total": {
    "type": "noul",
    "instructions": "抽出した「total」の値 \"¥264\" は、画像に印字されている内容と1文字も違わず同じか。空白の数の違いだけは同じとみなす。"
  }
}

I sent a total of 15 requests (5 documents × 3 patterns) to each of the three models, using the same images and the same questions.
Because image input for Jev is a preview feature, I added the header TypeSafe-Danger-Zone-Images: true and included the images in assets when sending.

1.4 Metrics

• Errors found: Of the 20 pre-inserted errors, the number judged to have a probability of being "the same as the image" below 0.8

• False positives: Of the 61 correct values, the number whose probability of being "the same as the image" ended up below 0.8 (the more there are, the more fields have to be reviewed by humans)

• Correct values judged correct: Of the 61 correct values, the number judged to have a probability of being "the same as the image" of 0.8 or higher (together with the false positives this totals 61. Answering "error" for everything would increase the errors found, so I also look at whether correct values were properly passed through)

For the judgment, I used the probabilities before rounding to two decimal places for display (so even a value shown as 0.90 in a table may be "below 0.9" at a threshold of 0.9).

2. Results

2.1 Errors found, false positives, and correct values judged correct

Gemini Decision 1 missed the fewest errors, but it also judged 24 correct values to be errors, and only 37 of the 61 correct values were judged correct.
The number of correct values judged correct was 54 for Decisions API and 50 for Jev.
Most of the false positive cases were ones where it output a probability of around 0.5 to 0.8 for a correct value.
For example, for the store name on the SROIE receipt (BOOK TA .K (TAMAN DAYA) SDN BHD), it output a probability of 0.26 to 0.38 even though the value was correct (this part of the print is faded and hard to read, and Jev and Decisions API also gave below 0.8 for it).

2.2 When changing the threshold

Since the probabilities from Gemini Decision 1 tended to come out lower even for correct values, I changed the judgment threshold and compared.

The numbers in the table represent errors found / false positives / correct values judged correct.

With Gemini Decision 1, raising the threshold to 0.8 or 0.9 drastically reduces the number of correct values judged correct (out of 61: 37 at 0.8, 6 at 0.9).
On the other hand, few errors slip through at any threshold, and at 0.6 to 0.8 it detected 19 of the 20 errors.
Looking at this graph, I felt that setting the threshold for Gemini Decision 1 to around 0.7 would give a good balance.

2.3 Missed errors

Here are the errors that at least one of the three models missed.
The numbers are the probability of being "the same as the image," and bold means 0.8 or higher at the threshold, that is, an error that was missed.

For the error in the tax return's name with one extra n, both Gemini Decision 1 and Jev missed it.
Decisions API, on the other hand, judged errors such as reordered digits and the difference between 0 and O as "the same" with a probability of 1.00.

3. Full results

Here are the results. Bold indicates items whose judgment was wrong under the threshold of 0.8.

4. Summary

I passed images of five documents and their OCR values to Gemini Decision 1, Jev, and Decisions API, and had them look for the deliberately inserted errors.

Gemini Decision 1 can also find errors in OCR values.
Also, even when given the same images, it used fewer input tokens than the other two models.
I got the same result when I tested a Gemini model before, so this may be a characteristic of Gemini-family models.
It feels like it could be used at a practical level for Japanese documents as well.

However, its sense of thresholds differs from Jev's.
When actually integrating it, I felt that you need to prepare test data close to your use case and decide which threshold to use.

References

• Images | TypeSafe Docs