I showed document images (in Japanese) to Cloudflare's Clef and had it find errors in the OCR results

Hello, I'm Keima.

Setups that use a VLM to read document images and perform OCR are on the rise, but how to verify that the extracted values truly match the image remains a challenge.
Jev from TypeSafe, known as a decision model specialized for judgment, accepts text input only, so it could not directly look at the original document image to check whether "the extraction result matches the image."

On the other hand, Cloudflare Workers AI's Clef is a decision model that supports image input.
You can pass it a document image together with the VLM's extraction result, have it directly judge whether each value is "exactly as printed in the image," and obtain a confidence score.

In the previous verification with English documents, when I passed Clef the document images and the OCR extraction results, it detected all 36 deliberately injected errors with zero false positives.
This time, I tested whether it can find errors in the same way in Japanese documents, which include registration numbers, reduced tax rates, amounts in formal kanji numerals, and a mix of full-width and half-width characters.

1. Test conditions

First, I had Claude Haiku 4.5 extract fields from Japanese invoices, quotes, and receipts (3 documents, 149 fields in total).
In the English test, Haiku extracted every field correctly, but with Japanese, 19 misreadings occurred naturally, so I included them as detection targets as well.
In addition, I deliberately injected four patterns of errors (36 in total) for testing.

• Digit change (12): change just one digit of a number in an amount or ID (51,750 → 51,790, MQ-2026-0918 → MQ-2026-0916, etc.)

• Similar character substitution (9): swap in a similar-looking character (katakana "ka" → kanji "chikara (power)", kanji "kuchi (mouth)" → katakana "ro", the long-vowel mark → kanji "ichi (one)", etc.)

• Full-width/half-width swap (3): "20 meals" written with full-width digits → the same with half-width digits, half-width MN-2407 → its full-width version, etc.

• Replacement with a different value (12): replace with another value plausible for the document ("ordinary (savings)" → "current (checking)" for the account type, "Hikari Logistics Co., Ltd." → "Hikari Transport Co., Ltd.", etc.)

For each document, I prepared the following four extraction results.

• Haiku's raw output: the result Haiku extracted (no additional errors injected; includes Haiku's own 19 misreadings)

• With errors A / With errors B: Haiku's output with 6 deliberately injected errors each for testing

• Ground truth (no errors): a version consisting only of correct values (control data to observe behavior when there are no errors at all)

The judgment and verification conditions are as follows.

• Judgment method: pass one document image and the extraction result, and for each field ask with noul (a question that returns the probability of yes) whether "this value is exactly as printed in the image." A probability below 0.5 is judged as an "error"

• Models: clef (27B) and clef-flash (9B)

{
  "model": "clef",
  "images": ["data:image/png;base64,..."],
  // state: The premise for the judgment (an OCR system read a Japanese invoice image and extracted keys and values)
  "state": {
    "task": "An OCR system read the attached Japanese invoice (請求書) image ...",
    "extracted": { "issuer_name": "株式会社さくらフーズ" }
  },
  "questions": {
    "field.issuer_name": {
      "type": "noul", // A yes/no question. Returns the probability of yes
      // instructions: The judgment instruction (Does the extracted value of issuer_name match the string printed in the document image exactly, character for character?)
      "instructions": "Is the extracted value of \"issuer_name\" (\"株式会社さくらフーズ\") exactly what is printed in the document image, character for character?"
    }
  }
}

The invoice used in the test. A seal impression overlaps the issuer name and registration number

2. Results

2.1 Were the errors detected?

Clef returns, for each field, "the probability that the value matches the image."
Fields with a probability below 0.5 are judged as "errors," and I tallied the counts from the following three perspectives.

• Detection of injected errors: of the 36 injected errors, the number correctly judged as "errors"

• Detection of Haiku's errors: of Haiku's own 19 misreadings (the "Haiku's raw output" version), the number judged as "errors"

• Correct values judged as errors: the number of values that were actually correct but were judged as "errors" (false positives)

Clef detected most of the injected errors, but did not reach the full detection seen in the English test.
Also, it detected only 7 of Haiku's own 19 misreadings.

By error type, cases of overlooked full-width/half-width swaps and Haiku's own misreadings stood out.

The items judged incorrectly numbered 23 for clef and 41 for clef-flash.
Note that "missed" here means an item that had an error but was given 0.5 or higher, and "false positive" means an item that was a correct value but was given below 0.5.

The 7 correct values that clef wrongly judged as errors are as follows.

Of these, 5 (the 5 in the ground truth) were the correct values of fields that Haiku had misread.
For example, for the quantity on the receipt, it gave 0.34 to the correct full-width "3 days" and 0.89 to Haiku's misread half-width "3 days," judging full-width and half-width the wrong way around.
It is more reliable not to leave the full-width/half-width distinction to Clef, and instead to normalize the strings before comparison to absorb the difference.

2.2 How the probabilities were distributed

To determine where the threshold should be set, I summarized the spread of probabilities given to the injected errors (36), Haiku's misreadings (19), and the correct values (503).
The bottom 5% and top 5% are the values at the 5% and 95% positions when the probabilities are sorted from lowest to highest.

Unlike in the English test, the probabilities for errors and correct values overlapped significantly.
In particular, the median for Haiku's misreadings was high at 0.691, making them hard to distinguish from the correct values (median 0.874).

The results when changing the threshold are as follows.

Raising clef's threshold to 0.8 catches all the injected errors, but about 17% of the correct values (85 items) also get sent to human review.
Even so, it catches only 13 of Haiku's 19 misreadings, so it was difficult to eliminate the misses by threshold tuning alone.
clef-flash detects fewer errors and produces more false positives than clef at every threshold, so clef still seems better suited to Japanese documents.

2.3 Time and cost

There is one request to Clef per document.
Each request sends one image together with questions about all fields in that document (30 to 63 questions).
With 3 documents × 4 extraction results, each model was called 12 times.

The times are measured values from calling the REST API from within Japan, and costs are converted at $1 = ¥150.

3. Summary

For Japanese documents, clef was generally able to detect errors that turn a value into something clearly different (digits, replacements, different words), but it tended to miss full-width/half-width differences and dropped single characters, and its accuracy was not as high as with English.
In particular, it caught only 7 of the 19 misreadings that the OCR (Haiku) actually made.

Support for image input and low cost are major strengths of Clef.
If you entrust it with checking at this point, its accuracy is by no means high enough to be trusted unconditionally, and it does not yet seem to reach a level where a threshold alone lets you say "this is definitely fine."
Still, compared with having humans check every item from the start, it seems worth trying for narrowing down suspicious items and reducing the cost of human review to some extent.

References

• Models: clef | Cloudflare Workers AI docs