I Had OpenAI's Decisions API Classify Document Types and Compared It with Clef

Hello, I'm Keima.

I Had Cloudflare's Clef Classify Document Types | DevelopersIO I gave Cloudflare's Clef only document images and had it classify them into types such as invoice and quotation. It classified 27 of 28 correctly, and the one it got wrong could be identified by its low confidence.
OpenAI's Decisions API (model: gpt-6-luna) can also receive images and choose one from a set of choices, so it should be usable in a similar way.

So this time, I gave the Decisions API the same 28 images and the same question as in the Clef test, and compared accuracy, confidence, processing time, and cost with Clef.
I hope this is helpful when you are considering which to use for automatic document sorting.

1. Test conditions

choice is a question type that returns the selected choice, the probability of each choice, and a confidence (confidence).
The probabilities are values that sum to 1 across the 8 choices, and the confidence expresses on a 0 to 1 scale how decisively the answer was chosen.
In the Clef test, having a person check only the documents with a confidence below 0.8 was enough to catch every misclassification.

The request to the Decisions API looks like this (only 2 of the 8 choices are shown).
The comments are for explanation and are not included in the JSON actually sent.

{
  "model": "gpt-6-luna",
  "input": [{
    "role": "user",
    "content": [
      // Premise for the judgment (the attached image is one page of a business document; determine what kind of document it is)
      {"type": "input_text", "text": "The attached image is one page of a business document. Decide what kind of document it is."},
      {"type": "input_image", "image_url": "data:image/png;base64,..."}
    ]
  }],
  "questions": [{
    "type": "choice", // Choose one from the choices
    "name": "doc_type",
    // Which kind of business document is shown in the image?
    "instructions": "Which kind of business document is shown in the image?",
    "choices": [
      // invoice: a document requesting payment for goods or services already provided (with an amount due and a payment deadline)
      {"value": "invoice", "description": "Invoice / bill: the seller requests payment for goods or services already provided, with an amount due and a payment deadline."},
      // quotation: a document proposing prices before an order is placed (usually with a validity period)
      {"value": "quotation", "description": "Quotation / estimate: the seller proposes prices before an order is placed, usually with a validity period."}
    ]
  }]
}

2. Data

The data is the same as in the Clef test.
For each of the 7 types (invoice, quotation, receipt, delivery note, purchase order, contract, and resume), I prepared 2 documents each in Japanese and English, and the second document of each pair contains confusing elements such as the following.

• The title is missing, small, or an alternative name (Estimate, Packing Slip, Purchase Order, Work History)

• The body text mentions a different document (a quotation number in the body of an invoice, a purchase order and an invoice in the notes of a quotation)

• A resume filled with business-document terminology (work experience handling invoices and purchase orders)

An example with confusing elements. There is no title, and the body mentions a quotation number

3. Results

The characteristics of the 28 documents and each model's answer and confidence are as follows (errors are in bold).

3.1 Accuracy

The Decisions API classified all 28 documents correctly.
It also correctly classified the Japanese purchase order and the quotation with a small title, which were each missed by one of the two Clef models.

3.2 Confidence

The Decisions API's confidence was 1.0 for all 28 documents, with a probability of 1.0 for the chosen type and 0.0 for the other 7 types.
Clef, on the other hand, had confidence scattered between 0.7957 and 0.9707 even for correctly classified documents, and it dropped to 0.2384 and 0.5175 for the misclassified ones.

With these 28 documents, the Decisions API's confidence could not be used to identify "documents it was unsure about".
Since every answer was correct there was no real harm, but to set up an operation like the Clef test's "have a person check only documents with confidence below 0.8", we would need to verify with other data whether the Decisions API's confidence drops when it misclassifies.

3.3 Time and cost

Time is the measured round-trip time of REST API calls from Japan, and the conversion rate is 150 yen to the dollar.

For the same images, the Decisions API used about 1.8 times as many input tokens as Clef, but because its unit price is lower, the cost came to about three quarters of clef's.

With the Decisions API too, only input tokens are charged.

Input costs $0.10 per 1M tokens. You pay only for input tokens: there are no cache-read, cache-write, or output-token charges.

Source: Decisions API | OpenAI API

4. Summary

In this test of 28 documents, the Decisions API classified all of them correctly, including the Japanese documents that the two Clef models missed, and the cost also came in lower than clef.
Models like Jev and Clef are often English-centric and still have challenges with Japanese, but since the Decisions API is based on Luna, I got a strong sense that it is also good with Japanese business documents.

On the other hand, as a decision model to embed in a business system, the confidence value that shows how sure the model is will be the key to operational design.
If confidence is returned as 1.0 for every judgment, we cannot design an escalation that routes documents at risk of misclassification to human review, and running an automation system on its own remains a concern.
In real operation, what matters is not that it always outputs 1.0, but whether the value drops appropriately when the model is unsure.

The sample conditions I prepared this time may have been favorable, and the judgments may have been easy for the model.
I plan to investigate further with more confusing documents and more diverse samples, determine how much the confidence varies and where to set a suitable threshold, and then proceed with embedding it into a system.