developersIO
[TypeSafe] I tried to see if Jev can classify OCR-processed documents by page type

[TypeSafe] I tried to see if Jev can classify OCR-processed documents by page type

When I had Jev classify text from OCR-processed trade documents, it got 20 out of 20 pages correct simply by passing the immediately preceding page. It handles pages without titles and misreadings well, and since probability values can also be utilized, it seems possible to build a practical check flow that combines automated classification with human review.
TypeSafeOCRAmazon BedrockAmazon TextractJev
2026.09.21

This page has been translated by machine translation. View original

Hello, I'm Keema.

If you're handling documents with Jev, you need to first extract text using OCR before passing it along.

So this time, assuming OCR has already been completed, I tried passing that text to Jev to classify document types.
Trade documents often arrive with I/V and P/L scanned together into a single PDF.
Sometimes there are blank pages in between, or pages mixed in that don't have titles like "INVOICE."
I verified whether Jev could classify what each document was by passing such PDFs one page at a time.
For comparison purposes, I also passed the same input to Claude Sonnet 4.6 on Amazon Bedrock.

To give the conclusion upfront, when passing a single page at a time, 19 out of 20 pages were correct; when also passing the previous page together, all 20 pages were correct.
Writing classification rules in State did not change the number of correct answers, and OCR coordinate information was also unnecessary.
Also, the pages where Jev made incorrect classifications all showed low probability scores, so combining this with a workflow where humans only review low-probability pages could create a practical checking flow.

1. What I tried

1.1 Verification conditions

Item Content
Jev jev-latest (resolved to jev-1.13.0 at runtime)
Comparison Claude Sonnet 4.6 on Amazon Bedrock (global.anthropic.claude-sonnet-4-6, called via Converse API from Tokyo region)
Input Text assuming OCR has been completed. 5 PDFs, 20 pages total
Calling method 1 request per page. Executed once per condition
Execution environment macOS, Python 3.12, boto3 1.43.96 (used only for Sonnet 4.6)
Execution date September 21, 2026

For both Jev and Sonnet 4.6, I ran 4 conditions combining the following 2 factors.

  • Previous page: not passed / passed together

  • Classification rules: not written in State / written in State

Since Jev has the highest accuracy in English, the questions and rules are written in English.

1.2 Input data

I did not create PDF or image files; instead I directly prepared data equivalent to OCR results.
The original data uses the same format as Amazon Textract LINE blocks, with each read line having a string (Text), page number (Page), and position on the page (Geometry.BoundingBox).
What is passed to the model is only the string for each line.
The verification results when passing coordinates are summarized in 2.6.

All company names, numbers, and items are fictional.

In actual scanned trade documents, it is not uncommon for document names to not be written directly, or for OCR reading errors to occur.
Therefore, this time I intentionally included not only standard pages with clearly stated titles, but also the following difficult cases commonly encountered in practice when creating the data.

  • Documents without titles (where you need to identify by columns or items)

  • Continuation pages from page 2 onward (without headers, containing only line items and total figures)

  • Cover sheets and completely blank pages

  • OCR misreads (such as "COMMERCIAL INVOICE" being read as "C0MMERC1AL INV0ICE")

  • Noise such as stamps, handwritten notes, and junk characters

The 5 PDFs prepared are as follows.

PDF Page Content
pdf-1 p.1 Invoice (with title)
p.2 Invoice continuation page (no header, only continued line items and total)
pdf-2 p.1 Invoice
p.2 Packing List
p.3 Packing List continuation page (only continued line items and total)
p.4 B/L
p.5 Certificate of Origin
p.6 Insurance Policy
pdf-3 p.1 FAX cover sheet
p.2 Invoice
p.3 Completely blank page (no lines)
p.4 Packing List
pdf-4 p.1 Invoice (no title)
p.2 Packing List (alternative name "WEIGHT AND MEASUREMENT LIST")
p.3 B/L (no title)
p.4 Certificate of Origin (no title)
pdf-5 p.1 Invoice (title misread as "C0MMERC1AL INV0ICE", stamp mixed in)
p.2 Nearly blank (only 4 junk characters)
p.3 Packing List (title misread as "PACK1NG L1ST", handwritten notes mixed in)
p.4 Packing List continuation page (with noise)

All 20 pages of content are included in the collapsed sections below.
The text for each line passed to the model is listed in order from top to bottom.

All pages of pdf-1 (click to expand)

p.1: Invoice page 1 (with title) (correct answer is invoice)

COMMERCIAL INVOICE
SAMPLE EXPORT CO., LTD.
No. INV-2026-0901
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 01, 2026
Sold to:
Payment: T/T 30 DAYS
EXAMPLE IMPORTS PTE. LTD.
Terms: CIF SINGAPORE
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Description
Quantity
Unit Price
Amount
CERAMIC DINNER PLATE 27CM WHITE
1,200 PCS
USD 3.20
USD 3,840.00
CERAMIC SOUP BOWL 15CM WHITE
1,500 PCS
USD 2.10
USD 3,150.00
CERAMIC MUG 350ML BLUE
2,000 PCS
USD 1.80
USD 3,600.00
Continued on next page

p.2: Invoice page 2 (only continued line items and total) (correct answer is invoice)

STAINLESS BOTTLE 500ML SILVER
800 PCS
USD 6.50
USD 5,200.00
STAINLESS BOTTLE 750ML BLACK
600 PCS
USD 7.90
USD 4,740.00
TOTAL
USD 20,530.00
SAY US DOLLARS TWENTY THOUSAND FIVE HUNDRED THIRTY ONLY
SAMPLE EXPORT CO., LTD.
Authorized Signature
All pages of pdf-2 (click to expand)

p.1: Invoice (with title, single page) (correct answer is invoice)

COMMERCIAL INVOICE
SAMPLE EXPORT CO., LTD.
No. INV-2026-0902
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 01, 2026
Sold to:
Payment: T/T 30 DAYS
EXAMPLE IMPORTS PTE. LTD.
Terms: CIF SINGAPORE
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Description
Quantity
Unit Price
Amount
BAMBOO CUTTING BOARD 30X20CM
900 PCS
USD 4.40
USD 3,960.00
BAMBOO SERVING TRAY 40X28CM
400 PCS
USD 8.25
USD 3,300.00
COTTON KITCHEN TOWEL 3-PACK
1,000 SETS
USD 2.75
USD 2,750.00
TOTAL
USD 10,010.00
SAMPLE EXPORT CO., LTD.
Authorized Signature

p.2: Packing List page 1 (with title) (correct answer is packing_list)

PACKING LIST
SAMPLE EXPORT CO., LTD.
Ref. No. INV-2026-0901
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 01, 2026
Consigned to:
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Marks: EXAMPLE / SINGAPORE / C/NO. 1-UP / MADE IN JAPAN
Case No.
Description
Quantity
N.W. (KG)
G.W. (KG)
M3
1-10
CERAMIC DINNER PLATE 27CM WHITE
1,200 PCS
840.0
910.0
2.40
11-20
CERAMIC SOUP BOWL 15CM WHITE
1,500 PCS
600.0
660.0
2.10
21-30
CERAMIC MUG 350ML BLUE
2,000 PCS
640.0
700.0
2.60
Continued on next page

p.3: Packing List page 2 (only continued line items and total) (correct answer is packing_list)

31-38
STAINLESS BOTTLE 500ML SILVER
800 PCS
240.0
288.0
1.92
39-44
STAINLESS BOTTLE 750ML BLACK
600 PCS
228.0
270.0
1.68
TOTAL: 44 CARTONS
6,100 PCS
2,548.0
2,828.0
10.70
SAMPLE EXPORT CO., LTD.
Authorized Signature

p.4: B/L (with title) (correct answer is bill_of_lading)

Shipper
BILL OF LADING
SAMPLE EXPORT CO., LTD.
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
No. SMPL-YOK-260901
Consignee
TO ORDER OF EXAMPLE BANK LTD.
Notify Party
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
Ocean Vessel / Voy. No.
Port of Loading
Port of Discharge
MV SAMPLE STAR V.012
YOKOHAMA, JAPAN
SINGAPORE
Container No. / Seal No.
No. of Pkgs
Description of Goods
Gross Weight
SMPU 123456-7 / 00112233
44 CARTONS
CERAMIC TABLEWARE AND
2,828.0 KGS
STAINLESS BOTTLES
SHIPPER'S LOAD, COUNT AND SEAL
FREIGHT PREPAID
SHIPPED ON BOARD
Date: SEP. 05, 2026
Number of Original B(s)/L: THREE (3)
RECEIVED by the Carrier the goods in apparent good order and
condition. One original must be surrendered duly endorsed in
exchange for the goods.
SAMPLE OCEAN LINE CO., LTD. AS CARRIER

p.5: Certificate of Origin (with title) (correct answer is certificate_of_origin)

CERTIFICATE OF ORIGIN
1. Exporter (Name, address, country)
Reference No. CO-2026-004512
SAMPLE EXPORT CO., LTD.
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
2. Importer (Name, address, country)
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
3. Transport details
FROM YOKOHAMA TO SINGAPORE BY SEA, ON OR ABOUT SEP. 05, 2026
4. Marks, numbers, number and kind of packages; description of goods
44 CARTONS OF CERAMIC TABLEWARE AND STAINLESS BOTTLES
HS CODE 6911.10 / 7323.93
5. Country of origin
JAPAN
6. Declaration by the exporter
7. Certification
The undersigned declares that the above details are correct and
It is hereby certified that the declaration by the exporter
that all the goods were produced in JAPAN.
is correct.
EXAMPLE CHAMBER OF COMMERCE AND INDUSTRY
YOKOHAMA, SEP. 03, 2026

p.6: Insurance Policy (with title) (correct answer is insurance_policy)

MARINE CARGO INSURANCE POLICY
Assured
No. MC-26-0098431
SAMPLE EXPORT CO., LTD.
Amount Insured
USD 22,583.00 (110% OF CIF VALUE)
Ship or Vessel
Sailing on or about
MV SAMPLE STAR V.012
SEP. 05, 2026
From
To
YOKOHAMA, JAPAN
SINGAPORE
Subject-matter Insured
44 CARTONS OF CERAMIC TABLEWARE AND STAINLESS BOTTLES
Conditions
INSTITUTE CARGO CLAUSES (A)
INSTITUTE WAR CLAUSES (CARGO)
INSTITUTE STRIKES CLAUSES (CARGO)
Claims, if any, payable at SINGAPORE in USD
Claim agent: EXAMPLE SURVEYORS PTE. LTD.
SAMPLE MARINE INSURANCE CO., LTD.
Authorized Signatory
All pages of pdf-3 (click to expand)

p.1: FAX cover sheet (correct answer is other)

FAX TRANSMISSION
To: EXAMPLE IMPORTS PTE. LTD. / Purchasing Dept.
From: SAMPLE EXPORT CO., LTD. / Export Section
Date: SEP. 02, 2026
Pages: 4 (including this page)
Dear Sirs,
Please find attached the shipping documents for your order.
Kindly confirm receipt by return.
Best regards,

p.2: Invoice (with title, single page) (correct answer is invoice)

INVOICE
SAMPLE EXPORT CO., LTD.
No. INV-2026-0903
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 02, 2026
Sold to:
Payment: T/T 30 DAYS
EXAMPLE IMPORTS PTE. LTD.
Terms: CIF SINGAPORE
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Description
Quantity
Unit Price
Amount
CERAMIC DINNER PLATE 27CM WHITE
1,200 PCS
USD 3.20
USD 3,840.00
CERAMIC SOUP BOWL 15CM WHITE
1,500 PCS
USD 2.10
USD 3,150.00
TOTAL
USD 6,990.00

p.3: Completely blank page (not a single line) (correct answer is blank)

(no lines)

p.4: Packing List (with title, single page) (correct answer is packing_list)

PACKING LIST
SAMPLE EXPORT CO., LTD.
Ref. No. INV-2026-0903
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 02, 2026
Consigned to:
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Marks: EXAMPLE / SINGAPORE / C/NO. 1-UP / MADE IN JAPAN
Case No.
Description
Quantity
N.W. (KG)
G.W. (KG)
M3
1-10
CERAMIC DINNER PLATE 27CM WHITE
1,200 PCS
840.0
910.0
2.40
11-20
CERAMIC SOUP BOWL 15CM WHITE
1,500 PCS
600.0
660.0
2.10
TOTAL: 20 CARTONS
1,570.0
All pages of pdf-4 (click to expand)

p.1: Invoice (no title, identified by unit price and amount columns) (correct answer is invoice)

SAMPLE EXPORT CO., LTD.
No. SE-260904
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 03, 2026
Sold to:
Payment: T/T 30 DAYS
EXAMPLE IMPORTS PTE. LTD.
Terms: CIF SINGAPORE
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Description
Quantity
Unit Price
Amount
BAMBOO CUTTING BOARD 30X20CM
900 PCS
USD 4.40
USD 3,960.00
BAMBOO SERVING TRAY 40X28CM
400 PCS
USD 8.25
USD 3,300.00
COTTON KITCHEN TOWEL 3-PACK
1,000 SETS
USD 2.75
USD 2,750.00
TOTAL
USD 10,010.00

p.2: Alternative name for Packing List (WEIGHT AND MEASUREMENT LIST) (correct answer is packing_list)

WEIGHT AND MEASUREMENT LIST
SAMPLE EXPORT CO., LTD.
Ref. No. SE-260904
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. 03, 2026
Consigned to:
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Marks: EXAMPLE / SINGAPORE / C/NO. 1-UP / MADE IN JAPAN
Case No.
Description
Quantity
N.W. (KG)
G.W. (KG)
M3
31-38
STAINLESS BOTTLE 500ML SILVER
800 PCS
240.0
288.0
1.92
39-44
STAINLESS BOTTLE 750ML BLACK
600 PCS
228.0
270.0
1.68
TOTAL: 14 CARTONS
558.0

p.3: B/L (no title) (correct answer is bill_of_lading)

Shipper
SAMPLE EXPORT CO., LTD.
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
No. SMPL-YOK-260901
Consignee
TO ORDER OF EXAMPLE BANK LTD.
Notify Party
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
Ocean Vessel / Voy. No.
Port of Loading
Port of Discharge
MV SAMPLE STAR V.012
YOKOHAMA, JAPAN
SINGAPORE
Container No. / Seal No.
No. of Pkgs
Description of Goods
Gross Weight
SMPU 123456-7 / 00112233
44 CARTONS
CERAMIC TABLEWARE AND
2,828.0 KGS
STAINLESS BOTTLES
SHIPPER'S LOAD, COUNT AND SEAL
FREIGHT PREPAID
SHIPPED ON BOARD
Date: SEP. 05, 2026
Number of Original B(s)/L: THREE (3)
RECEIVED by the Carrier the goods in apparent good order and
condition. One original must be surrendered duly endorsed in
exchange for the goods.
SAMPLE OCEAN LINE CO., LTD. AS CARRIER

p.4: Certificate of Origin (no title) (correct answer is certificate_of_origin)

1. Exporter (Name, address, country)
Reference No. CO-2026-004512
SAMPLE EXPORT CO., LTD.
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
2. Importer (Name, address, country)
EXAMPLE IMPORTS PTE. LTD.
45 SAMPLE ROAD, SINGAPORE 000000
3. Transport details
FROM YOKOHAMA TO SINGAPORE BY SEA, ON OR ABOUT SEP. 05, 2026
4. Marks, numbers, number and kind of packages; description of goods
44 CARTONS OF CERAMIC TABLEWARE AND STAINLESS BOTTLES
HS CODE 6911.10 / 7323.93
5. Country of origin
JAPAN
6. Declaration by the exporter
7. Certification
The undersigned declares that the above details are correct and
It is hereby certified that the declaration by the exporter
that all the goods were produced in JAPAN.
is correct.
EXAMPLE CHAMBER OF COMMERCE AND INDUSTRY
YOKOHAMA, SEP. 03, 2026
All pages of pdf-5 (click to expand)

p.1: Invoice (title misread, stamp text mixed in) (correct answer is invoice)

|
C0MMERC1AL INV0ICE
SAMPLE EXPORT CO., LTD.
No. INV-2O26-O9O5
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. O4, 2026
Sold to:
Payment: T/T 30 DAYS
EXAMPLE IMPORTS PTE. LTD.
Terms: CIF SINGAPORE
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Description
Quantity
Unit Price
Amount
CERAMIC DINNER PLATE 27CM WHITE
1,200 PCS
USD 3.20
USD 3,840.00
CERAMIC SOUP BOWL 15CM WHITE
1,500 PCS
USD 2.10
USD 3,150.00
T0TAL
USD 6,99O.OO
RECE1VED
26. 9. -7

p.2: Nearly blank (only 4 junk characters) (correct answer is blank)

l
. _
|
- 2

p.3: Packing List (title misread, handwritten notes mixed in) (correct answer is packing_list)

PACK1NG L1ST
SAMPLE EXPORT CO., LTD.
Ref. No. INV-2O26-O9O5
1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN
Date: SEP. O4, 2026
Consigned to:
EXAMPLE IMPORTS PTE. LTD.
OK checked
45 SAMPLE ROAD, SINGAPORE 000000
From: YOKOHAMA, JAPAN
To: SINGAPORE
Per: MV SAMPLE STAR V.012
Marks: EXAMPLE / SINGAPORE / C/NO. 1-UP / MADE IN JAPAN
Case No.
Description
Quantity
N.W. (KG)
G.W. (KG)
M3
1-10
CERAMIC DINNER PLATE 27CM WHITE
1,200 PCS
840.0
910.0
2.40
11-20
CERAMIC SOUP BOWL 15CM WHITE
1,500 PCS
600.0
660.0
2.10
Cont'd
| |

p.4: Packing List continuation page (with noise) (correct answer is packing_list)

21-30
CERAMIC MUG 350ML BLUE
2,000 PCS
640.0
700.0
2.60
T0TAL: 3O CART0NS
2,27O.O
_ .
l

For example, page 3 of pdf-2 (the Packing List continuation page) is passed to the model as follows.
The column headers are absent, and only numbers are listed.

["31-38", "STAINLESS BOTTLE 500ML SILVER", "800 PCS", "240.0", "288.0", "1.92",
 "39-44", "STAINLESS BOTTLE 750ML BLACK", "600 PCS", "228.0", "270.0", "1.68",
 "TOTAL: 44 CARTONS", "6,100 PCS", "2,548.0", "2,828.0", "10.70",
 "SAMPLE EXPORT CO., LTD.", "Authorized Signature"]

1.3 State and Questions

In State, put the lines of the page you want to classify into page.
Add previous_page for conditions where you pass the previous page, and classification_guide for conditions where you write the rules.
For the first page of a PDF, set previous_page to null.

{
  "classification_guide": "<Only for conditions where rules are written. How to tell document types apart>",
  "previous_page": ["<Only for conditions where the previous page is passed. Each line of the previous page>"],
  "page": ["<Each line of the page to be classified>"]
}

As an actual example of State, we show the one used when classifying page 3 of pdf-2 (a continuation page of a packing list), passing the previous page and also writing the rules.
classification_guide contains how to tell document types apart, previous_page contains page 2 (the first page of the packing list), and page contains each line of page 3.
For conditions without rules, remove classification_guide from this State, and for conditions without passing the previous page, remove previous_page.

Actual example of State (click to expand)
{
  "classification_guide": "How to tell the document types apart. Titles are often missing, renamed, or misread by OCR, so rely on the fields, not on the title.\n- invoice: a request for payment. Has a unit price column and an amount column, a total amount in a currency, and usually payment terms or trade terms such as CIF or FOB.\n- packing_list: describes how the goods are packed. Has case or carton numbers, net weight, gross weight and measurement (M3). It has no unit price and no amount. It may be titled \"Weight and Measurement List\" or similar.\n- bill_of_lading: issued by a carrier. Has Shipper, Consignee, Notify Party, vessel and voyage, port of loading, port of discharge, container and seal numbers, \"shipped on board\", the number of original B/Ls, and a clause about surrendering an original.\n- certificate_of_origin: numbered boxes for exporter, importer, transport details and country of origin, a declaration by the exporter, and a certification by a chamber of commerce or authority.\n- insurance_policy: issued by an insurer. Has the assured, amount insured, conditions such as Institute Cargo Clauses, and where claims are payable.\n- blank: no lines at all, or only a few stray characters such as \"|\", \"_\", \".\" or a single letter left by stamps and ruled lines. A page with a few meaningless fragments is still blank.\n- other: a readable page that is none of the above, such as a fax cover sheet or a printed e-mail.\nContinuation pages: the second and later pages of an invoice or packing list often have no title and no header. They start directly with table rows and end with a total or a signature. Decide their type from the columns (prices and amounts mean invoice; weights and measurements mean packing_list).\nOCR noise: \"0\" and \"O\", \"1\" and \"I\" or \"L\" are often confused. Stamps such as \"RECEIVED\" and handwritten notes may be mixed in. Ignore them.",
  "previous_page": [
    "PACKING LIST",
    "SAMPLE EXPORT CO., LTD.",
    "Ref. No. INV-2026-0901",
    "1-2-3 EXAMPLE-CHO, YOKOHAMA, JAPAN",
    "Date: SEP. 01, 2026",
    "Consigned to:",
    "EXAMPLE IMPORTS PTE. LTD.",
    "45 SAMPLE ROAD, SINGAPORE 000000",
    "From: YOKOHAMA, JAPAN",
    "To: SINGAPORE",
    "Per: MV SAMPLE STAR V.012",
    "Marks: EXAMPLE / SINGAPORE / C/NO. 1-UP / MADE IN JAPAN",
    "Case No.",
    "Description",
    "Quantity",
    "N.W. (KG)",
    "G.W. (KG)",
    "M3",
    "1-10",
    "CERAMIC DINNER PLATE 27CM WHITE",
    "1,200 PCS",
    "840.0",
    "910.0",
    "2.40",
    "11-20",
    "CERAMIC SOUP BOWL 15CM WHITE",
    "1,500 PCS",
    "600.0",
    "660.0",
    "2.10",
    "21-30",
    "CERAMIC MUG 350ML BLUE",
    "2,000 PCS",
    "640.0",
    "700.0",
    "2.60",
    "Continued on next page"
  ],
  "page": [
    "31-38",
    "STAINLESS BOTTLE 500ML SILVER",
    "800 PCS",
    "240.0",
    "288.0",
    "1.92",
    "39-44",
    "STAINLESS BOTTLE 750ML BLACK",
    "600 PCS",
    "228.0",
    "270.0",
    "1.68",
    "TOTAL: 44 CARTONS",
    "6,100 PCS",
    "2,548.0",
    "2,828.0",
    "10.70",
    "SAMPLE EXPORT CO., LTD.",
    "Authorized Signature"
  ]
}

Questions consists of only one Choice question (a question where one answer is selected from the options), asking the model to choose the document type from 7 options.
The descriptions of the options were kept to just the document names, so as not to serve as hints in conditions where rules are not written.

{
  "doc_type": {
    "type": "choice",
    "instructions": "`page` holds the OCR result of one page of a scanned PDF, line by line from top to bottom. Decide which type of trade document this page belongs to.",
    "criteria": {
      "invoice": "Commercial invoice.",
      "packing_list": "Packing list.",
      "bill_of_lading": "Bill of lading.",
      "certificate_of_origin": "Certificate of origin.",
      "insurance_policy": "Cargo insurance policy or certificate.",
      "blank": "Blank page.",
      "other": "None of the above."
    }
  }
}

For conditions where the previous page is passed, the following sentence is appended to the end of instructions.

`previous_page` is the page immediately before it in the same PDF (null for the first page). Use it only as context, for example to recognise a continuation page that has no title or header. Classify `page`, not `previous_page`.

# (Japanese translation)
# `previous_page` is the immediately preceding page in the same PDF (null for the first page).
# Use it only as context, such as for recognizing a continuation page that has no title or header.
# Classify `page`, not `previous_page`.
How to tell document types apart as written in State (click to expand)
How to tell the document types apart. Titles are often missing, renamed, or misread by OCR, so rely on the fields, not on the title.
- invoice: a request for payment. Has a unit price column and an amount column, a total amount in a currency, and usually payment terms or trade terms such as CIF or FOB.
- packing_list: describes how the goods are packed. Has case or carton numbers, net weight, gross weight and measurement (M3). It has no unit price and no amount. It may be titled "Weight and Measurement List" or similar.
- bill_of_lading: issued by a carrier. Has Shipper, Consignee, Notify Party, vessel and voyage, port of loading, port of discharge, container and seal numbers, "shipped on board", the number of original B/Ls, and a clause about surrendering an original.
- certificate_of_origin: numbered boxes for exporter, importer, transport details and country of origin, a declaration by the exporter, and a certification by a chamber of commerce or authority.
- insurance_policy: issued by an insurer. Has the assured, amount insured, conditions such as Institute Cargo Clauses, and where claims are payable.
- blank: no lines at all, or only a few stray characters such as "|", "_", "." or a single letter left by stamps and ruled lines. A page with a few meaningless fragments is still blank.
- other: a readable page that is none of the above, such as a fax cover sheet or a printed e-mail.
Continuation pages: the second and later pages of an invoice or packing list often have no title and no header. They start directly with table rows and end with a total or a signature. Decide their type from the columns (prices and amounts mean invoice; weights and measurements mean packing_list).
OCR noise: "0" and "O", "1" and "I" or "L" are often confused. Stamps such as "RECEIVED" and handwritten notes may be mixed in. Ignore them.

# (Japanese translation)
# How to tell document types apart. Titles are often missing, renamed, or misread by OCR, so judge based on the fields rather than the title.
# - invoice: a payment request. Has a unit price column and an amount column, a total amount with a currency unit, and usually includes trade terms or payment terms such as CIF or FOB.
# - packing_list: shows the packing condition of the cargo. Has case numbers or carton numbers, net weight, gross weight, and volume (M3). Has no unit price or amount, and may have an alternative name such as "Weight and Measurement List".
# - bill_of_lading: issued by a carrier. Has shipper, consignee, notify party, vessel name and voyage number, port of loading, port of discharge, container and seal numbers, "shipped on board" notation, the number of original B/Ls issued, and a clause regarding submission of the original.
# - certificate_of_origin: has numbered boxes for exporter, importer, transport details, and country of origin, a declaration by the exporter, and a certification by a chamber of commerce or public authority.
# - insurance_policy: issued by an insurance company. Has the insured, insured amount, insurance conditions such as Institute Cargo Clauses, and the place where insurance claims are payable.
# - blank: no lines at all, or only garbage characters such as "|", "_", "." or single-character fragments from stamps or ruled lines. A page with only a few meaningless fragments is also considered blank.
# - other: a readable page that does not fall into any of the above categories (e.g., a fax cover sheet or a printed email).
# Continuation pages: the second and later pages of invoices or packing lists often have no title or header. They start directly with table rows and end with a total or signature. Determine the type from the column contents (prices and amounts mean invoice; weights and volumes mean packing_list).
# OCR noise: "0" and "O", "1" and "I" or "L" are easily confused. Stamps such as "RECEIVED" or handwritten notes may be mixed in, but ignore them.

For Sonnet 4.6, the same State was passed as a user message in JSON format.
The question text and options were specified in the system prompt, and the answer was received as a tool call argument (doc_type).

2. Results

2.1 Summary of Results

Costs are converted at 150 yen per dollar.
Token counts and costs are totals for 20 pages.

Model Previous page Rules Correct type Needs review (probability < 0.8) Median latency Input tokens Output tokens Estimated cost for 20 pages
Jev Not passed None 19/20 2 pages ~0.65 sec (649ms) 13,827 1,497 $0.0006 (~0.09 yen)
Jev Not passed With rules 19/20 2 pages ~0.65 sec (654ms) 22,427 1,496 $0.0009 (~0.14 yen)
Jev Passed None 20/20 1 page ~0.63 sec (630ms) 19,070 1,497 $0.0008 (~0.12 yen)
Jev Passed With rules 20/20 0 pages ~0.66 sec (660ms) 27,670 1,497 $0.0012 (~0.17 yen)
Sonnet 4.6 Not passed None 20/20 - ~1.18 sec (1,179ms) 21,062 738 $0.0743 (~11.14 yen)
Sonnet 4.6 Not passed With rules 20/20 - ~1.15 sec (1,146ms) 30,622 738 $0.1029 (~15.44 yen)
Sonnet 4.6 Passed None 20/20 - ~1.15 sec (1,150ms) 25,693 738 $0.0881 (~13.22 yen)
Sonnet 4.6 Passed With rules 20/20 - ~1.12 sec (1,116ms) 35,253 738 $0.1168 (~17.52 yen)

Costs were calculated by multiplying the actual measured token counts by the published prices.
Jev costs $0.042 per million input tokens, and output tokens are free (Models | TypeSafe Docs).
Sonnet 4.6 costs $3.00 per million input tokens and $15.00 per million output tokens, and the costs in the table include the output token portion as well (Amazon Bedrock Pricing | AWS).

"Needs review" is the number of pages where the probability returned by Jev for the document type was less than 0.8.
This was counted assuming an operational workflow where pages with a probability of 0.8 or higher are processed automatically, while pages below 0.8 are flagged for human review.

2.2 Answers and Probabilities per Page

This is a list of how Jev answered each of the 20 pages.
The values show the probability of the type selected by Jev, rounded to two decimal places.
For answers that differed from the correct answer, the type selected by Jev is also noted.
"Single" refers to conditions where only one page was passed, and "With previous" refers to conditions where the previous page was also passed together.

Page Content Correct answer Single · no rules Single · with rules With previous · no rules With previous · with rules
pdf-1 p.1 Invoice invoice 1.00 1.00 1.00 1.00
pdf-1 p.2 Invoice continuation invoice 0.99 1.00 1.00 1.00
pdf-2 p.1 Invoice invoice 1.00 1.00 1.00 1.00
pdf-2 p.2 Packing list packing_list 1.00 1.00 1.00 1.00
pdf-2 p.3 Packing list continuation packing_list 0.84 invoice 0.71 needs review 1.00 1.00
pdf-2 p.4 B/L bill_of_lading 1.00 1.00 1.00 1.00
pdf-2 p.5 Certificate of origin certificate_of_origin 1.00 1.00 1.00 1.00
pdf-2 p.6 Insurance policy insurance_policy 1.00 1.00 1.00 1.00
pdf-3 p.1 Fax cover sheet other 1.00 1.00 1.00 1.00
pdf-3 p.2 Invoice invoice 1.00 1.00 1.00 1.00
pdf-3 p.3 Completely blank blank 1.00 1.00 0.97 0.96
pdf-3 p.4 Packing list packing_list 1.00 1.00 1.00 1.00
pdf-4 p.1 Invoice (no title) invoice 0.99 1.00 0.99 1.00
pdf-4 p.2 Packing list (alternative name) packing_list 1.00 1.00 1.00 1.00
pdf-4 p.3 B/L (no title) bill_of_lading 1.00 1.00 1.00 1.00
pdf-4 p.4 Certificate of origin (no title) certificate_of_origin 1.00 1.00 1.00 1.00
pdf-5 p.1 Invoice (misread) invoice 1.00 1.00 1.00 1.00
pdf-5 p.2 Garbage characters only blank other 0.49 needs review 0.99 0.50 needs review 0.83
pdf-5 p.3 Packing list (misread) packing_list 1.00 1.00 1.00 1.00
pdf-5 p.4 Packing list continuation (with noise) packing_list 0.63 needs review 0.70 needs review 1.00 1.00

Both of the 2 pages where Jev made incorrect judgments had probabilities below 0.8 and were flagged as needing review.
For pages that passed the 0.8 threshold and were routed to automatic processing, all answers were correct across all 4 conditions.
With the current verification data, setting the threshold at 0.8 ensured that no incorrect judgments slipped through without human review.

2.3 Correct classification even for pages without titles or with misreadings

From here, we explain Jev's verification results based on the table in 2.2.
Jev correctly classified invoices, B/Ls, and certificates of origin without titles, packing lists with alternative names, and pages with misread titles, under all 4 conditions.
For these pages, the probabilities returned by Jev were all 0.99 or higher.
Jev appears to be able to identify document types from the arrangement of fields such as unit price and amount columns, and weight and volume columns.

2.4 With a single page, continuation pages without headers and garbage character pages are unstable

In the two conditions where only a single page was passed, Jev made one incorrect judgment each.
In the condition without rules, Jev classified the page with only garbage characters (pdf-5 p.2) as other.
In the condition with rules, Jev classified the packing list continuation page (pdf-2 p.3) as invoice.
The only pages where Jev's probability dropped below 0.8 were these two pages and one other continuation page (pdf-5 p.4).

As shown in 1.2, pdf-2 p.3 has no column headers and only numbers lined up.
It is likely that there was insufficient material to distinguish from a single page whether the numbers represented weights or amounts, causing Jev's probability to be split.

2.5 All pages correct when passing the previous page together

When passing the previous page together as previous_page, Jev correctly classified all 20 pages regardless of whether rules were present.
Even for continuation pages where the probability was split when only a single page was passed, Jev's probability rose to 0.99 or higher.
Jev appears to have been able to determine that a page was a continuation by referencing the headers and column names on the previous page.
Passing the previous page increases Jev's input token count by approximately 1.2 to 1.4 times, but the cost for Jev stays at around $0.0008 to $0.0012 (approximately 0.12 to 0.17 yen) for 20 pages.
If using Jev to handle PDFs that include continuation pages, passing the previous page as well is the safest approach.

2.6 OCR coordinates do not need to be passed

We also tested a condition where Textract-format LINE blocks were passed along with their coordinates (BoundingBox).
When passing only a single page, the number of correct answers was 20/20 without rules and 19/20 with rules, which is almost the same as when passing only the text lines.
The incorrectly classified page was also the same packing list continuation page.
On the other hand, the number of input tokens ballooned from 13,827 to 49,931 — approximately 3.6 times more — in the case without rules.
For the document type classification task in this study, there was no discernible benefit to passing the coordinates as well.

2.7 Speed and Cost

The median latency per page was approximately 0.65 seconds (630–660ms) for Jev and approximately 1.15 seconds (1,116–1,179ms) for Sonnet 4.6.
The estimated cost for 20 pages under the condition of passing the previous page without writing rules was $0.0008 (approximately 0.12 yen) for Jev and $0.0881 (approximately 13.22 yen) for Sonnet 4.6, a difference of approximately 110 times.
A simple estimate for 10,000 pages under the same conditions gives approximately 60 yen for Jev and approximately 6,600 yen for Sonnet 4.6.

3. Summary

Simply passing OCR-processed text as a sequence of lines to Jev enabled stable document type classification even for pages without titles or pages containing misreadings.
There was no need to write detailed classification rules into State, and coordinate information was also unnecessary.
Particularly effective was "passing the previous page together."
Continuation pages without headers showed split judgments with Jev when passed alone, but correct classification became possible with the addition of information from the previous page.

In terms of accuracy alone, Sonnet 4.6 was excellent, achieving a perfect score even with single pages.
However, Jev can achieve equivalent accuracy as long as the previous page is passed, while keeping costs at approximately one-hundredth, latency at approximately half, and additionally providing probability values for the classifications.

What I found most practically useful this time was precisely these probability values.
Even for pages with limited judgment material, such as continuation pages without headers or pages with only garbage characters, Jev did not force a definitive answer but instead returned a lower probability.
Both incorrectly classified pages had probabilities below 0.8.
In other words, if a human reviews only the pages where Jev indicates "low confidence," all incorrect classifications could be caught with the current data.
Rather than leaving all pages entirely to Jev, combining low-probability pages flagged for human review with human oversight creates a practical checking workflow for real-world use.
If you need to process a large number of pages and want to build such an operational workflow, Jev is the right fit.

References

Share this article

AWSのお困り事はクラスメソッドへ