[TypeSafe] I tried to see if passing PDFs and PNGs in Base64 to Jev, which is text-only, would allow it to read them

Hi, I'm Keema.

I've been playing around with TypeSafe's Jev and finding it quite interesting.
I noticed there could be some use cases for it in projects I usually work on, so I was curious about it.

One project involves OCR, which is what got me thinking about whether Jev could be put to good use there.

However, looking at the official documentation, it clearly states that the input is text only, and that images, audio, and video are not accepted.

Input: Text only. String, JSON object, or array of text values. No image, audio, or video input.

Source: Models | TypeSafe Docs

So I decided to try something: "What if I convert it to Base64 and pass it as text — could it maybe read it?"

This time, I'll use TypeSafe's Playground to convert trade document PDFs and PNGs to Base64, pass them in, and see if it can identify the type of document.

Specifically, I encoded a Bill of Lading (B/L) into Base64 and tried to have it classify the document as one of the following: B/L, Invoice, L/C, Waybill, P/L, or Unknown.

To give you the conclusion upfront: it was completely unable to read either the PDF or the PNG.

1. What I Tried

In the TypeSafe Playground, I ran each of the PDF and PNG once under the following conditions.

Bill of Lading (B/L): A trade document issued by a shipping company indicating that cargo has been received.

The B/L used is as follows.

The B/L used in the verification (company names, vessel names, and numbers are all fictional)

The correct answer is B/L, so if it could be read, doc_type should return bill_of_lading.
I also used Noul (a question that returns yes/no probabilities) to check whether "the text inside could actually be read at all."
Since Jev performs best in English, the questions are written in English.

The State was set to the Base64-encoded string of the PDF as-is.

State (for PDF):

{
  "document": {
    "mime_type": "application/pdf",
    "encoding": "base64",
    "file_base64": "<Base64-encoded value>"
  }
}

Questions (for PDF):

{
  "doc_type": {
    "type": "choice",
    "instructions": "`document.file_base64` holds a base64-encoded PDF of a single trade document. Decide which type of trade document it is.",
    "criteria": {
      "bill_of_lading": "Bill of Lading (B/L). A negotiable transport document issued by the carrier. Typical fields: Shipper/Exporter, Consignee, Notify Party, Ocean Vessel/Voyage, Port of Loading, Port of Discharge, B/L Number, Number of Original B/Ls, and a \"surrender one original duly endorsed\" clause.",
      "commercial_invoice": "Commercial Invoice. Issued by the seller to the buyer to claim payment. Typical fields: Invoice No., description of goods, quantity, unit price, total amount, trade term (FOB, CIF), and payment terms.",
      "letter_of_credit": "Letter of Credit (L/C). A payment undertaking issued by a bank. Typical fields: Issuing Bank, Applicant, Beneficiary, L/C Number, Expiry Date, Documents Required, and SWIFT MT700 tags such as 40A, 31D, 46A, 47A.",
      "waybill": "Waybill (Sea Waybill or Air Waybill / AWB). Evidences carriage but is non-negotiable; goods are released without surrender of an original. Marked \"Non-negotiable\" or carries an AWB Number.",
      "packing_list": "Packing List (P/L). Centres on Case No., number of packages, contents, Net Weight, Gross Weight and Measurement (CBM). Normally shows no unit prices or total amount.",
      "unknown": "None of the above, or the content cannot be read so there is no basis to decide the document type."
    }
  },
  "is_readable": {
    "type": "noul",
    "instructions": "Can you actually read the words written on the document from the content of `document.file_base64`?",
    "criteria": {
      "true": "The document text and field labels are actually legible, giving a real basis for the decision.",
      "false": "It looks only like a base64-encoded string; the document content is not legible."
    }
  }
}

For the PNG case, I changed the mime_type in State to "image/png", and only changed the instructions of doc_type in Questions as follows.
Everything else remained the same as the PDF case.

"instructions": "`document.file_base64` holds a base64-encoded PNG scan of a single trade document. Decide which type of trade document it is."

2. Results: Still Couldn't Read It

PDF execution result

PNG execution result

In both cases, the correct answer bill_of_lading was not selected, and unknown — meaning "cannot determine because it's unreadable" — was returned with nearly 100% confidence.
The true rating for is_readable was also quite low, suggesting that Jev itself is aware that it cannot read the content.

3. Summary

My faint hope that "converting to Base64 as a string might work" was thoroughly dashed.
As stated in the official documentation, non-text inputs such as images and PDFs need to be converted to text or structured data in advance before being passed to State (Models | TypeSafe Docs).

If you want to handle PDFs or images with Jev, the reliable approach is to first extract the text using OCR or a text extraction tool, then pass that text to State.

Through this verification, I also learned that it's important to be mindful of the input token limit.
Jev 1.13 has the following two token limits (Models | TypeSafe Docs):

• Overall request total limit: maximum 64k tokens
  • state (premise data) + total of all questions must be within 64k
• Per-question individual limit: maximum 32k tokens
  • The combination of state (premise data) + the single longest question must be within 32k

Converting to Base64 inflates the character count beyond the file size (approximately 1.33x), and since it becomes a string of random alphanumeric characters, it also consumes a considerable number of tokens.
In fact, when I first tried passing a regular PDF I had on hand directly as Base64, it resulted in an error due to exceeding the limit.
For that reason, in this verification I reduced the PDF to approximately 2KB before testing.