I tried recognizing dumpling weighing images with a VLM

Introduction

Hello! I'm tanaka-takeru from Classmethod's Manufacturing Business Technology Department.

In a previous article, I used gyoza making as a subject to recognize work processes with NVIDIA SOP Monitoring Services.
When making gyoza, I had weighed them on a household digital scale and taken image data, so this time I'll use that data to try automatically extracting information such as weight.

As a use case, I'm envisioning reading the displays of existing equipment in factories (7-segment displays, gauges, etc.) and using them as one basis for understanding current status and detecting anomalies.
Of course, if the equipment has an API or can electrically output signals, more reliable data collection can be expected, but I want to prioritize making as few modifications as possible to existing processes and equipment. I also want to verify the strength of VLMs in interpreting image information and expressing it in natural language.

This article explains three experiments in the following flow.

• Task description
• Experiment 1. Recognition accuracy for 50 normal images: How well can a VLM recognize them? Comparison with a classical method using fixed ROI
• Experiment 2. Recognition accuracy for shifted images: How well can a VLM handle changes in the position of the recognition target?
• Experiment 3. Recognition of anomalous images: How can a VLM handle unexpected data?

Task Description

I'll use images of a total of 50 gyoza made by workers A and B.
The information I want to read from a single image is as follows.

An example image is shown below.

• Normal image
  

Regarding the criteria for correct answers:

• Worker, sequence number, weight: Only images where all 3 items are correct are considered a complete match.
• Gyoza ROI: Considered correct if the polygon IoU between the ground truth and the inference result is 0.8 or higher.

The image data was reduced to 1.77 megapixels to match the typical volume zone of general industrial cameras and webcams. It was resized to 1152×1536 px and JPEG-compressed, with each image being approximately 490 KB.

The challenging part will likely be where shadows from hands appear on the 7-segment display area, reducing contrast.

Experimental Environment

• PC: MacBook Pro Apple M5, 10-core CPU, 16GB memory
• OS: macOS 26.5.2
• VLM inference: Ollama 0.31.1

Experiment 1. Recognition Accuracy for 50 Normal Images

VLM Settings and Prompt

No fixed ROI was given to the VLM; instead, the entire scaled-down image was input one by one. The main generation settings are organized by comparison below.

The inference prompt is as follows.

You are drafting annotations for a gyoza weighing image.
Read only visible evidence. Do not guess unreadable values.

For a normal image:
- worker_id is the A or B value shown on the smartphone.
- sequence_no is the integer shown below the worker on the smartphone.
- weight_g is the integer grams shown on the digital scale.
- gyoza_bbox_1000 is the smallest axis-aligned rectangle around visible gyoza pixels,
  excluding the plate, cast shadow, and thread-like foreign matter.

Coordinates use the full displayed image: left=0, top=0, right=1000, bottom=1000.
For an occluded, empty, or otherwise unintended image, explain the anomaly in Japanese.
Set human_review_required=true for anomalies. Use null for values that cannot be confirmed.
Return only data conforming to the supplied JSON schema.

What the prompt provides is the meaning of each value and the definition of "which pixels to include" for the gyoza ROI. The positions of the smartphone and scale, and fixed ROI coordinates, are not provided. The output was also constrained by type and value range using the following Schema.

All fields were made mandatory, and properties not in the Schema were prohibited.
This VLM pipeline will be used in subsequent experiments.

Fixed ROI Pipeline Settings

To serve as a reference for evaluating the VLM, I also tried reading the worker, sequence number, and weight using fixed ROI OCR.
It uses Tesseract 5.5.2 and template matching.

tesseract phone.png stdout \
  --oem 1 \
  --psm 11 \
  -l jpn+eng \
  --tessdata-dir <TESSDATA_DIR> \
  -c load_system_dawg=0 \
  -c load_freq_dawg=0

Results

• The fixed ROI pipeline was able to recognize the worker up to 49/50. On the other hand, the exact match for all items including sequence number and 7-segment weight dropped to 28/50. More pipeline refinement is needed.
• The VLM pipeline result was 50/50. I felt it was a very significant advantage to be able to execute multiple character recognition tasks as well as an object detection task to crop the gyoza location all at once.

Experiment 2. Recognition Accuracy for Shifted Images

In the previous experiment, I compared the VLM pipeline and the fixed ROI pipeline under fixed shooting conditions. However, in actual workplaces, it is often difficult to fix shooting conditions. It may not be necessary to state again, but VLMs extract global and local features from image data, and there is a hypothesis that they can adapt to this challenge.
Therefore, I tried recognition with the VLM pipeline on 300 images created by applying translation and rotation to the 50 normal images.

• Translation: move left, right, up, and down by 5% of each side's pixel distance (4 patterns) ※ Empty areas are filled with edge pixels
• Rotation: ±5 degree rotation (2 patterns)

→ 6 shifted images are generated for each of the 50 normal images, evaluating 50 × 6 = 300 images.

Examples of transformed images are shown below.

Original image	Shifted left (5% of short side length in pixels)	Rotated 5 degrees counterclockwise

Results

The accuracy for all 3 text items exact match was 99.3%. I believe the strength of the VLM manifested as robustness, demonstrating high accuracy in handling position shifts as well.
For the images where the weight was incorrect, both were left-shifted images where 18g was read as 8g.

A portion of the recognition results is shown below.

The image where the weight was recognized as 8g is shown below.

Experiment 3. Recognition of Anomalous Images

Finally, I show the VLM output results for anomalous images: one where the subject is hidden by a hand, and one where neither gyoza nor a smartphone is in the frame.

Results

No gyoza or smartphone in the frame

• Input
  

• Output

{
  "status": "review_required",
  "readings": {
    "worker_id": {"value": null, "confidence": null},
    "sequence_no": {"value": null, "confidence": null},
    "weight_g": {"value": 0, "confidence": null}
  },
  "gyoza_bbox": null,
  "anomaly": {
    "detected": true,
    "code": "empty_scene",
    "message": "画像には餃子が写っていません。空の皿と電子レンジが写っています。"
  },
  "human_review": {
    "required": true,
    "state": "pending",
    "reason": "画像に餃子が存在しないため、重量や位置情報は取得できません。異常な状況です。人間によるレビューが必要です。"
  }
}

• The fact that "detected" under "anomaly" is true indicates that the anomaly is being recognized.
• The "message" outputs the content of the anomaly, but it incorrectly states that a microwave oven is visible.
• The fact that "required" under "human_review" is true indicates that the need for human verification is recognized.
• Looking at "reason," it can be said that the reason for the anomaly is appropriately output.

Occlusion by hand

• Input
  

• Output

{
  "status": "review_required",
  "readings": {
    "worker_id": {"value": null, "confidence": null},
    "sequence_no": {"value": null, "confidence": null},
    "weight_g": {"value": null, "confidence": null}
  },
  "anomaly": {
    "detected": true,
    "code": "empty_scene",
    "message": "画像には餃子が写っていません。手と金属の皿しか見えません。"
  },
  "human_review": {
    "required": true,
    "state": "pending",
    "reason": "撮影ミスまたは意図しない画像と判断します。人間のレビューが必要です。"
  }
}

• The fact that "detected" under "anomaly" is true indicates that the anomaly is being recognized.
• The "message" outputs the content of the anomaly, and it provides appropriate information.
• The fact that "required" under "human_review" is true indicates that the need for human verification is recognized.
• Looking at "reason," it can be said that the reason for the anomaly is appropriately output.

In Closing

Here is a summary of the strengths of VLMs.
・Integration of multiple tasks including OCR, object detection, and interpretation
・Handling of positional shifts in the subject being photographed
・Escalation to humans during anomalies, and interpretability of anomaly content

It was confirmed that VLMs can compress video information very efficiently and flexibly (in a natural language sense).
This can be expected to have lower startup costs compared to conventional rule-based processing, and there is also the possibility of handling process changes (for example, replacing the scale, or switching from gyoza to shumai) without changing the program.

Rather than relying solely on VLMs,

• Use fixed ROI pipelines selectively for parts requiring real-time performance
• Use local/cloud selectively depending on usage frequency (whether daily or only during improvement reviews)

I believe that many types of information can be appropriately extracted through such approaches.
I would like to continue exploring different VLM models, changes in computational resources, and architectural considerations for practical field operations.