Dumpling Making Process Recognition　NVIDIA SOP Monitoring Services

Introduction

Hello! I'm tanaka-takeru from Classmethod's Manufacturing Business Technology Department.

Previously, I conducted time study and motion study using a video of gyoza wrapping.

Time Study and Motion Study with Gyoza Making

As a follow-up, to address the challenge of "needing improvements to increase interval estimation accuracy," I used NVIDIA's publicly available SOP Monitoring Blueprints to automate process recognition from work videos using a fine-tuned AI model.

This article covers the following:
・Overview of the SOP Monitoring Blueprints inference pipeline
・Execution environment
・Task description
・Results
※ Environment setup procedures and annotation/training techniques will be covered in separate articles.

Overview of the SOP Monitoring Blueprints Inference Pipeline

The structure is broadly as follows:

The key point is the structure where ①DDM-Net first segments each process (pick wrapper → add filling → apply water → pleat → place on tray), and then ②VLM estimates what type of process each segment is. Here is a brief explanation of each model's characteristics and role.

①DDM-Net

DDM-Net is a model specialized in "finding where actions switch in a video (boundaries)," acting as the video segmentation component. Think of it as advancing through the video frame by frame and placing markers at points where "the nature of the movement has changed." It only captures changes in properties in the image space and temporal space, and does not deal with what those changes mean.

②VLM

The VLM understands both video and language, and acts as the classifier that determines for each short video clip segmented in ① whether it is, for example, "the pleat process." Additionally, Cosmos-Reason2-2B used here is a model published by NVIDIA, a VLM with strong capabilities in understanding work videos, post-trained (additionally trained) on top of Qwen3-VL-2B-Instruct.

The following prompt is sent to the VLM:

There are 6 possible steps for the SOP (Standard Operation Procedure) of the given video.
What step is the operator doing?
(1) pick: reaching to the stack of gyoza wrappers, taking one wrapper, and bringing it back to the working position
(2) fill: scooping filling with a spatula, placing it onto the wrapper, and spreading it over the wrapper
(3) water: dipping a fingertip into the water dish and wetting the edge of the wrapper
(4) pleat: folding the wrapper in half and pinching the edge to form pleats and seal the dumpling
(5) place: placing the finished dumpling onto the metal tray and arranging it
(6) doing action not belong to the defined SOP

Simply showing the video and asking "what task is this?" gives too much freedom in the answer, making it difficult to get appropriate responses. Therefore, the question is framed to ask which of the 5 pre-defined processes it belongs to, or whether it belongs to none of them.
During training, an LLM (Llama-3.1-70B) generates large amounts of Q&A with varied phrasing.

For Recognizing Gyoza Making at My Home

The above general-purpose AI is fine-tuned to better adapt it to the current data.
Note that VLM training requires a large number of "video + question + answer" pairs, and preparation would be enormously costly, so this time data augmentation is performed from a small amount of annotation data. This uses an LLM (Llama-3.1-70B) via the NVIDIA NIM API. The video I manually annotated this time totals approximately 10 minutes combined for training and validation.

Finally, here is an overview of the entire flow:

Execution Environment

Since this involves full fine-tuning of the VLM, the official requirements are 4 × A100 80GB.
The environment was prepared using NVIDIA's cloud GPU service Brev.

For information on Brev, please also refer to our previous articles (CLI edition and connection edition).

The models were obtained as follows:

Hugging Face and NVIDIA NGC API Keys are required.

Task Description

The source video is the same as last time: gyoza wrapping work (18 minutes 32 seconds) filmed with a camera fixed in a top-down view.
This time, one worker visible on the upper side was targeted, and the upper portion was cropped for use. The process definitions are also the same as last time.

Training and validation are split by time. The validation interval is not used for training.

The number of cycles equals the number of gyoza, and approximately 6 videos were used for each of training and validation.

Annotation was performed using the Web UI included with the Blueprint.
The task involved segmenting approximately 10 minutes of combined training and validation video into 66 intervals, which took about 40 minutes.

Results

First, here are the final results after segmenting processes with DDM-Net and classifying them with Cosmos-Reason2-2B.

Let's also look at the individual results for each.

DDM-Net (Process Segmentation): precision 1.0 / recall 0.52

F1 is the boundary detection score. It is judged by whether "the predicted boundary time matches the ground truth boundary within a tolerance (±1 second)."
precision 1.0 / recall 0.52 represents the current capability.
Regarding how recall 0.52 affects the results intuitively, the graph mentioned earlier shows that narrow purple and light blue bands are noticeably missed.
In a word, it can be summarized as "no incorrect boundaries are placed, but short processes (water application and tray placement) tend to be missed."

Training time was approximately 1 hour and 20 minutes.

Cosmos-Reason2-2B (Process Classification): Accuracy 0.967

Breakdown by process:

All 5 defined processes achieved 100%. Even visually similar processes such as "add filling / apply water / pleat folding" were correctly classified.
A small idle time was missed, but I was surprised that just 6 cycles of annotation could achieve this level of discrimination.

Training time was approximately 1 hour.

Conclusion

Regardless of the manual process involved, understanding how much time is spent on what is always necessary for improving processes.

In the data from this study, while interval segmentation accuracy is still experimental, process recognition was achieved with high accuracy.
By designing for robustness to fit the constraints of a real-world environment, it becomes possible to analyze focusing on the elements you want to stratify.
For example, "I want to recognize even when the worker changes and check whether there are differences in how jigs are used," or "the workbenches and layouts differ between locations, but I just want to see the overall cycle time."

I would like to combine approaches such as rule-based processing, QR code reading, and physical switch timestamps to build a solution that can operate stably.