Tracing back to work videos from quality information. I tried grasping the current status of processes.

Introduction

• I have been trial and error in tracking the actual record of when and through what processes products were completed, but have often given up.
  • Recording with a stopwatch was too much trouble to capture all occurrences
  • Installing QR codes and positioning jigs was physically difficult
  • Adjusting image recognition parameters was too troublesome to keep up with drawing revisions
• However, with recent advances in VLMs, I feel there is a strong possibility of capturing process actuals as data in a very flexible way.
• This time, I would like to try analyzing work videos using a VLM to extract time and content, and then linking that with the quality information of finished products (in this case, weighing results) to understand the current situation.
• It's yet another gyoza video, but... since no gyoza-specific processing such as fine-tuning has been applied, I believe this can be extended to a wide variety of tasks.

Dashboard Demo

First, here is a look at the dashboard in action.

https://youtu.be/guoTjn2Yg00

In this article, I would like to dive deeper into the design intent of the dashboard and the flow of understanding the current situation, with regard to the process of two workers each making 25 gyoza (50 in total).

Design Intent

• A dashboard of on-site management indicators always comes paired with a question.
  For example, in order to improve economic efficiency beyond the current process, it is standard practice to design the whole package together: "What hypotheses do you hold, what indicators do you look at, and what actions do you take?"

• This time, I would like to use the question "What should we do to make gyoza of the same quality faster than we do now?" to understand the current situation.

• First, I will list the indicators I want to manage, then think about the data needed to express them and the visualizations needed to understand them.

  • Quality:
    • Here, the weight of the finished gyoza is used as the indicator. Getting the right amount in is surprisingly difficult.
    • The recipe I referenced says to wrap 20g of filling, so with a wrapper weight of 3g, 23g per gyoza seems like a reasonable weight.
    • Since it was too much trouble to measure and record each gyoza one by one, I took photos of each measurement and used a VLM to read the scale display and convert it to data (taking the photos was also quite a challenge).
    • What kind of distribution will the weights of the 50 finished gyoza show? An I-MR control chart seems like a good way to check.
  • Lead time:
    • The time to make a single gyoza (cycle time) is used as the indicator.
    • The video is split into small chunks and fed into the VLM. The detection target is defined as the time from when a human hand picks up a wrapper to when the hand places the gyoza on the tray and moves away, and the cycle time is converted to data.
    • There are two workers. Will cycle times differ between workers? Also, both workers were beginners at making gyoza — will their cycle times improve as they make more?
    • A box plot that reveals variability seems like a good way to check.

• The general data flow turned out to be as follows.
  

• After trying multiple VLM models, I am using Qwen/Qwen3.5-4B for video and Qwen3-VL 4B Instruct for images.

• To grasp all of this information in one place, the following layout was adopted.
  

• It is very simple, but the flow is to find points of interest in the weight and cycle time distributions, then review the corresponding images and video.

• Also, from a UX perspective, what is important is that when you click on a single point on the control chart, the box plot, images, and video all update in sync. This makes it easy to develop an intuitive feel for the relationships between indicators and between indicators and the images/video.

Understanding the Current Situation

Let's watch the video again and think about "What should we do to make gyoza of the same quality faster than we do now?"
https://youtu.be/guoTjn2Yg00
(This is the same video as at the beginning of the article.)

I apologize for how hard it is to see, but I hope you will read along while pausing the video at the relevant points.
1. First, let's look at the control chart for weight. Video timestamp: 3 seconds

• From the I chart, the average is trending around 20.4g. No points exceed ±3σ.
• Given that 23g is considered appropriate, the overall amount of filling is a little low. Perhaps they could pack in more as they get used to it...
• Comparing Worker A (blue) and Worker B (yellow), B may tend to make heavier gyoza overall.
• Also, looking at the MR chart, Worker B seems to have more abrupt fluctuations than Worker A.

2. Let's look at the cycle times. Video timestamp: 7 seconds

• Worker A has a median of 43 seconds, and Worker B has a median of 39 seconds.
• There does not appear to be any reduction in cycle time as they make more gyoza.
• Looking at the distribution, it seems possible to converge cycle times to around 40–45 seconds. What was happening in cases where it took more than 45 seconds?

3. Let's dig deeper into the longest cycle time (outlier). Video timestamp: from 1 minute 22 seconds onward

• First, the weight of the gyoza made in the outlier cycle is distributed around 20g, so there is no problem with the weight.
• Since it would be risky to judge by weight alone, looking at the photo of the finished gyoza confirms that it is neatly wrapped.
• Now, watching the work video:
  • Before picking up the wrapper, the worker counts the number of gyoza made so far.
  • Also, after scooping the filling twice, the filling sticks to the wrapper and spoon and cannot be placed smoothly, and the worker adjusts it several times.
  • While wrapping, a large piece (cabbage?) pops out of the filling, and the worker pushes it back in.
  • Finally, when placing it on the tray, the space is tight, and the worker squeezes it in and rearranges the row.
• From this, we can learn the following:
  • The outlier is a correctly detected cycle time (it is not a value that should be excluded).
  • There is no anomaly in the weight of the finished gyoza, but four elements combined during the process of making it may have significantly extended the cycle time.

Closing

• Having obtained four hypotheses, in a real work setting I would immediately interview Worker A and likely arrive at action items such as:

  • Considering a mechanism that allows gyoza to be placed on the tray with more room to spare and at a fixed, unambiguous position, rather than placed freely
  • Considering a jig that can scoop a fixed amount of filling and place it on the wrapper easily
  • Investigating the variation in particle size of the filling ingredients
    ・・・and so on (I won't be doing this at home).

• I came away with the impression that tackling hypothesis-building while maintaining an intuitive feel for the raw data like this, and then investigating statistically significant relationships, seems like a smooth path to improvement.

• For a process as simple as making gyoza, plenty of improvement points can be identified just by watching, but what I find most noteworthy is that VLMs have made it possible to verbalize and structure human work and process conditions, including the dimension of time.

• If information can be easily extracted from video and linked with various manufacturing data including quality information, it can be expected to lead to an appropriate understanding of complex current situations and to concrete actions.