I tried connecting factory floor videos with process documents to enable searching for abnormality candidates in operations
This page has been translated by machine translation. View original
Hello! I'm tanaka-takeru from Classmethod's Manufacturing Business Technology Division.
On the manufacturing floor, with so much information available — equipment signals, quality records, standard work instructions, daily reports, and more — it is not easy to comprehensively understand processes in an integrated, cross-functional way as production continues and conditions constantly change.
- Are there any safety issues? Can we meet today's production targets? Are we deviating from standards?
In response to this situation, NVIDIA has announced the Factory Operations Blueprint, commonly known as FOX. It is a configuration that connects equipment signals, quality information, work procedures, and on-site alerts, with a Factory Manager AI that understands and supports the state of the shop floor.
On the other hand, since it is not realistic to immediately aim for a factory-wide data platform or full autonomy, Mini-FOX — introduced by my colleague Mori Shigeru — is the idea of starting small with one line, one camera, and one use case for validation.
Introduction
This time, drawing on the Mini-FOX concept, I built a small integrated search and verification demo using manufacturing data, narrowing it down to one process, one worker's-perspective video, and one question: "I want to know about anomalies that occurred outside of standard work."
This article introduces the pipeline — from extracting review candidates from a work video, linking them to document information, and querying in natural language — along with a demo video.
Data
For the video, I used factory001_worker001_00002.mp4 from Egocentric-10K.
This is a first-person perspective video of press machine work, and the file used this time is approximately 20 minutes long, 1920×1080, 30FPS. It was shot wide-angle with a near-fisheye lens, and some distortion aberration is also visible.
Egocentric-10K is an open dataset containing first-person videos from inside factories. The video used this time does not come with annotation information such as process names, work standards, or anomaly labels. Therefore, rather than judging "what the correct work is" by watching the video alone, I decided to create process documents and combine them with the video.
Pipeline
The processing follows this flow.
- The work video is sampled at 5FPS, and how much movement has occurred from the previous frame is quantified from frame differences. Additionally, the die area and hand movements are detected.
- The time series of motion quantity is analyzed using autocorrelation, and intervals that deviate from the usual repetitive motions are extracted as "review candidate intervals." At this stage, they are not confirmed as anomalies.
- Only the review candidate intervals are re-extracted at 10FPS and passed to a VLM running locally (Qwen3-VL-4B-Instruct). Here, the observed movements are recorded as observation memos.
- A human reviews the candidates as needed to determine whether they are anomalies (asynchronously). Unreviewed candidates remain as-is.
- Time series data, VLM observation memos, human reviews, and process-related documents are consolidated into event JSON.
- When a user submits a question, the relevant event JSON and document chunks are selected and passed to an LLM (Claude Sonnet 5). Claude constructs a response along with its evidence.
In the first stage, the entire video is not passed to a large-scale model. The video is processed at 5FPS to create time series data of overall motion quantity, die area tracking, and hand position and movement, and from that, intervals that differ from the normal repetitive motion are extracted as review candidates to focus on information worth attention.
This is similar to the stepwise structure of Mini-FOX: "first convert to events, then look deeply only at the necessary candidates."
Input Information
Video Data
On the video side, the following information was detected and made available on a common time axis.
| Information | Acquisition method | Usage this time |
|---|---|---|
| Motion quantity | Frame difference | Detection of disruptions in repetitive motion and motion stagnation |
| Die area | Tracking with SAM2 | Supplementary information on whether the operator is near the die and whether the area is occluded |
| Hand movement | MediaPipe Hand Landmarker | Supplementary information on hand position, speed, and left/right movement |
As a premise, how quickly and cheaply the pipeline can be horizontally deployed to other processes and products is extremely important, so basically no fine-tuning or optimization is performed, and the pipeline is built to work with zero-shot or one-shot.
The following shows the video with extracted information overlaid.
- Regarding motion quantity, since first-person perspective footage often changes angle according to the work stage, extracting the overall motion quantity from the video is expected to roughly capture periodicity. It is plotted at the bottom of the demo video (inter-frame change amount, dimensionless). Autocorrelation over a 30-second window is used to find locations that deviate from the stable repetitive motions throughout the video. The frame difference calculation itself is very lightweight and can be handled even on edge devices such as Raspberry Pi.
- The die area is tracked with Meta's SAM2 as it is considered most related to value-added time. It is the green region in the center of the video. By extracting changes in this die area, cycle time and production count can be tracked.
https://github.com/facebookresearch/sam2?tab=readme-ov-file - Hand movement is detected with MediaPipe Hand Landmarker. Note that in actual work environments where gloves are worn, accuracy may decrease depending on their color, bulk, and the presence of oil or dirt.
https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker#models
Document Data
The following demo documents were created.
| Document | Main role |
|---|---|
| Safety Work Standard Regulations | High-level safety rules |
| Standard Work Procedure Manual | Work procedures and safety notes |
| Standard Work Combination Sheet | Standard takt and elemental work for P-040 |
| QC Process Chart | Quality standards during workpiece setup |
| Daily Inspection / Production Daily Report | Records of results, warnings, and delays |
An example is shown below.
There are a very large number of types and quantities of process-related documents, but I assumed they can be broadly divided into three categories.
- First, information defined at the company or division level. This includes rules, regulations, and operational responsibilities; in this case, the company-wide "Safety Work Standard Regulations."
- Next, information that defines the process. A QC Process Chart, Standard Work Combination Sheet, and Standard Work Procedure Manual were prepared.
- Finally, information that flows through the process on a daily basis. This includes production plans and staffing assignments; in this case, daily inspection records and production daily reports.
Using the fictionally assigned process code P-040 and document numbers as keys, events and related documents can be traced.
Event JSON
Information from intervals identified as "possibly different from usual movement" is consolidated into units that are easy for the downstream LLM to handle.
As shown in the table below, five types of information are combined.
| Layer | Event JSON field | What it contains |
|---|---|---|
| 1. Measured values | time_series_evidence |
Time series data of motion quantity, die area, hand position, etc. |
| 2. Rule-based candidate extraction | candidate_detection |
Extraction signals and aggregated values that produce the same result given the same input and conditions |
| 3. VLM video observation | visual_evidence |
Description of movements visible in the video and notes on how things appear |
| 4. Human review | human_review |
Facts recorded by a person after reviewing the video |
| 5. Document evidence | document_links |
References to standard work instructions, QC process charts, daily reports, etc. |
As an example, the JSON for a review candidate event that includes human review information looks like the following. For readability, aggregated values, VLM memos, and document links are excerpted.
{
"entity_type": "tact_deviation_candidate",
"event_id": "P040-CAND-001",
"process_id": "P-040",
"time_range_sec": [335.0, 394.8],
"focus_sec": 370.0,
"status": "Requires review",
"candidate_detection": {
"signals": ["Disappearance of repetitive motion", "Deviation from normal cycle"],
"metrics": {
"observed_period_sec": [7.6, 7.8, null],
"period_strength_range": [0.147, 0.304]
}
},
"time_series_evidence": {
"motion": {"source": "motion.jsonl", "motion": {"mean": 0.1115}},
"die_roi": {"source": "tracks.jsonl", "present_ratio": 0.91},
"hands": {"source": "hands.jsonl", "by_side": {"Left": {}, "Right": {}}}
},
"visual_evidence": {
"sampling": {"fps": 10, "max_frames_per_inference": 16},
"llm_excerpt": [{
"range_sec": [368.6, 370.1],
"visual_observation": "The right hand is adjusting a metal part placed on the jig.",
"caveat": "Hand movement is minimal, making it difficult to judge detailed actions."
}]
},
"human_review": [{
"event_id": "P040-TD-002",
"severity": "minor",
"category": "workpiece_set_failure",
"summary": "The workpiece could not be set properly in the crimping machine, resulting in a re-set."
}],
"document_links": [
{"document_id": "STD-WI-P040-001", "section": "Basic information"},
{"document_id": "QC-PR-2026-042", "section": "Workpiece setup"}
]
}
The fields may look somewhat complex, but they are separated to avoid conflating measured values, rule-based candidate extraction, VLM observations, human reviews, and document evidence.
- To avoid concluding that something is an anomaly despite it only being a candidate
- To make the evidence behind events easier to trace
- To properly allow human review to be inserted
For example, "the hand appears to be adjusting the jig" is a VLM output, and whether that is actually true can only be determined after a person reviews the video. If these were merged into a single summary sentence, it would become difficult for later viewers or processing LLMs to determine where the measurement results end and where the interpretation begins, which is why the above structure was adopted.
Demo
In the demo screen, the work video and time series graphs are on the left, and the question UI is on the right. The video has the die area tracked by SAM2 and the hands detected by MediaPipe overlaid on it. The video playback position and the graph position are always synchronized.
Clicking on a review candidate interval in the graph allows you to view the VLM output for that interval along with detection signals, die and hand time series information, and whether a human review has been performed.
This time, I visually inspected the motion quantity graph and manually reviewed the following two cases.
| Human review event | Time | Content |
|---|---|---|
| P040-TD-002 | Around 6:05 | Workpiece could not be set properly in the crimping machine, resulting in a re-set |
| P040-TD-001 | Around 12:39–14:21 | Work was interrupted and the worker moved to another worker's location |
The automatic extraction produced 4 candidate intervals including these 2 cases. The 2 human-reviewed cases and the still-unreviewed candidates are clearly separated and recorded in the event JSON.
For example, when asked "I want to know about anomalies that occurred outside of standard work," the LLM centers its response on the human-reviewed workpiece setup failure and work interruption, while distinguishing unreviewed intervals as "candidate detection only." A constraint is also applied so that the video observation memos alone are not used to definitively conclude causes or safety violations.
When actually operating the screen, I found that the cycle of looking at time series changes, returning to the video of intervals of interest, and asking questions while referencing document evidence enables far more multifaceted information catch-up compared to watching a 20-minute work record in full.
Number of Frames Passed to the VLM
The video of review candidate intervals was passed to Qwen3-VL-4B-Instruct running in a local environment for observation. Here, how to balance between passing only a few representative frames per interval versus passing all frames of the video becomes a challenge.
The length of these intervals is quite difficult to predict. While it is possible to constrain them with a fixed threshold, the preference is to obtain information as straightforwardly as possible. The intervals actually flagged in the demo ranged in length from approximately 60 seconds to approximately 195 seconds. The former is a relatively short candidate interval, while the latter is a case where the worker leaves the work area and moves to another worker's location.
In that case, for example, inputting only 3 frames — start, middle, and end — to the VLM risks missing minor adjustments at hand or changes leading up to the worker leaving. On the other hand, passing several minutes of video directly to the VLM is not realistic in terms of input size, inference time, and cost.
Therefore, the following approach was taken.
- Convert the candidate interval to an image sequence at 10fps
- Pass a maximum of 16 frames (equivalent to 1.6 seconds) per inference
- Retain observation memos in units of 16 frames, and save their excerpts and references to the source data in the event JSON
In other words, rather than having one inference comprehend the entire long candidate interval, the interval is divided into small units in chronological order for observation. In this run, approximately 38 inferences were made for the ~60-second candidate and 122 inferences for the ~195-second candidate. Since there is no single correct answer for the number of frames across all processes, this is designed to be adjusted according to the length of the candidate interval and the desired granularity of review.
10FPS was chosen not to enable complete continuous understanding of the video, but to a degree that makes it less likely to miss short changes such as minor hand adjustments near the jig.
Also, the role of the VLM is limited to describing the movements visible within the interval. It is not used to confirm anomalies, estimate causes, or judge safety violations.
About RAG
When a question is received, first the event JSON related to the question and the process code is selected.
Next, the document chunks referenced by those events are gathered, and the events, documents, and human reviews are collectively passed to the LLM.
The response prompt includes the following constraints:
- Do not conflate candidate detections with human-reviewed events
- Do not use VLM observation memos as definitive evidence for causes or violations
- Include the event IDs and document numbers that serve as evidence in the response
- State "unknown" for causes not present in the data
This time, no vector search infrastructure was prepared; instead, documents were chunked by semantic unit, and related chunks were selected using process codes, document IDs, tags, and keywords.
Closing
This time, using manufacturing videos and process documents, I built a small integrated search and verification demo as a Mini-FOX.
Rather than passing the entire video to a VLM, by extracting review candidates from lightweight video metadata, converting candidates into events, and then linking them with documents, I was able to build a pipeline that is mindful of both processing cost and ease of human review.
Adding information from upstream and downstream processes, as well as functional-axis information such as from production engineering, production management, and quality assurance, would lead to significant added value.
Furthermore, there is exciting potential in SCM-axis expansion such as sales and procurement information linkage, as well as PLM-axis expansion into development/FE feedback and improvement requests.
I believe the appeal of Mini-FOX lies in how these technologies can be started on a small scale. Starting with one process, one camera, and one question, and experiencing firsthand the transformation of video into information that can be referenced on the shop floor, is what I believe to be most important.
