I tried verifying the presence/absence determination of material input operations using a local VLM on DGX Spark

Hello! I'm tanaka-takeru from Classmethod's Manufacturing Business Technology Department.

In food and material manufacturing sites, the process of feeding materials into kettles and similar equipment is a routine operation.
Various mechanisms have been introduced to prevent human errors such as forgetting to add materials,
and this time, we conducted an actual verification experiment to determine whether "whether material feeding work was performed" can be judged from video data.

Data

I filmed a simple simulation video.
It shows the process of manually feeding the contents of transparent plastic bags and paper bags into a cardboard box.
Each bag has a QR code attached to indicate what is being fed.
The workflow involves showing the QR code to the camera and then feeding it into the cardboard box.

https://youtu.be/HLB12r4fH1M

A total of 4 scenarios were filmed, with 3 types of materials per scenario (set as vegetables, curry powder, and udon) being fed.

The video data has the following values.

※ The videos were filmed by Seiya Tanaka.

Analysis Pipeline

If we input all videos into a local VLM, memory will be exhausted during video decoding, or the VLM will significantly thin out frames, so it is necessary to appropriately control the video input size.
Therefore, it is effective to introduce processing that extracts segments (hereinafter referred to as clips) within the video that we particularly want the VLM to check. This applies to other video data as well, and especially when monitoring manufacturing sites or experimental processes, it becomes the basic form of an analysis pipeline. For preprocessing, classical image processing, object detection models, and skeleton detection are used to introduce real-time or high-speed processing, but this time we use QR code recognition.

We will create clips by cutting out frames for 15 seconds after a QR code is detected in the video and inputting them into the VLM. Since there is a possibility that the next feeding operation may be performed within 15 seconds, if the next QR code is detected within 15 seconds, we will cut off the clip at that point.

With the above processing, here is a list of clips created from the 4 videos shown earlier.

The Ground Truth column indicates whether feeding work was present or absent in the clip with Present / Absent. The task for the VLM is to correctly determine this Present / Absent.
Furthermore, judging Present when feeding work was not performed (false positive) can be a significant risk in actual operations, so careful checking is necessary. Therefore, as negative control data points, the following clips are also added to the experimental data.

A total of 16 clips are shown.

https://youtu.be/zUH7mBab9J8?si=_tXnZ7x1Y-0ZJ2LE

Also, here are two edge cases as supplementary notes.

• VEG2: The QR code was presented, but the feeding work was not performed, and the vegetable bag was left sitting beside the cardboard box.
• VEG3: The QR code was presented and feeding work was performed, but 1 vegetable remained in the bag and the full amount was not fed.
  ※ Since this experiment asks about whether feeding work occurred, whether VEG3 should be classified as Present or Absent is debatable, but considering the goal of error-proofing, we felt it was preferable to check more strictly, so we set Absent as the correct answer. In practice, the act of feeding and the quantity fed should be guaranteed separately as distinct functions.

The clip video data has the following values.

Tradeoffs of FPS Settings
The original videos were filmed at 30fps, but during clip processing they were reduced to 5fps. Naturally, a higher fps has the advantage of capturing even momentary movements, but when inputting to VLM, we adjusted it to balance with pixel count.
Specifically, for example, the processor of Cosmos3-Nano, one of the VLMs considered this time, automatically reduces resolution according to the number of frames to fit the entire video within a fixed number of tokens (approximately 12,000 as measured). In other words, increasing fps causes the amount of information the model receives to be adjusted on the side of pixels per frame. There is a tradeoff between temporal granularity and spatial granularity. It was confirmed that if input at 30fps as-is, the pixel count per frame would drop to one-sixth (approximately 310×170) compared to 5fps. Therefore, while visually checking the data, we set the minimum pixel count and fps that could capture the feeding action at 768×416 or higher and 5fps.

About Local VLM

The execution environment is as follows.

The following models were used for accuracy and speed comparison.

Among the 5 models, all 4 except InternVL3.5-8B-HF use Qwen-based models on the language side. In particular, Cosmos-Reason2-8B matches Qwen3-VL-8B-Instruct in all of architecture, number of layers, vocabulary size, and total weight file size (17.53GB), which suggests that NVIDIA performed post-training on that model.
Cosmos3-Nano has the same language-side and vision-side configuration as Qwen3-VL-8B-Instruct, but the parameter count is approximately 15.8B, about 1.8 times larger, and this difference stems from the diffusion model side (diffusion_expert_config) responsible for video generation. However, since the peak memory during inference is the same value as the other three Qwen-based models (21.4GB), the generation side is not used in reasoning tasks like this one. InternVL3.5-8B-HF uses text-only Qwen3-8B on the language side, but the vision side is a proprietary implementation, adopting a method of fixed allocation of 256 tokens per frame, which differs from all other models. Also, only Cosmos3-Edge has a configuration prioritizing smaller size and speed, with a vocabulary size of 131,072 and approximately 3.9B parameters.
This time, we selected 5 models to be able to compare results from the perspectives of the effect of post-training based on Qwen3-VL-8B-Instruct, Cosmos family model sizes, and architecture.

Prompt

The following string is used unchanged across all models. No system prompt is set.

ダンボールの中への投入動作の有無を観察し、有/無/わからないで判定してください。

Generation Parameters

The generation parameters for each model when processing clips were as follows.

We adopt greedy decoding with do_sample=False. A language model outputs a probability distribution over the entire vocabulary at each step, but with do_sample=True, one word is selected from this distribution using random numbers, so different outputs can be produced for the same input across runs (the sharpness of the distribution peak is adjusted with temperature, and lower-ranked candidates are pruned with top_k / top_p before extracting reasonable output). In contrast, greedy decoding (do_sample=False) always selects the word with the highest probability, does not use random numbers, and returns the same output for the same input. When the purpose is monitoring or decision-making at manufacturing sites, it is important to get the same result for the same input, so we adopt greedy decoding.
Note that the reproducibility of greedy decoding is premised on the hardware, library versions, and batch configuration being identical. These must be fixed during operation and recorded for each judgment.

Here is an example of the inference source code.

import torch
from transformers import AutoProcessor, Cosmos3OmniForConditionalGeneration

MODEL_ID = "nvidia/Cosmos3-Nano"
PROMPT = "ダンボールの中への投入動作の有無を観察し、有/無/わからないで判定してください。"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = Cosmos3OmniForConditionalGeneration.from_pretrained(
    MODEL_ID, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": [
    {"type": "video", "path": "clips/S3-take01/veg.mp4"},
    {"type": "text", "text": PROMPT},
]}]

inputs = processor.apply_chat_template(
    messages, fps=5, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device, torch.bfloat16)

generated = model.generate(**inputs, max_new_tokens=4096, do_sample=False)

trimmed = generated[0][inputs["input_ids"].shape[-1]:]
print(processor.batch_decode([trimmed], skip_special_tokens=True)[0]) 

Results

Accuracy

First, here is a summary of each model's output results for each clip. Models are arranged from left to right in order of performance.

Here is a summary of results by model.

Only Cosmos3-Nano achieved a perfect score. Qwen3-VL-8B-Instruct and Cosmos-Reason2-8B followed closely behind, while Cosmos3-Edge and InternVL3.5-8B-HF fell far behind.
The one mistake made by Qwen3-VL-8B-Instruct was VEG3, a clip where feeding action occurred but material remained in the bag, so we consider it to have performance very close to Cosmos3-Nano.

Speed and Load

Here is a summary of inference speed and load. The preprocessing breakdown consists of video decoding, tensorization, and GPU transfer processing. The cycle time is defined as this preprocessing time plus generation time. QR code reading processing is not included.
Also, prefill speed is the speed at which the model reads the input (tokens derived from video), calculated by dividing the number of input tokens by the execution time of model.generate(). Since the output in this case is approximately 2 tokens which is extremely short, almost all of the processing time is spent on reading the input, so this metric is used to evaluate the actual processing speed.
Results are summarized in the table in order of cycle time.

As expected, Cosmos3-Edge was the fastest. The variation in cycle time was less than 10% for Nano, Qwen3VL, Reason2, and Edge. Only InternVL had a range of 4.58 to 11.28 seconds, a 2.5x spread, which we believe stems from InternVL's specification where the token count is proportional to the number of frames.

Number of Generated Tokens and Output Language

Here is a summary of the number of generated tokens and output language.

There were zero cases where max_new_tokens=4096 was reached, and no repetition loops occurred.
Only Edge produced unstable output language, with different vocabulary items such as Yes, yes, 無, 不存在, and 不明 output for the same prompt.

Investigation of Judgment Confidence

The correct/incorrect answer for this task is determined by the first generated token (since it is either Present or Absent). We thought that by recording the probability of that token, it could be quantitatively interpreted as the confidence level of the judgment decision, and for example could be used in a human review workflow. Here are the results for the 7 clips where "Absent" is the correct answer.

With Cosmos3-Nano, the values drop for VEG3, which is the most difficult judgment, and for NC-C1~C3, which are the closest in appearance to having feeding present among the negative controls, so at first glance it seems potentially usable, but with other models there doesn't appear to be a relationship with the fact of no feeding being present.

Here is the range of values for each model across the 16 clips.

We would like to increase the number of data points and investigate whether the values are interpretable.

Summary

Regarding the task, in the experiment with 16 clips, Cosmos3-Nano answered all correctly. Also, for video data of approximately 10 to 15 seconds, the median cycle time on DGX Spark was 5.94 seconds. We believe it may currently be the best model for this task. We also confirmed that Qwen3-VL-8B-Instruct performs at a similar level.

For actual operation, it is important to increase the number of data points, centering on clips without feeding, to improve reliability.
Statistically, the 95% upper confidence limit at 0 observations (Clopper-Pearson, 1 - 0.05^(1/n)) is 77.6% at n=2. In other words, when there are only 2 data points, even if no problems happened to occur, statistically "there is a possibility of occurring with up to approximately 78% probability."
This time we intentionally added negative control clips, but in practice, we believe it is important to first launch the pipeline with a minimum of about 30 clips, then actually increase the data points to 300 or more in the field while varying dates, workers, lots, locations, and so on, and proceed with adjustments.

Reference Information

• nvidia/Cosmos3-Nano — README.md
• nvidia/Cosmos3-Edge — README.md
• nvidia/Cosmos-Reason2-8B — Model Card
• Qwen/Qwen3-VL-8B-Instruct — generation_config.json
• QwenLM/Qwen3-VL — README.md
• huggingface/transformers Issue #48659 (InternVL video processing shape error)