I tried real-time analysis of camera footage with the Live VLM WebUI on DGX Spark
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
Up until now, I've been using DGX Spark primarily for text-based applications such as local code completion and integration with Claude Code, but this time I'd like to try something a little different and work with "video."
I used the Live VLM WebUI from Playbooks to analyze webcam footage with a VLM (Vision-Language Model).
What is Live VLM WebUI?
Live VLM WebUI is a tool included in the Multimodal/Vision category of DGX Spark Playbooks. It analyzes video from a camera in real time using a VLM and displays the results in a browser.
You can choose from multiple backend engines, including Ollama, vLLM, SGLang, and NVIDIA NIM. What's interesting is that by customizing the prompt, you can ask various questions about the same camera footage from different angles, such as object detection, scene recognition, OCR, and safety monitoring.
Normally, it accesses the camera directly via the browser's WebRTC API and also supports RTSP stream input (Beta). This time I tested it with webcam input via WebRTC.
Incidentally, NVIDIA has also released an AI Blueprint called "Video Search and Summarization (VSS)" specialized for searching and summarizing recorded footage. Live VLM WebUI, which verbalizes "what is currently visible" in real time, and VSS, which "searches for specific scenes from past footage" — you can think of them as tools to be used depending on the purpose.
Configuration for This Time
This is a configuration that runs entirely within the local network. For this test, I used Ollama as the backend and compared two models: gemma3:4b and llama3.2-vision:11b.
Setup
Preparing the VLM Model
First, prepare a VLM model with Ollama. It's recommended to start with the lightweight gemma3:4b to verify operation.
ollama pull gemma3:4b
ollama pull llama3.2-vision:11b # For comparison
Installing Live VLM WebUI
You can install it without polluting the system Python environment using uv tool install.
uv tool install live-vlm-webui
In v0.2.1, 43 packages are installed, and two commands become available: live-vlm-webui and live-vlm-webui-stop.
Starting Up
Start with Ollama specified as the backend.
live-vlm-webui --model gemma3:4b --api-base http://localhost:11434/v1 --port 8090
Upon startup, a self-signed SSL certificate is automatically generated, and the DGX Spark's GPU (NVIDIA GB10) is also automatically detected.
Initialized VLM service:
Model: gemma3:4b
API: http://localhost:11434/v1 (Local)
Starting server on 0.0.0.0:8090
Auto-detected NVIDIA GPU (NVML available)
Detected NVIDIA DGX Spark (Version 7.2.3)
Access https://<DGX Spark IP>:8090 in your browser to open the WebUI.
Trying It Out
Pointing the Camera at the Desk
I pointed the webcam at the desk and asked the VLM "what is visible in this footage." On the desk, I arranged a DGX Spark and its box, a mechanical keyboard, Ghana chocolate, and a Xiaomi monitor.
With a single-sentence English prompt (Describe what you see in this image in one sentence.), this output was returned:
A NVIDIA DGX Spark monitor is sitting on a desk next to a Xiaomi laptop, a grey keyboard, and a box of Ghana Milk Chocolate.
The DGX Spark box is misidentified as a "monitor," but the Xiaomi monitor, keyboard, and Ghana chocolate are correctly identified. Latency was approximately 1,100ms. For real-time analysis, this level of accuracy in just over a second is sufficiently practical.
Changing the Prompt
I tried changing the prompt for the same scene.
When switching to object enumeration (List all objects visible in this image.), the output changes to individually describing each object.
- NVIDIA DGX Spark Box: A green cardboard box with the NVIDIA logo.
- Lotte Ghana Milk Chocolate Bar: A rectangular chocolate bar with Japanese writing.
- Keyboard: A silver and grey mechanical keyboard.
- Computer Monitor: A black Xiaomi monitor.
- Wall Decor: A leafy, vertical wall decoration monitor.
I was surprised that it even recognized the Japanese packaging of the Ghana chocolate.
It also responds naturally in Japanese with a Japanese prompt (この映像に映っているものを説明してください。).
- NVIDIA DGX Spark：NVIDIA のゲーミンググラフィックボードです。
- キーボード：白色のメカニカルキーボードです。
- Lotte Ghana Chocolate：ロッテのミルクチョコレートの箱です。
- Xiaomi モニター：Xiaomi のモニターです。
- 背景には植物の模様の壁飾りが写っています。
It's a bit amusing that DGX Spark is described as a "gaming graphics board," but the overall recognition accuracy is high.
Inference Speed
I measured the latency with gemma3:4b. The setting queries the VLM once every 30 frames (approximately 1-second intervals).
| Prompt | Latency | Feel |
|---|---|---|
| Single English sentence | Approx. 1,100ms | Nearly real-time |
| English list | Approx. 2,000ms | A slight wait |
| Japanese | Approx. 1,100–2,500ms | Depends on output length |
The experience is like "AI verbalizing the situation every few seconds." It feels less like real-time video surveillance and more like periodic situation reports.
Comparing Models
I compared the output of gemma3:4b and llama3.2-vision:11b on the same desk scene.
The English output from llama3.2-vision:11b was more detailed than gemma3:4b, recognizing even the jacket in the background and the pattern on the curtains. However, when switching to a Japanese prompt, the output breaks down.
| Aspect | gemma3:4b | llama3.2-vision:11b |
|---|---|---|
| English accuracy | Accurately recognizes major objects | More detailed (9 items), picks up background |
| Japanese support | Outputs natural Japanese | Grammar breaks down, not practical |
| Latency (single English sentence) | Approx. 1,100ms | Approx. 3,000–4,000ms |
| GPU usage | 13–14% | Approx. 90% |
| Memory usage | Approx. 15 GB | Approx. 54 GB |
It seems that for real-time video analysis, bigger is not always better when it comes to model size. gemma3:4b has a good balance of speed, Japanese support, and resource efficiency, with GPU usage at only 13% relative to DGX Spark's 128GB memory. The remaining resources can be used to run other tasks simultaneously.
llama3.2-vision:11b is suitable for cases requiring detailed recognition in English only, but the latency being 3–4 times longer creates a significant trade-off with real-time performance.
Difference from Edge AI (YOLO)
Here I'll also summarize the differences from another approach. Edge devices that can execute YOLO-based object detection directly on the camera itself, such as Seeed Studio's ReCamera 2002w, are increasing. I actually had ReCamera's YOLO detect the same desk scene.
With ReCamera's YOLOTIn Detection, the mouse was correctly detected as "mouse," but Ghana chocolate was misclassified as "book." Since "chocolate" is not included in the COCO dataset's 80 classes, it was assigned to "book," which has a similar shape. No bounding boxes appeared for the DGX Spark box or the keyboard.
In contrast, gemma3:4b's VLM accurately described the same scene in natural language as "NVIDIA DGX Spark Box," "Lotte Ghana Milk Chocolate Bar," "Keyboard," and "Xiaomi Monitor." This difference is exactly what distinguishes the YOLO and VLM approaches.
| Aspect | Edge YOLO (ReCamera) | DGX Spark (VLM) |
|---|---|---|
| Output format | Bounding boxes + labels | Natural language sentences |
| Vocabulary | Fixed, e.g., COCO 80 classes | Unlimited |
| Context understanding | Individual object detection only | Grasps relationships in the entire scene |
| Custom questions | Not possible | Free to ask with prompts |
| Inference speed | Tens of ms | 1–4 seconds |
| Required hardware | Camera alone (USB powered) | GPU server |
It's not a question of which is better — they serve completely different purposes. If you want to quickly know "where is that object," use YOLO; if you want to ask "describe this situation in words," use VLM. YOLO is closer to eyes, and VLM is closer to a brain.
In a production environment, a pipeline that quickly filters with YOLO before passing to a VLM is also conceivable. For example, a camera in a factory corridor selects only "frames where a person appears," and then the DGX Spark side checks whether helmets and safety equipment are being worn. Since video never leaves the local network and everything is handled on-premises, this is an easy-to-prototype configuration even in environments with privacy restrictions.
Bonus
The Live VLM WebUI log records VLM responses with timestamps. By saving to a file with tee at startup, you can later track the chronological changes in recognition results.
live-vlm-webui --model gemma3:4b --api-base http://localhost:11434/v1 --port 8090 2>&1 | tee vlm-log.txt
As a test, I spent about 2 minutes putting objects in and out on the desk. The prompt was List all objects visible in this image. The following is an excerpt of recognition results extracted from the log.
18:25:41 Nvidia DGX Spark box, notebook, chocolate box (Lotte Ghana Milk Chocolate),
keyboard, computer monitor, desk, jacket. (1307ms)
18:25:59 Nvidia DGX Spark box, Keyboard, Mouse, Bottle of Kirin Lemon Tea,
Box of Japanese sweets (1253ms)
18:26:03 NVIDIA DGX Spark box, bottle of Kirin tea, keyboard, mouse, desk,
plant, plant pot, tea bag. (1538ms)
18:26:09 NVIDIA DGX Spark box, water bottle, LOTTE Ghana Milk Chocolate box,
keyboard, mouse, monitor, USB cable, reflection (1377ms)
18:26:33 Nvidia DGX Spark box, monitor, keyboard, mouse, desk, cardboard,
shadows (1181ms)
18:26:53 Nvidia DGX Spark box, computer monitor, computer keyboard,
computer mouse, two paper cups, plant, cables (1329ms)
18:27:41 Nvidia DGX Spark box, plush toy (blue whale), keyboard, mouse,
computer monitor, cable, gray speaker. (1366ms)
18:27:56 Box, Keyboard, Mouse, Phone, Computer monitor, Wall, Tree (1166ms)
18:28:05 NVIDIA DGX Spark box, bottle of tea, computer monitor,
computer keyboard, computer mouse, desk, water, background blur. (1351ms)
When a drink is placed, it recognizes "Bottle of Kirin Lemon Tea"; when a stuffed animal is placed, it recognizes "plush toy (blue whale)"; when a smartphone is brought out, it recognizes "Phone" — recognition updates within 1–2 seconds as objects are added or removed. It's interesting to see the limits of a 4B parameter model when the type of stuffed animal fluctuates between "blue whale," "plush shark," and "plush dolphin" from frame to frame.
On the other hand, objects that are consistently in the frame, such as the DGX Spark box and keyboard, continue to be recognized stably. Latency also stayed within the 1,100–1,500ms range, and about 2 minutes of analysis logged 58 frames.
By aggregating this log, you can also perform quantitative analysis such as "in which frames was a specific object recognized" and "how stable is the recognition." It seems fully usable as a simple video monitoring prototype.
Summary
I installed Live VLM WebUI on DGX Spark and tried real-time VLM analysis of camera footage.
The setup is easy. Installation is completed with a single uv tool install command, and it can be up and running immediately in combination with Ollama. With gemma3:4b, footage can be analyzed at approximately 1-second intervals, and it properly handles Japanese prompts as well. Since GPU usage stays at around 13%, the ability to share DGX Spark's 128GB memory with other workloads is a significant advantage.
On the other hand, while llama3.2-vision:11b can provide detailed recognition in English, its broken Japanese output and latency 3–4 times longer made it feel unsuitable for real-time use. Personally, discovering that "bigger model ≠ always better" was the most interesting finding this time.
VLMs handling video are evolving rapidly, so these limitations should change as model generations advance. There are all kinds of things to try depending on your ideas, such as combining with IP camera RTSP streams or integrating with edge-side YOLO.
Test Environment
| Item | Spec |
|---|---|
| DGX Spark | 128GB LPDDR5x, GB10 (Grace Blackwell) |
| Live VLM WebUI | v0.2.1 |
| VLM Model | gemma3:4b / llama3.2-vision:11b |
| Backend | Ollama |
| Camera | HD Pro Webcam C920 (1080p) |
| Comparison Device | Seeed Studio ReCamera 2002w (YOLO) |
