I tried running a video search AI agent on DGX Spark (VSS Agent)

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In a previous article, I tried real-time camera footage analysis using Live VLM WebUI. The experience of having "what's currently visible" verbalized in 1-2 seconds was interesting, but in actual business operations, I think the need to "find specific scenes from past footage" is more common.

This time, I tried running NVIDIA's AI Blueprint "VSS (Video Search and Summarization) Agent" on DGX Spark. It's a video search AI agent where you feed it recorded footage and ask in natural language "where is the scene with someone in a red outfit?" and it finds the relevant section.

If Live VLM WebUI is a tool that "verbalizes camera footage in front of you in real time," then VSS is a tool that "searches and summarizes accumulated footage after the fact." This time, I tried running both real-time analysis and accumulated video search locally on DGX Spark.

What is VSS Agent

VSS (Video Search and Summarization) Agent is a video AI Blueprint based on the NVIDIA Metropolis platform. When you upload footage, it automatically generates captions using a VLM and stores them in a vector database and graph database. After that, you can search the video content using natural language or generate summaries.

What it can do

Once you upload footage, you can use natural language queries like "What is happening in this scene?" or text searches like "Scene where a person wearing a red hat is walking." It also supports generating summaries of the entire video, audio transcription (multilingual support), and action recognition (detection of specific actions).

Architecture

Behind VSS, multiple AI models and databases work together.

It adopts Cosmos-Reason2-8B (NVIDIA's latest video understanding model, based on Qwen3-VL) as the VLM, and the LLM can be flexibly swapped via an OpenAI-compatible API (default is Llama 3.1 8B). For search, CA-RAG (Context-Aware RAG) combining Milvus (vector DB) and Neo4j (graph DB) is used, and a key feature is the ability to search while considering spatiotemporal relationships.

Two Modes

VSS has two deployment modes.

Event Reviewer is a lightweight mode for "verifying alerts generated by CV pipelines (object detection, etc.) with a VLM." Standard VSS is full-featured, with all video search, summarization, and Q&A capabilities available. I focused on Standard VSS for this verification.

Setup

Prerequisites

Environment Setup

Clone the VSS GitHub repository and deploy with a single GPU configuration for DGX Spark.

git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization

On DGX Spark (ARM64), the IS_SBSA=1 flag is required.

# Run cache cleaner (recommended for ARM environment)
sudo sh deploy/scripts/sys_cache_cleaner.sh

# NGC login
docker login nvcr.io

The NGC API Key can be obtained from the NGC setup page. The HuggingFace Token is available from Settings.

export NGC_API_KEY="your-ngc-api-key"
export HF_TOKEN="your-hf-token"

Starting NIM Containers

VSS uses three NIM containers: LLM, Embedding, and Reranker. The official documentation introduces a procedure to start all three with NIM, but this time I replaced only the LLM with Ollama.

Since the LLM part of VSS simply calls an OpenAI-compatible API, swapping the endpoint means it works with inference servers other than NIM. Ollama natively provides an OpenAI-compatible API (/v1/chat/completions), so all you need to do is change the base_url in config.yaml.

# Prepare Llama 3.1 8B with Ollama
ollama pull llama3.1:8b

Start Embedding and Reranker with NIM as-is. These two are compact at about 3GB each and don't cause memory issues.

export LOCAL_NIM_CACHE=/tmp/nim-cache
mkdir -p $LOCAL_NIM_CACHE

# Embedding NIM (supports 26 languages, including Japanese)
docker run -d --name vss-embedding-nim \
  -u $(id -u) --gpus '"device=0"' --shm-size=16GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8006:8000 \
  nvcr.io/nim/nvidia/llama-3.2-nv-embedqa-1b-v2:1.9.0

# Reranker NIM (supports 26 languages, including Japanese)
docker run -d --name vss-reranker-nim \
  -u $(id -u) --gpus '"device=0"' --shm-size=16GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8005:8000 \
  nvcr.io/nim/nvidia/llama-3.2-nv-rerankqa-1b-v2:1.7.0

The first startup of NIM containers involves model downloads, so it takes a fair amount of time. Monitor the progress with docker logs -f vss-embedding-nim while waiting.

Pointing the LLM Endpoint to Ollama in config.yaml

In VSS's configuration file config.yaml, point the LLM endpoint to Ollama. The three locations to change are chat_llm, summarization_llm, and notification_llm.

chat_llm:
  type: llm
  params:
    model: llama3.1:8b
    base_url: 'http://host.docker.internal:11434/v1' # Ollama
    max_tokens: 2048
    temperature: 0.2
    top_p: 0.7

host.docker.internal is a DNS name for accessing services on the host from inside a Docker container. Since Ollama listens on port 11434 on the host side, VSS's container can connect via this URL.

Deploying VSS Itself

Once you confirm that the Embedding/Reranker NIMs return ready and the LLM responds via Ollama, start the VSS core.

cd deploy/docker/local_deployment_single_gpu

# Set NGC_API_KEY and HF_TOKEN in .env
source .env

# ARM64 flag (required for DGX Spark)
export IS_SBSA=1

docker compose up -d

The key point for DGX Spark (ARM64) is that the VSS core image name gets a -sbsa suffix. Setting IS_SBSA=1 causes vss-engine:2.4.1-sbsa to be selected inside compose.yaml.

On first startup, the download of Cosmos-Reason2-8B (VLM) will run. Since this is a gated model on HuggingFace, you need to agree to the license on the Cosmos-Reason2-8B page in advance.

Once startup is complete, access http://<DGX Spark IP>:9100 in your browser.

Running with the Default Configuration (Llama 3.1 8B)

Uploading Sample Footage

First, I'll use the sample footage included in the VSS repository (warehouse surveillance footage warehouse.mp4) to verify the overall flow. When you upload an mp4 file from the "FILE SUMMARIZATION" tab in the Web UI, Cosmos-Reason2-8B (VLM) automatically generates captions (text descriptions) for each frame and stores them in Milvus and Neo4j as GraphRAG.

Generating a Summary

After the upload is complete, press the "Summarize" button. The flow is that the VLM generates captions per chunk, then an LLM (Llama 3.1 8B) summarizes and aggregates them. For the warehouse footage, processing completed in 113 seconds.

The generated summary was organized into four categories.

• Unsafe Behavior — Unstable working posture on a ladder, failure to wear safety harness
• Operational Inefficiencies — Waiting time between workers
• Potential Equipment Damage — Insufficient safety checks around forklifts
• Unauthorized Personnel — Entry without protective equipment

Since the default prompt is configured for warehouse monitoring, the summary is well-organized from a safety management perspective.

Asking Questions in English

In addition to summaries, you can ask questions about the video content in chat format.

Q: "What safety violations were detected?"

Based on the English captions generated by the VLM and GraphRAG search results, the LLM returns an answer. The content was consistent with the summary, including failure to wear PPE on the ladder and insufficient safety checks around forklifts.

Asking Questions in Japanese

I also tried asking questions in Japanese about the same footage.

Q: "この動画で確認された安全上の問題は何ですか？" (What safety issues were identified in this video?)

Even with Llama 3.1 8B, a response in Japanese comes back. However, it was limited to basic observations such as "working in an unstable posture on a ladder" and "failure to wear protective equipment," with less information compared to the English response. Since VLM captions are generated in English, there is a gap with Japanese queries.

Trying Japanese Search with Nemotron 9B

This is what I most wanted to try in this article. The default VSS configuration is English-centric, but DGX Spark has a Japanese LLM verified in the Nemotron series article. How much more usable would Japanese video search become if the LLM were swapped?

Rewriting config.yaml

Since Nemotron 9B-v2-Japanese is already running in Ollama, all it takes is changing the model name in config.yaml. Rewrite the three locations: chat_llm, summarization_llm, and notification_llm.

 chat_llm:
   type: llm
   params:
-    model: llama3.1:8b
+    model: nemotron-9b-jp-nothink
     base_url: "http://host.docker.internal:11434/v1"

After saving config.yaml, restart the VSS via-server container. There is no need to re-upload the footage (the GraphRAG data remains as-is).

Comparison with Warehouse Footage

Running Summarize on the same warehouse.mp4 produced a clear difference in results.

While Llama 3.1 8B's output was a category-level overview like "Unsafe Behavior" and "Operational Inefficiencies," Nemotron 9B listed each event individually with timestamps and specific situations, such as "0:08-0:10: Worker in unstable posture on a ladder." Processing time is about 3 times longer, but there was a considerable difference in information granularity.

The difference in Japanese Q&A was even more pronounced.

Q: "この動画で確認された安全上の問題は何ですか？" (What safety issues were identified in this video?)

Testing with Free Stock Footage

Since testing only with warehouse footage might mean "the compatibility happened to be good," I also tried with free stock footage downloaded from Pexels.

Intersection Footage

After uploading footage of a busy intersection (38 seconds), the initial result was "no anomalies." The cause was the prompt. With the default warehouse-oriented prompt, it looks for "abnormal behavior inside a warehouse," so nothing comes up for traffic footage.

In the VSS Web UI, you can edit three types of prompts when running Summarize.

"What counts as an anomaly" is defined through these three levels of prompts, so it can be customized to suit the industry or monitoring target.

After rewriting the prompt for traffic monitoring, it properly detected traffic violations such as running red lights and lane departures. Processing time was 178 seconds.

VLM caption generation describes footage fairly generically, but at the LLM summarization/aggregation stage, if the prompt doesn't fit the domain, the result can be "zero detections." The interesting thing about VSS is that different insights can be drawn from the same footage depending on the prompts.

Motorcycle Factory Footage

For manufacturing line footage (70 seconds), I verified it with prompts adjusted for factory use.

Processing time was 648 seconds (about 10 minutes for 70 seconds of footage). With more chunks, the aggregation of the summary takes more time.

When I asked in Japanese Q&A "Are there any quality control concerns?" Nemotron 9B responded as follows:

It has been confirmed that after engine parts are placed on the conveyor belt, no visual inspection is performed. (...) From the fact that no visible sensors for real-time monitoring are installed on the conveyor belt, a lack of an automated quality control system can be inferred.

It provided specific observations such as the lack of visual inspection and the absence of sensors, using terminology appropriate to the manufacturing context. This is where Nemotron 9B's Japanese language capability shines.

On the other hand, for general questions like "What tasks are the workers performing?", responses sometimes came back in English. Since VLM captions are stored in English, the LLM's output language fluctuates depending on the ratio of search results. It should stabilize if the VLM captions are also Japanized, but the quality of Japanese captions from Cosmos-Reason2-8B is a topic for future investigation.

Comparison with Live VLM WebUI

Let me compare Live VLM WebUI and VSS as video AI tools running on the same DGX Spark.

A natural combination would be "real-time monitoring with Live VLM WebUI while accumulating recordings in VSS for later searching." Live VLM WebUI is for when you want to instantly know "what's happening now," and VSS is for when you want to "find a specific moment from past footage." However, running both simultaneously on DGX Spark's 128GB unified memory is quite tight, so switching between them based on the use case is more practical.

Thinking About Use Cases

If a video AI agent can run locally on DGX Spark, it seems usable for video analysis even in environments with strict privacy or network requirements.

In the verification with motorcycle factory footage, we actually obtained observations such as "safety glasses not worn," "lack of visual inspection," and "absence of a real-time monitoring system." The use case of accumulating factory camera footage in VSS and searching in Japanese for "deviations from work procedures" and "failure to wear safety equipment" seems sufficiently practical. Even in environments where footage cannot be sent outside, placing a DGX Spark at the line side means everything runs locally.

Beyond manufacturing lines, it seems broadly applicable to any situation where you want to "search accumulated footage with natural language," such as searching meeting recordings (in combination with audio transcription) or retail store security camera analysis.

Bonus: Testing Live Stream Functionality Without a Camera

Up to this point, I've been testing video search via file uploads, but VSS has another feature called "LIVE STREAM SUMMARIZATION." When you connect an RTSP stream, it splits the footage into real-time chunks and performs caption generation and summarization.

No surveillance camera RTSP stream is needed. By pseudo-broadcasting recorded footage as RTSP, you can try it on DGX Spark alone. Here, I used MediaMTX (a lightweight RTSP server) and FFmpeg to loop-broadcast the same warehouse.mp4 from earlier.

Setup

Start MediaMTX with Docker and use FFmpeg to serve the file as RTSP.

# Start MediaMTX (RTSP server)
docker run --rm -d --name mediamtx --network host bluenviron/mediamtx:latest

# Loop-broadcast warehouse.mp4 at real-time speed
ffmpeg -re -stream_loop -1 \
  -i ~/videos/vss-test/warehouse.mp4 \
  -c copy \
  -f rtsp rtsp://localhost:8554/warehouse

-re is a flag to read at real-time speed. Without it, the file is sent at full speed, which doesn't work as a live stream. Using -c copy for H.264 passthrough without transcoding means CPU load is almost zero.

Register the stream with the VSS API.

curl -X POST http://localhost:8100/live-stream \
  -H "Content-Type: application/json" \
  -d '{
    "liveStreamUrl": "rtsp://host.docker.internal:8554/warehouse",
    "description": "Warehouse Safety Monitoring RTSP Loop",
    "camera_id": "camera_1"
  }'

host.docker.internal is the DNS name for VSS's container to reach the host-side MediaMTX.

Live Stream Processing

After registering the stream, calling the /summarize API causes the VLM (Cosmos-Reason2-8B) to start generating captions in 10-second chunks. The design is that once 30 seconds worth of chunks have accumulated, a summary is aggregated.

Looking at the actual logs, captions were being generated per chunk like this:

Chunk 6: "Between the timestamps 68.8 and 70.8, an individual wearing safety gear exits through the restricted zone marked by caution tape without authorization."

Chunk 11: "Between 114.8s and 120.9s, a person is carrying two boxes but drops one while walking through the aisle, posing a safety hazard due to potential injury from falling objects."

In the 30-second summaries, these chunks are aggregated into category-based reports like "Worker is not wearing any safety equipment and is handling materials without proper protective gear." Although it's the same footage as the file upload version, a characteristic of live streams is that NTP timestamps are attached.

Differences from File Upload

Processing the same warehouse.mp4 with two methods reveals clear differences in the processing model.

The large difference in Q&A response time occurred because the VLM is constantly running chunk processing during live streams. Since it shares VLM resources with Q&A, the Q&A is queued behind during streaming. With file uploads, VLM resources free up after ingestion is complete, so Q&A returns in about 10-20 seconds.

In practical terms, live streams seem best suited for "automatic summary and alert generation" as the main use case. For real-time Q&A, solutions like adding a dedicated GPU for the VLM or widening the chunk processing interval would be necessary.

Summary

I deployed VSS Agent on DGX Spark and tried everything from searching and summarizing recorded footage, to swapping in a Japanese LLM, to the live stream feature. DGX Spark's 128GB unified memory can't be called spacious, and I saved about 40GB of memory by switching the LLM from NIM to Ollama. Thanks to VSS's design of standardizing on an OpenAI-compatible API, this switch only required changing the base_url in config.yaml.

It was a pleasant discovery that simply swapping the LLM to Nemotron 9B-v2-Japanese visibly improved the granularity of summaries and the quality of Japanese Q&A. On the other hand, domain adaptation of prompts also greatly affects detection accuracy. Processing intersection footage with the default warehouse-oriented prompt results in "no anomalies," but rewriting it for traffic monitoring enables violation detection. Since VLM captions remain in English, the response language fluctuates depending on the type of question — this remains a future challenge alongside Japanization on the VLM side.

While the previous Live VLM WebUI was a tool that verbalizes "what is currently visible" in real time, VSS filled in the piece of "searching past footage using text." The fact that both run locally on the same DGX Spark seems especially valuable in environments where footage cannot be sent outside.

Verification Environment

Reference Links

• VSS Agent Playbook
• VSS Official Documentation
• VSS GitHub
• VSS ARM Deployment Guide
• Cosmos-Reason2-8B (HuggingFace)
• NVIDIA Metropolis
• Scripts from this article (GitHub)