I tried the Warehouse Blueprint of VSS 3.0.0 EA on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In my previous article, I tried running the VSS (Video Search and Summarization) Agent on DGX Spark, covering the basic usage of the video search AI and swapping in a Japanese LLM. By switching to Nemotron 9B-v2-Japanese, I confirmed that the quality of Japanese Q&A improved considerably.

My previous validation was on 2.4.x, but VSS already has a 3.0.0 Early Access release available. The architecture has been revamped from monolithic to microservices, and quite a lot has changed internally, including MCP-based orchestration, industry-specific Blueprints, and integrated video and text Embeddings (Cosmos-Embed1).

Since it's EA (Early Access), production use is not recommended — but that's precisely why I thought there would be value in reporting "what works at this stage and what has changed," so I decided to try it out early on DGX Spark.

VSS 2.4.x and 3.0.0 — What Changed

First, let me summarize what changed between the previous 2.4.x and the current 3.0.0 EA.

Architecture Overhaul

2.4.x had a monolithic configuration, where a single VSS Engine container handled everything from video ingestion to retrieval. In 3.0.0, this has been decomposed into 3 layers of microservices.

The RTVI (Real-Time Video Intelligence) layer handles real-time video processing, the analytics layer performs behavior analysis and calibration, and the agent layer orchestrates everything using MCP.

Key Changes

In 2.4.x, replacing Embeddings or Rerankers required building a custom NIM-compatible proxy, and the CV pipeline had to be set up separately — there were parts that were difficult to swap out. The microservice architecture in 3.0.0 is moving in a direction that fundamentally resolves these constraints.

Industry-Specific Blueprints

3.0.0 offers three industry-specific Blueprints.

All of them are reference implementations built on top of the common VSS core platform, with industry-specific detection models, behavior analysis logic, and agent configurations layered on top. This time I'll use the "Warehouse Blueprint" for warehouses. It is officially supported on DGX Spark, and according to NVIDIA's published performance measurements, E2E latency is 102ms at 4 streams 30fps. This is a reference value, but it's faster than IGX Thor (222ms).

The following 4 profiles are available.

This time I'll use the Warehouse Operations Blueprint, "2D Vision AI with Agents" (bp_wh) profile. It's a full-stack configuration that covers everything from object detection with DeepStream, to event detection with Behavior Analytics, all the way to natural language queries via VSS Agent.

Preparing to Deploy on DGX Spark

Prerequisites

I checked the environment required to run 3.0.0 EA on DGX Spark.

NGC CLI was the only thing not installed on DGX Spark, so I downloaded the ARM64 version from the NGC official site and installed it.

# Installing NGC CLI (ARM64)
curl -sL "https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.14.0/files/ngccli_arm64.zip" \
  -o /tmp/ngccli_arm64.zip
unzip /tmp/ngccli_arm64.zip -d /tmp/ngc-cli-install
mkdir -p ~/.local/bin
cp -r /tmp/ngc-cli-install/ngc-cli ~/.local/ngc-cli
ln -sf ~/.local/ngc-cli/ngc ~/.local/bin/ngc

# Configuration
ngc config set

Fetching the NGC Package

Download the Warehouse Blueprint compose package from NGC.

ngc registry resource download-version \
  "nvidia/vss-warehouse/vss-warehouse-compose:3.0.0"

A 46.72MB package will be downloaded. When extracted, the structure looks like this.

vss-warehouse-compose_v3.0.0/
├── deployments/
│   ├── compose.yml           # Main Compose definition
│   ├── foundational/         # Kafka, Elasticsearch, Redis, etc.
│   ├── monitoring/           # Prometheus / Grafana
│   ├── vst/                  # Video Storage Tool
│   ├── warehouse/            # Warehouse Blueprint core
│   │   ├── .env              # Environment variables (edit this)
│   │   ├── warehouse-2d-app/ # 2D pipeline
│   │   └── vss-agent/        # VSS Agent configuration
│   ├── nim/                  # LLM / VLM NIM definitions
│   ├── agents/               # Agent + UI
│   ├── rtvi/                 # RTVI microservices
│   ├── lvs/                  # Long Video Summarization
│   └── auto-calib/           # Camera calibration
└── modules/                  # Utility scripts

In 2.4.x, you had to clone a GitHub repository and decipher the compose.yaml inside it, but in 3.0.0 it's organized as an NGC package. The compose.yml is split by module, with the main compose.yml loading each service via include.

.env Settings for DGX Spark

This was the most time-consuming part of this deployment. The .env file requires several DGX Spark-specific settings, and it won't work with the defaults.

# Hardware profile (change H100 → DGX-SPARK)
HARDWARE_PROFILE='DGX-SPARK'

# Use shared mode for single GPU
LLM_MODE=local_shared
VLM_MODE=local_shared

# Device ID (DGX Spark has GPU 0 only)
LLM_DEVICE_ID='0'
VLM_DEVICE_ID='0'

# Path settings
MDX_SAMPLE_APPS_DIR="/path/to/deployments"
MDX_DATA_DIR="/path/to/vss-warehouse-app-data"
HOST_IP='<DGX Spark IP address>'

# NGC API Key
NGC_CLI_API_KEY='<your-ngc-api-key>'

On DGX Spark (ARM64 / SBSA), container image tags also need to be changed. There were commented-out DGX-SPARK settings inside the .env file.

# Image tag for Perception (DeepStream)
PERCEPTION_TAG="3.0.0-sbsa"    # x86 uses "3.0.0"

# Image tags for VST containers (change all 7 to -sbsa)
VST_SENSOR_IMAGE_TAG="3.0.0-sbsa"
VST_RTSPSERVER_IMAGE_TAG="3.0.0-sbsa"
VST_RECORDER_IMAGE_TAG="3.0.0-sbsa"
VST_STORAGE_IMAGE_TAG="3.0.0-sbsa"
VST_REPLAYSTREAM_IMAGE_TAG="3.0.0-sbsa"
VST_LIVESTREAM_IMAGE_TAG="3.0.0-sbsa"
NVSTREAMER_IMAGE_TAG="3.0.0-sbsa"

DGX Spark Support Status of the EA Package

While reading through compose.yaml, I noticed that the EA package's DGX Spark support still has parts that require manual work. Although the official documentation states "DGX-SPARK: Supported," a few additional steps are needed to get it running out of the box.

Since it's at the EA (Early Access) stage, I expect these areas will be improved toward GA. This time, I used the DGX-THOR (same ARM64 / Grace Hopper architecture) settings as a reference and manually created hw profiles for DGX Spark.

Deployment

Starting Up

Once configuration is complete, start all services with compose up.

cd /path/to/vss-warehouse-compose_v3.0.0/deployments

docker compose -f compose.yml \
  --env-file warehouse/.env \
  up --detach --pull always --force-recreate --build

On the first run, 36 container images need to be pulled, so it may take 30 minutes or more depending on network speed. On DGX Spark, images with the -sbsa tag are automatically selected.

Troubleshooting Points

When actually starting up, a few additional steps were needed.

Registering NVIDIA Container Runtime

The NVIDIA runtime may not be registered with Docker daemon on DGX Spark. If you get an unknown or invalid runtime name: nvidia error, register it with the following commands.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Pre-creating Data Directories

Containers will fail to start if the bind mount target directories for Elasticsearch and Redis don't exist. I read through the compose volume definitions and pre-created the necessary directories.

mkdir -p /path/to/data/data_log/{elastic/{logs,data},kafka,redis/{data,log}}
mkdir -p /path/to/data/videos/nv-warehouse-4cams
mkdir -p /path/to/data/models

Directory Permissions

There were cases where Docker created directories as root, causing Redis to fail to start with a Permission denied error. This was resolved by fixing ownership and permissions with chown and chmod.

LLM NIM ARM64 Issue

This was the biggest stumbling point in this validation.

The VLM (Cosmos-Reason2-8B) NIM officially supports ARM64 and started without issues. However, the LLM (Nemotron-Nano-9B-v2) NIM fails to start with an exec format error.

exec /opt/nvidia/nvidia_entrypoint.sh: exec format error

The Docker multi-architecture manifest includes arm64/linux, and docker image inspect shows Architecture: arm64, but the binaries inside the container appear to have actually been built for x86. Even /bin/bash could not be executed.

This appears to be an issue with the EA-stage image build.

Using NGC vLLM as an Alternative

Since NIM wouldn't run, I used the vLLM container published by NVIDIA on NGC (nvcr.io/nvidia/vllm) as an alternative. This officially supports ARM64.

docker run -d --name vllm-nemotron-nano \
  --runtime nvidia \
  -p 30081:8000 \
  --shm-size 16g \
  -e NVIDIA_VISIBLE_DEVICES=0 \
  -e HF_TOKEN=$HF_TOKEN \
  nvcr.io/nvidia/vllm:26.01-py3 \
  python3 -m vllm.entrypoints.openai.api_server \
    --model nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
    --served-model-name nvidia/nvidia-nemotron-nano-9b-v2 \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --gpu-memory-utilization 0.2 \
    --max-model-len 16384 \
    --max-num-seqs 4

By setting --served-model-name to the same nvidia/nvidia-nemotron-nano-9b-v2 as NIM, you can connect directly without changing the VSS Agent configuration. Since the OpenAI-compatible API endpoint format is the same, it works as a drop-in replacement.

However, there is one caveat. NGC vLLM 26.02 requires CUDA 13.1 / Driver 590+, which causes a compatibility error with DGX Spark's Driver 580.126.09. It was necessary to use 26.01 (compatible with CUDA 13.0).

Startup Results

In the end, 41 compose services + the vLLM container, totaling 42 services, were running. ds-configurator exits after generating its configuration, so the number of actively running services is 41.

NAME                           STATUS
alert-bridge                   Up 35 minutes
bp-configurator-2d             Up 45 minutes (healthy)
cosmos-reason2-8b-shared-gpu   Up 39 minutes (healthy)
mdx-elastic                    Up 45 minutes (healthy)
mdx-kafka                      Up 45 minutes (healthy)
mdx-kibana                     Up 45 minutes (healthy)
mdx-nvstreamer-2d              Up 9 minutes
metropolis-vss-ui              Up 35 minutes
perception-sdr-2d              Up 45 minutes
vss-agent                      Up 35 minutes (healthy)
vss-auto-calibration           Up 45 minutes (healthy)
vss-behavior-analytics-2d      Up 45 minutes
vss-va-mcp                     Up 44 minutes (healthy)
vss-video-analytics-api-2d     Up 45 minutes
vss-video-analytics-ui         Up 44 minutes
vst-mcp-2d                     Up 44 minutes
# ... 25 other services omitted

GPU memory usage is as follows.

About half of the GB10's 128 GB unified memory is in use. In 2.4.x, VLM + LLM used around 20 GB, so 3.0.0 gives the impression of allocating memory with considerably more headroom. Memory consumption can be further reduced by adjusting --gpu-memory-utilization on the vLLM side.

Service Configuration

Starting with the bp_wh (2D with Agents) profile brings up the following services.

In 2.4.x there was just a Web UI (:9100) and the backend VSS Engine, but in 3.0.0 dedicated UIs are provided for each purpose.

Running It

Opening the UI

Once all services are running, you can access 3 web UIs. They're built on Next.js and feel quite polished compared to the simple UI in 2.4.x.

VSS Agent UI (:3000)

A ChatGPT-like interface displays "Hi, I'm Warehouse Agent." It has a 4-tab layout — Chat / Alerts / Dashboard / Video Management — and also supports drag-and-drop video uploads. The bottom left shows "Version 3.0-EA."

Video Analytics UI (:3002)

A video analytics dashboard based on NVIDIA Metropolis. You can view DeepStream detection results in real time.

Kibana (:5601)

An Elasticsearch dashboard used for visualizing event data detected by Behavior Analytics.

Chatting with the Agent

I queried the Agent through the VSS Agent UI chat field and via API.

curl -s -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello, can you describe what you can do?"}]}'

The Agent returned the following response.

I can help with several tasks related to warehouse video surveillance:

1. Sensor/Camera Information
2. Occupancy Monitoring
3. Snapshots/Pictures
4. Incident Reporting

In 2.4.x, the main features were vector search and Q&A over video, but in 3.0.0 more practical capabilities are provided via MCP tools, such as sensor management, occupancy monitoring, and incident report generation.

However, one problem arose here.

The Tool Calling Hurdle and How It Was Resolved

The VSS Agent uses the LLM's tool calling feature internally to invoke MCP tools. The first stumbling block was a mismatch between Nemotron-Nano-9B-v2's tool call format and vLLM's built-in parser.

Nemotron uses an array format <TOOLCALL>[{...}, {...}]</TOOLCALL>, but vLLM's hermes parser expects individual tag format <tool_call>{...}</tool_call>. Without any fix, the raw TOOLCALL tags were returned as-is in the response.

The solution was to apply a custom template using vLLM's --chat-template option. Based on the model's default template, I unified the tag name to <tool_call> and modified it to output tool calls as individual tags rather than as an array.

# Start with a custom template specified
python3 -m vllm.entrypoints.openai.api_server \
  --model nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
  --chat-template /tmp/chat_template.jinja \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  ...

With this fix, VSS Agent's tool integration started working correctly. When asked "Show me available sensors," the Agent automatically calls the get_sensor_ids tool and returns a list of 5 camera streams.

Here are the available sensors:
1. warehouse_cam4
2. warehouse
3. warehouse_cam3
4. warehouse_cam1
5. warehouse_cam2

Using the combination of NGC vLLM + custom template, I confirmed that the full-stack Agent functionality including tool calling works even in environments where NIM is unavailable.

Restoring the Perception Pipeline

The Perception pipeline (object detection and tracking) via DeepStream didn't work on first startup. Checking the error logs, there was a message saying the object detection model couldn't be found.

ERROR: Cannot access ONNX file '/opt/storage/rtdetr_warehouse_v1.0.fp16.onnx'
ERROR: failed to build network since parsing model errors.

Upon investigation, the startup script ds-start.sh is designed to copy models/mtmc/*.onnx from the base image to the working directory, but this directory itself didn't exist in the Perception base image. Only an ITS (Intelligent Traffic Systems) model (resnet50_market1501.etlt) was present in models/rtdetr-its/, and the Warehouse-specific RT-DETR ONNX model didn't appear to be bundled yet. Since it's at the EA stage, I expect this will be addressed going forward.

However, the model itself was published in the NGC catalog. It's called nvidia/tao/rtdetr_2d_warehouse, a model trained with TAO Toolkit using an RT-DETR + EfficientViT/L2 backbone. It's a warehouse-specialized model that detects 7 classes (Person, Humanoid ×2, Nova Carter, Transporter, Forklift, Pallet), provided under the NVIDIA Open Model License (commercially usable).

# Download the RT-DETR Warehouse model from NGC
ngc registry model download-version \
  nvidia/tao/rtdetr_2d_warehouse:deployable_efficientvit_l2_v1.0

# Place the FP16 ONNX model (136MB) in the mount directory
cp rtdetr_2d_warehouse_vdeployable_efficientvit_l2_v1.0/rtdetr_warehouse_v1.0.fp16.onnx \
  $MDX_DATA_DIR/models/mtmc/

# Fix permissions (read access is needed as container runs as UID 1000)
chmod 644 $MDX_DATA_DIR/models/mtmc/rtdetr_warehouse_v1.0.fp16.onnx

# Restart the Perception container
docker restart perception-2d

On first startup, TensorRT engine building runs. Looking at the logs, an explicit FP16 flag build first failed, then fell back to strongly typed mode (using the FP16 operations embedded in the model as-is) and succeeded. Since cases where running RT-DETR in FP16 on DeepStream causes LayerNorm overflow producing nan have been reported on the forums, the automatic fallback to strongly typed mode was actually a convenient result.

After the engine build completed, the pipeline ran successfully.

** INFO: <bus_callback:623>: Pipeline running
**PERF:
30.00000 (31.19770)  source_id : 0 stream_name warehouse

It's processing the warehouse stream at 30fps. This matches the DGX Spark performance listed on the NGC model card (supporting 3 streams at 30fps).

Note that this Perception pipeline is a newly added layer in 3.0.0. Whereas 2.4.x processed video understanding with VLM alone, the 3.0.0 Warehouse Blueprint uses a hybrid configuration that adds a DeepStream CV pipeline for real-time object detection and tracking on top of the VLM.

What Worked, What Didn't

Here's a summary of the verification results:

The LLM NIM ARM64 image and Alert Bridge configuration each required workarounds, but through alternatives and patch application, we achieved an almost fully operational full-stack result.

Overview of UI and Pipeline

Let's look at the relationship between the actually deployed services using the official architecture diagram. The configuration has browsers accessing each UI from the front, while behind the scenes the Perception → Behavior Analytics → Elasticsearch data pipeline keeps running at 30fps.

Source: NVIDIA VSS 3.0.0 Warehouse Blueprint - 2D Vision AI with Agents Profile

Video flows from the Input Source (NvStreamer) on the left through VIOS to DeepStream (Perception), and detection metadata is passed to Behavior Analytics via Kafka. Analysis results accumulate in the ELK Stack (Elasticsearch + Kibana) and can be referenced from the UIs on the right.

The VSS UI (:3000) is the main entry point. When you ask the Warehouse Agent a question from the Chat tab, the Agent pulls data from Elasticsearch via VA-MCP and generates a response with the LLM. The Dashboard tab embeds Kibana in an iframe, visualizing Perception detection data in real time.

MCP Server Configuration

One of the headline features of 3.0.0 is MCP (Model Context Protocol)-based orchestration. The VSS Agent internally communicates with the VA-MCP Server (Video Analytics MCP Server) to retrieve video analysis results.

Looking at the VSS Agent's config.yml, two MCP clients are defined: video_analytics_mcp and vst_mcp.

function_groups:
  video_analytics_mcp:
    _type: mcp_client
    server:
      transport: streamable-http
      url: ${VIDEO_ANALYSIS_MCP_URL}/mcp
    include:
      - video_analytics.get_incidents
      - video_analytics.get_incident
      - video_analytics.get_fov_histogram
      - video_analytics.get_sensor_ids

  vst_mcp:
    _type: mcp_client
    server:
      transport: streamable-http
      url: ${VST_MCP_URL}/mcp

Whereas 2.4.x statically defined pipelines in config.yaml, in 3.0.0 functionality is exposed as MCP tools. The agent determines which tools to use based on the situation.

Running the VLM-as-Verifier Pipeline

Up to this point, we covered basic deployment and operational verification of each component. From here, we'll try running the VLM-as-Verifier pipeline, the centerpiece of the 3.0.0 Warehouse Blueprint.

The Two-Stage Detection Mechanism

Safety monitoring in the Warehouse Blueprint uses a two-stage detection design.

1. Rule-based detection (Behavior Analytics): Detects intrusions into ROIs (regions of interest) and tripwires based on DeepStream object detection results. Generates incident candidates based on thresholds
2. VLM visual verification (Alert Bridge → Cosmos-Reason2-8B): For incident candidates, passes video from the relevant timestamp to the VLM to visually verify "is this really an incident?"

It's a hybrid configuration where CV broadly and quickly narrows down candidates, and the VLM deeply and accurately verifies them.

Metadata detected by Perception at 30fps is passed to Behavior Analytics via Kafka and sent again to Kafka as incident candidates. Alert Bridge receives these, deduplicates them with Redis, retrieves the relevant video from VST, and performs visual verification with the VLM (Cosmos-Reason2-8B). The verification results accumulate in Elasticsearch's mdx-vlm-incidents index.

Incident Detection Configuration

To generate incidents in Behavior Analytics, you configure ROIs and tripwires in calibration.json. This time we used the official sample calibration and set up a tripwire in the warehouse aisle.

# Configure via Calibration UI (:8003), or edit JSON directly
# Restart Behavior Analytics after configuration
docker restart vss-behavior-analytics-2d

When calibration is applied while the Perception pipeline is running at 30fps, an incident is sent to Kafka every time a person crosses the tripwire. This can be confirmed in real time on the Kibana Dashboard.

Pain Points in the VLM Pipeline

However, even though Behavior Analytics was generating incidents, VLM analysis results were not reaching Elasticsearch. This is where the EA-style debugging begins. We tackled each cause one by one.

Invalid VST Stream ID Header

When Alert Bridge retrieves video from VST, there is code that attaches a streamId to the HTTP header, but this header was causing VST to return 503.

- headers = {"streamId": stream_id}
- response = await client.get(url, headers=headers)
+ response = await client.get(url)

It seems streamId should be passed as a URL parameter, but it was being sent twice as a header.

Uninitialized video_url

In Alert Bridge's enhance_alert_with_vlm.py, when the video URL could not be retrieved from VST, the video_url variable was passed to subsequent processing undefined, causing it to crash with UnboundLocalError.

+ video_url = None
  try:
      video_url = await get_video_from_vst(...)
  except Exception as e:
      logger.error(f"Failed to get video: {e}")

A classic variable scope issue — it was probably only tested against the happy path.

VLM Endpoint Misconfiguration

In Alert Bridge's config.yaml, the VLM endpoint was pointing to an external IP (the default value), causing connection timeouts. The model name was also still set to the old cosmos-reason1-7b.

vlm:
  base_url: "http://localhost:30082/v1"  # ← container uses host network
  model: "nvidia/cosmos-reason2-8b"
  num_frames: 5

num_workers was also changed from the default of 10 to 1. On a single-GPU environment, it's more stable to limit VLM parallel processing.

Stream ID Resolution Failure Due to Deleted Sensors

When trying to generate an incident report from the VSS Agent, the sensor list returned by the VST MCP's sensor_list API included old sensors with state: "removed", and when the Agent searched by the name "warehouse," it would pick up the stream ID of a deleted sensor.

  for sensor in result:
      if isinstance(sensor, dict) and "sensorId" in sensor:
+         if sensor.get("state") == "removed":
+             continue
          sensor_id = sensor["sensorId"]
          sensor_objects[sensor_id] = sensor

The cause was that sensors deleted when changing from a 4-camera to a 1-camera configuration kept persisting in the API response.

Additionally, Redis dedup keys (TTL 300 seconds) remained even after restart, blocking reprocessing of the same incidents. This was resolved by manually deleting vlm:warehouse:* keys with redis-cli.

VLM Analysis Results

After applying the four patches, the VLM-as-Verifier pipeline started working. Cosmos-Reason2-8B began analyzing warehouse video, and results accumulated in Elasticsearch's mdx-vlm-incidents index.

Inference time was approximately 19–30 seconds per incident. Five frames are extracted and passed to the VLM, which analyzes PPE (personal protective equipment), helmet usage, and worker behavior. The VLM even read shelf labels (C–F) and included location information in the report.

Incident Report Generation by Agent

For incidents verified by VLM-as-Verifier, reports can be generated from the VSS Agent's Chat UI.

Generate a report for incident <incident_id> with sensor id warehouse.

The Agent internally calls multiple MCP tools to collect detailed incident information, snapshots at the relevant time, and video clips to generate the report.

Generated reports can be downloaded in Markdown / PDF format and include links to video snapshots and clips. The VLM determined the incident type as "Box Retrieval" and described in detail the situation of a worker retrieving cardboard boxes near shelf D. It's a structured report that even includes floor conditions (smooth concrete) and lighting conditions (bright artificial lighting).

In 2.4.x, vector search of video and natural language Q&A were the main features. In 3.0.0, the safety monitoring workflow of rule-based detection → VLM verification → structured report generation is integrated via MCP, representing evolution in a more practical direction.

Swapping in a Japanese LLM

Since we're using NGC vLLM instead of NIM this time, we can switch to any model on HuggingFace simply by changing the --model parameter. As a bonus, we tried swapping in the Nemotron 9B-v2-Japanese we used in V1 as well.

docker run -d --name vllm-nemotron-jp \
  --runtime nvidia --network host --shm-size 16g \
  -e NVIDIA_VISIBLE_DEVICES=0 -e HF_TOKEN=$HF_TOKEN \
  nvcr.io/nvidia/vllm:26.01-py3 \
  python3 -m vllm.entrypoints.openai.api_server \
    --model nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
    --served-model-name nvidia/nvidia-nemotron-nano-9b-v2 \
    --trust-remote-code \
    --enable-auto-tool-choice --tool-call-parser hermes \
    --chat-template /tmp/nemotron_hermes_template.jinja \
    --gpu-memory-utilization 0.2 --max-model-len 16384 --max-num-seqs 4 \
    --port 30081

By setting --served-model-name to the same name as the default Nemotron, no changes to the VSS Agent configuration are needed.

When asked in Japanese "please tell me what you can do," the Agent responded in complete Japanese.

Hello. I am a routing agent that supports incident reporting in a warehouse video surveillance system. I am capable of the following:

• Listing sensors/cameras
• Checking real-time congestion status
• Capturing snapshots
• Listing incidents
• Generating detailed incident reports

In V1, the response would sometimes revert to English depending on the question content, but in the 3.0.0 + vLLM configuration, Japanese is maintained stably.

Japanese Incident Report Generation

We tried passing incident data accumulated in Elasticsearch to the Japanese model to generate a report. Since Nemotron 9B-v2-Japanese has a thinking mode, we use /no_think during report generation so the entire response is used as the report body.

The results were quite practical. When passing incident data for a restricted area violation, the following report is returned:

Incident Summary
On March 8, 2026 at 2:17:56 AM (UTC), an incident of unauthorized entry into a restricted area occurred in a restricted zone (Room-1) within the warehouse facility. Detected by sensor "warehouse," it was confirmed that object ID "329" had entered the ROI.

Recommended Response Measures

• Promptly share the relevant video with stakeholders and conduct an investigation to identify the intruder and their motives.
• Consider expanding the monitoring coverage of the restricted area and installing additional cameras to eliminate blind spots.
• It is recommended to strengthen automated access authorization checking and introduce a system that detects and warns in real time when unauthorized areas are approached.

It output UTC → JST conversion, as well as classifications for immediate response, medium-to-long-term measures, and legal response, all in Japanese. For a 9B parameter model, this quality seems sufficient for a simple on-site report.

However, in this verification, 42 microservices and vLLM are coexisting on a single DGX Spark, so resources are always running tight. It sometimes takes several minutes for the Agent's Chat UI to return a response, or timeouts between services occur mid-tool-call.

Since there's no issue with the LLM's standalone report generation capability, this is a constraint of the environment where everything is loaded onto one machine rather than a model limitation. If the LLM and VLM could be distributed to separate nodes, stable operation in a Japanese environment seems feasible.

Points to Note for EA

Here's a summary of EA-stage constraints noticed from actually running it.

Since DGX Spark operates on a single GPU (GB10), in local_shared mode the LLM and VLM share the same GPU. Memory management is tight, and as with 2.4.x, techniques for reducing LLM memory consumption via Ollama may be effective.

Cosmos-Embed1 (video + text integrated Embedding), a new feature in 3.0.0, was trained on English only, and it's important to note that search performance may degrade with Japanese text. The maximum token count for text is also a short 128. The search profile using this Cosmos-Embed1 for vector search is also in Alpha state, and in the current Warehouse Blueprint (bp_wh), CV-based event detection via Behavior Analytics is used by default.

The 2.4.x config.yaml and REST APIs are not compatible with 3.0.0, and no migration guide is currently available. It is not possible to migrate an environment built with 2.4.x as-is. As a DGX Spark-specific issue, as mentioned above, the LLM NIM ARM64 image not working was a deployment blocker. We were able to substitute with NGC vLLM + custom template, but manual work typical of EA — including manual placement of Perception models — should be expected.

There are also several known issues listed in the official release notes. The VA-MCP initialization delay is caused by HTTP 429 (Rate Limit) returned when making unauthenticated requests to HuggingFace, and can be resolved by setting HF_TOKEN in .env. There can also be slight discrepancies in snapshot timestamps due to implementation limitations in frame extraction, which is reportedly scheduled to be fixed in the next release.

Summary

We ran VSS 3.0.0 EA on DGX Spark.

The "inflexibility due to monolithic architecture" that we felt in 2.4.x is moving toward resolution through microservice architecture and MCP-based orchestration. The configuration of 42 services working together may look complex at first glance, but the design that lets you select only the functionality you need via compose profiles is a step in the right direction.

There were also several EA-specific pain points. The LLM NIM ARM64 image was broken, the Perception models were not included in the package, and there were 4 code bugs in Alert Bridge. However, by finding workarounds for each — LLM substitution with NGC vLLM + custom template, manual model placement from NGC, and Alert Bridge patches — we were able to get the video AI pipeline running end-to-end on a single DGX Spark.

Personally, the most satisfying part was the VLM-as-Verifier pipeline. The two-stage configuration of quickly narrowing down incident candidates with CV and then verifying by watching the video with the VLM feels highly practical as an approach to reducing false positives while enabling deep analysis. Having the entire flow including Agent report generation work end-to-end left a strong impression of a fairly complete implementation.

On the other hand, we also saw the limits of running everything on a single DGX Spark. In an environment where 42 services + LLM + VLM coexist, the GPU memory and CPU are always under heavy load, sometimes causing the Agent's chat response to take several minutes or service-to-service communication timeouts.

However, this is also the flip side of the microservice configuration in 3.0.0 working well. Whereas in the monolithic 2.4.x configuration distributed placement itself was difficult, in 3.0.0 you can eliminate bottlenecks simply by moving the LLM or VLM to a separate node. It feels like distributing the load across multiple machines would bring us much closer to production deployment.

Reference Links

• VSS 3.0.0 EA Documentation
• VSS 3.0.0 Release Notes
• Warehouse Blueprint (2D with Agents)
• Warehouse Blueprint Prerequisites
• VSS 2.4.1 Documentation
• VSS for DGX Spark
• Previous article: Running a Video Search AI Agent on DGX Spark