I tried to organize the overall picture of NVIDIA VSS

Introduction

Hello! I'm tanaka-takeru from Classmethod's Manufacturing Business Technology Department.

Recently, I tried launching NVIDIA's VSS (Video Search and Summarization) Blueprint (a reference implementation published by NVIDIA) on Brev. This time, I'd like to organize the overall picture of VSS and what it can do.
First, I'll cover the system architecture from the official documentation, and then walk through the representative workflows that VSS provides.

What you'll learn from this article:

• VSS system architecture (what microservices it consists of and how they interact)
• What each of the 5 representative workflows does (Q&A, long-form summarization, alert verification, real-time alerts, and search)

Let me introduce the main players first.

• VSS (Video Search and Summarization): A blueprint for performing video search, summarization, Q&A, and alerts in natural language
• LLM (Large Language Model): A model that reads and writes text. The recommended VSS configuration uses Nemotron series and others, but this can be changed depending on the profile and settings.
• VLM (Vision Language Model): A model that watches video/images and puts them into words. You can choose from models supported by VSS, such as the Cosmos Reason series.
• NIM (NVIDIA Inference Microservices): NVIDIA's mechanism for providing models as APIs. VSS can use not only locally deployed NIMs but also remote inference endpoints depending on the configuration.

This is a bit long, but I hope you'll join me for at least the parts that interest you.

System Architecture

Let's start with the official system architecture diagram. The VSS documentation is as follows:

• NVIDIA Video Search and Summarization — Documentation

The official high-level architecture diagram can be found on the following page:

• VSS High Level Architecture (Official)

What the diagram shows is that VSS is not a single model, but a collection of multiple microservices with different roles. The official documentation divides these services into three broad areas:

VSS is organized into three areas of processing and analysis: real-time video intelligence (feature extraction, embeddings, and stream understanding with results published to a message broker), downstream analytics (enrichment of metadata into trajectories, incidents, and verified alerts), and agentic and offline processing (orchestrated tools for search, Q&A, summarization, and clip retrieval, including via the Model Context Protocol).

Source: video-search-and-summarization — README

The image is that it is divided into the following three areas:

• ① real-time video intelligence: Extracts features from video. Handles object detection (RT-CV), embedding generation (RT-Embedding), and VLM-based interpretation (RT-VLM).
• ② downstream analytics: Analyzes extracted metadata (Behavior Analytics) and processes it into meaningful events.
• ③ agentic and offline processing: Bundles tools such as reports, Q&A, and search to handle natural language requests.

Let me summarize the main components appearing in the diagram according to these 3 areas:

The models and middleware in the table are representative examples, and the actual configuration varies depending on the VSS version, profile, and settings. Also, not all of these are always running in the Blueprint. For example, the base profile handles Q&A and report generation, while long-form summarization is handled by the lvs profile. Each profile is designed not to start detection or embedding services that are unnecessary for that workflow.

Organizing Each Workflow with Sequence Diagrams

Now that we have an overview of the system architecture, let's dig deeper into what it can do. VSS provides multiple Agent Workflows that combine these components.

We provide multiple reference Agent Workflows which demonstrate how the individual components can be leveraged by an agent:

Source: video-search-and-summarization — README

Depending on whether you want to summarize, generate alerts, or search, you combine components to realize a workflow.

1. Q&A and report generation (short clips)
2. Long-form video summarization
3. Alert verification
4. Real-time alerts
5. Video search (alpha)

Let's start with the simplest one: Q&A and report generation.

Workflow 1: Q&A and Report Generation

The flow I tried in a previous article corresponds to this. The image is uploading one short video and asking "What is happening in this video?" The official quickstart describes it as follows:

Video retrieval, VLM-based Q&A, and report generation on short video clips

Source: Q&A and Report Generation — VSS Documentation

Let me put together a sequence diagram showing how user input flows through the system.

Let me briefly summarize the participants appearing in the diagram (common to subsequent sequence diagrams as well).

• VSS Agent UI: The user's window (Web UI)
• VSS Agent: The command center that decides which components to call and in what order upon receiving a request
• VLM (e.g., Cosmos Reason): Watches video/images and puts them into words
• LLM (e.g., Nemotron): Reads and writes text
• VIOS / VST: Video data management

The VLM watches the video, and the LLM reads and writes based on that — this division of roles is the basic form of VSS responses.

In Q&A, the video_understanding tool performs VLM analysis according to the question. In report generation, dedicated prompts and templates are used to format the VLM analysis results into Markdown or PDF. The number of VLM calls varies depending on the profile and settings.

Note on video length and tokens

NVIDIA explains that for standard VLMs, the video processed at one time is typically a short clip of less than 1 minute.

Standard VLMs are limited to processing short video clips, usually less than 1 minute, depending on the number of subsampled frames and level of detail required.

"1 minute" is not a fixed upper limit but a guideline to keep in mind. We'll look at this in more detail later.

Workflow 2: Long-Form Video Summarization

This is a workflow for summarizing large video data and outputting the content as text. The key feature is processing by dividing the video data into multiple chunks.

The Video Summarization Workflow enables analysis and summarization of video content without being constrained by the standard VLM context window limitations, allowing for the analysis of long-form video content.

Source: Long Video Summarization — VSS Documentation

The flow looks like this in a diagram:

First, the long video is cut into chunk_duration-second segments that the VLM can process. Each chunk is captioned in detail by the configured VLM, and the CA-RAG of the Video Summarization Microservice aggregates them as timestamped events. Then, the configured Summarization LLM generates an overall summary. Both the VLM and LLM can be swapped out through configuration.

chunk_duration is an important parameter. Shorter chunks mean more VLM calls but finer granularity. Longer chunks mean fewer calls, but more frame skipping within each chunk, making it easier to miss details.

Supplement: On the relationship between resolution, fps, video duration, and tokens
The number of tokens is determined by the number of sampling frames passed to the VLM and the level of detail required for each frame.

Here is an estimate for Cosmos3 Nano Reasoner, which is standardly adopted in some profiles of the current VSS (this formula cannot be directly applied to VSS in general or other VLMs).
Cosmos3 Nano Reasoner does not simply tokenize each video frame as a separate still image; instead, it combines 32×32 pixels spatially and 2 frames temporally — that is, 32 × 32 × 2 = 2,048 pixel-frames — into a single visual token.

Therefore, the number of visual tokens can be estimated roughly as follows:

Total visual tokens ≈ number of sampling frames × width × height / 2,048

What's important here is not the total number of frames in the original video, but the number of frames actually passed to the VLM after preprocessing. Increasing this frame count and resolution increases the number of visual tokens and inference cost. Actual values also depend on resolution rounding and preprocessing. The Cosmos3 model-specific API reference shows that a 32×32×2 patch corresponds to 1 multimodal token and that it works best with 16K or fewer multimodal tokens. NVIDIA Cosmos3 Reasoner API

Based on the formula above, the number of tokens per frame when input as a video is roughly as follows. Since Cosmos3 processes 2 frames as a unit, this differs from the number of tokens when a single image is input.

Input Resolution	Tokens per video frame
720p (1280×720)	approx. 460
1080p (1920×1080)	approx. 1,020
2048×2048	approx. 2,048

※Assumes an even number of frames, calculated by rounding up fractions in 32-pixel units.
※2048×2048 is a calculation example here and does not indicate the upper limit of input resolution.

For example, when sampling 1080p-equivalent frames at 2fps for 1 minute and passing all 120 frames to the VLM without downscaling, the number of tokens using 32×32×2 patches would be approximately:

120 / 2 × 1920 / 32 × 1080 / 32
= 60 × 60 × 34
= 121,500 tokens

This is approximately 16 times larger input than the 7,840 tokens/inference configuration used in VSS performance measurements. The performance measurements process 448×448, 80 frames as 10-second chunks, using 7,840 visual tokens per inference. NVIDIA RTVI-VLM Performance

Also, to be precise, resolution, fps, and duration don't always trade off against each other. VSS has two frame selection methods:

• In FPS-specified mode, sampling frames ≈ fps × chunk duration, so resolution, fps, and duration all affect token count.
• In standard fixed-frame-count mode, a fixed number of frames are sampled at equal intervals from each chunk. In this case, even if chunk_duration is longer, token count barely changes if the frame count is the same. However, the interval between frames widens, making the temporal information coarser.

In long-form summarization, the analysis results per chunk are aggregated downstream.
NVIDIA recommends 30-60 second chunks for the Video Summarization Microservice from a GPU utilization perspective. NVIDIA Video Summarization Microservice — Best Practices

Workflow 3: Alert Verification

Next is the task of detecting arbitrary events from continuously flowing video data. Since continuous VLM inference is GPU-intensive, object detection models and VLMs are used in combination.

Realtime processing of videos using perception (object detection, tracking) and behavior analytics to generate alerts, which are subsequently verified with VLM to reduce false positives

Source: Alert Verification — VSS Documentation

Here is the sequence diagram:

Here, RT-CV continuously processes the stream at the configured inference FPS. Then, only alert candidates that are flagged as suspicious when checked against rules are input to the VLM to confirm whether they are genuine. Object detection models cast a wide net for candidates (reducing missed detections), and the VLM confirms them (reducing false positives). This division of labor minimizes calls to the GPU-intensive VLM while achieving streaming processing.

As for how to define the events you want to capture, they can be defined in three parts:

• What to detect: Specify the key objects for the event in text (e.g., fallen bottle). Since Grounding DINO has an open vocabulary, you can add targets in natural language.
• What constitutes an alert: Write rules in the Behavior Analytics settings, such as zone intrusion, 5 or more present, or lingering in a zone.
• How to confirm if it's real: Write a confirmation prompt in the VLM verification settings (e.g., "Is a fallen bottle actually in the image? Answer yes/no").

Incidentally, while this workflow assumes live RTSP video, pre-recorded video can also be used. A tool called NVStreamer can input a video file as an RTSP stream and pass it through the same pipeline. During the verification stage, you can process pre-recorded video data in batch to check feasibility.

Workflow 4: Real-Time Alerts

Some anomalies are difficult to express with the rules described above. For example, cases where you want to articulate something vaguely like "unusual work" or "unsafe posture." In such cases, rather than generating candidates using only object detection and fixed rules, Real-Time Alerts — where the VLM directly evaluates sampled video segments — becomes an option.

Continuous processing of video streams through VLM for anomaly detection

Source: Real-Time Alerts — VSS Documentation

Here is the sequence diagram:

Compared to the previous alert verification (Workflow 3), the VLM is placed further upstream, and the VLM directly and continuously watches the video to determine anomalies. Accordingly, the GPU load is higher than configurations where only candidate segments are verified by the VLM. For this reason, this profile defaults to processing 1 stream simultaneously.
※However, this is a value based on the profile's standard settings and GPU requirements, and is not the intrinsic upper limit of VSS or RTVI-VLM. Whether multiple streams can be handled depends on hardware, model, number of input frames, resolution, and deployment settings. Since alert verification can limit VLM calls to candidate segments, when comparing both methods, you need to estimate both the benefit of flexibly defining events and the available computational resources.

The definition of anomalies is written by humans in natural language (e.g., "When a person without PPE enters," "When a box falls over"). Rather than leaving it to the VLM to spontaneously decide what is anomalous, humans specify what constitutes an anomaly in words, and the VLM's role is to determine whether that has occurred — this is the division of responsibility.

It seems good to choose how to write rules based on the VLM's strengths and weaknesses. Based on my experience trying it with short clips, my impression is that VLMs are relatively accurate at judging the presence or absence of something like "Is ◯◯ visible?" while they tend to give inconsistent answers when asked vaguely like "anything unusual in general." I think that even with flexibility, the more you specify rules with concrete and clear conditions that can be answered with yes/no, the more stable they become (this is still a consideration from a small amount of data).

Workflow 5: Video Search ※Alpha

The last one is a workflow for finding "that scene" from a large archive of recordings, rather than reading one video or monitoring. It is still in alpha.

Natural language search across video archives using video embeddings (alpha)

Source: Video Search — VSS Documentation

This search workflow has multiple paths. The following is a simplified diagram of the path that executes an Embed Search from Vision Agent Chat and confirms candidates with the Critic Agent. The paths for directly using the search API, as well as Attribute Search, Fusion Search, and Search by Image, are omitted.

While the previous Q&A was "read one video and answer," search is a workflow for "find and list from multiple video data." In the Embed Search shown in the diagram, each video is pre-vectorized using embeddings and accumulated, and search queries are also converted to vectors in the same space to retrieve nearby candidates.

The current Search Workflow has four search methods:

• Embed Search: Search by similarity between video embeddings and natural language queries
• Attribute Search: Search object attributes extracted by RT-CV
• Fusion Search: Search combining results from embeddings and attributes
• Search by Image: Search for scenes similar to an input image

When evaluating search quality, you need to check the distribution of the embedding space and its fit to the task. In particular, the order of events, count, and negation conditions cannot necessarily be distinguished by simple cosine similarity alone. It is important to evaluate using actual data with metrics such as Recall and Precision.

As a reference, in a small number of experiments using a Qwen-based model separately, similarity between different video clips concentrated in the range of 0.97-0.99, and anisotropy was considered as a hypothetical cause. Anisotropy is the phenomenon where vectors are biasedly distributed in a specific direction.
※Preprocessing, pooling, and normalization are also candidate causes, and it could not be concluded to be anisotropy; nor are these results indicative of the properties of Cosmos-Embed1 or VSS. Verifying the hypothesis requires analysis of similarity distributions, principal components, eigenvalues, etc. for many samples.

In the current Search profile, the Critic Agent is enabled by default, checking top candidates against search conditions with VLM and assigning confirmed or rejected. This is an officially provided mechanism for improving search accuracy. NVIDIA VSS — Search Workflow

Conclusion

Starting from the VSS system architecture, I organized the 5 workflows using sequence diagrams. When I first looked at VSS, I was confused by the large number of services and unique concepts, but summarizing them by use case and data flow helped me organize the overall picture.

Also, VSS allows you to configure VLM and LLM inference endpoints, and you can choose compatible local NIMs or remote services according to your needs. In other words, it is designed so that models can be swapped out.

Since LLM/VLM is the core component for achieving tasks, I feel that a phased approach works well:

Decompose what you want to do into minimal sufficient tasks
↓
Verify in small steps whether it can be solved with the core LLM/VLM
↓
Orchestrate with the VSS platform according to the operational phase

By considering things in stages this way, I felt it's possible to get closer to operations without rework.

While a single process optimization might only call for introducing a standalone VLM, I think the true value of VSS lies in treating video information as one of various manufacturing data sources and rolling it out and linking it across lines and factories.
I'd like to deepen my understanding by actually running each workflow.