I tried video data search and summarization with VSS AI on Brev

Introduction

• In factories, it is essential to accurately grasp various events occurring on the floor, such as the operating status of equipment, the movements of workers, and the condition of products. One means of doing this is extracting information from video data. Of course, this is not limited to factories — there are many situations where you may want to extract information from video, such as monitoring long-duration experiment processes, tracking logistics, or summarizing web conferences.
• In this article, we will verify how information can be obtained from videos by using NVIDIA VSS (Video Search and Summarization) to ask questions and give instructions in natural language.
• NVIDIA VSS is not a single AI model, but a video analysis pipeline that combines technologies such as VLM (Vision Language Model), LLM (Large Language Model), and RAG (Retrieval-Augmented Generation). By integrating these technologies, it can analyze not only video but also audio content, and flexibly retrieve the necessary information through natural language dialogue.

Execution Environment

Looking at the hardware requirements for VSS, the verified GPUs listed are H100, RTX PRO 6000 Blackwell, L40S, DGX SPARK, IGX Thor, and AGX Thor.
https://docs.nvidia.com/vss/latest/prerequisites.html#development-profile-gpu-requirements

Also, an official blueprint including a GUI tool is available, so this time we will use it via Brev.
https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization

You can experience VSS from Brev login to running in about 10 minutes.
※Excluding instance startup and video upload time.
※GPU instance charges will apply.

Preparation

• Create a Brev account and log in to the console　https://developer.nvidia.com/login
• Create an NGC account and issue an API Key
• For those who want to use the official sample data -> Install NGC CLI

This time, the flow is as follows:

1. Create a GPU instance: Create a GPU instance on Brev that includes all VSS resources
   ↓
2. Deploy VSS: Use the prepared scripts to download containers from NGC
   ↓
3. Launch the application

Steps

1. Create a GPU Instance

• Log in to the Brev console
  https://developer.nvidia.com/login

• Open the Launchable tab, search for video-search-and-summarization-blueprint, and open it.
  

• Click Deploy Launchable to start creating the instance (charges will apply from this point onward, so please check the displayed cost. The cost during verification was $8.35/hr).
  

• Wait until the instance creation is complete.
  

2. Deploy VSS

• Once the instance is created, click Open Notebook to access JupyterLab.
  

• Deploy VSS using the prepared scripts.
  Open /deploy/docker/scripts/deploy_vss_launchable.ipynb.
  In the first Code cell, enter the API Key you prepared for NGC_CLI_API_KEY.
  

• Run through the Code cell for 10. Access the UI using the default settings.
  

After waiting a moment, the application will start, and you can access the displayed URL.

VSS UI (via Brev secure link): https://7777-<ID>.brevlab.com

It has launched successfully.

Trying It Out

• Let's upload some videos and try VSS.
  First, I asked questions about a video of boxes flowing along a conveyor belt.

• Asking what the video is about

Question: What is this video about?
Answer: The video is about a curved conveyor belt system in an industrial setting, specifically designed for transporting items such as cardboard boxes and packaged goods. The system features a durable rubber-like belt supported by blue metal frames with vertical supports. The conveyor operates continuously and efficiently, moving objects from one end to the other through a curved path without disruptions. The environment is a warehouse or factory with a concrete floor, and the items on the belt are labeled with barcodes, indicating a logistics or distribution process.
The video is titled "sample-sim-box-conveyor" and shows packages moving along a conveyor belt, demonstrating its functionality and efficiency in handling various items.

It gives a detailed explanation.

• Asking how many boxes passed (the correct answer is 8)

Question: How many boxes are moving along the conveyor belt?
Answer: The final answer is: There are 8 boxes moving along the conveyor belt.

It got the correct answer.

• Asking in Japanese
  
  Question: How many boxes flow along the conveyor belt?
  Answer: The number of boxes moving on the conveyor belt increases over time according to the analysis. The maximum observed count was 14 boxes at timestamp 80.1s. However, the exact final count at the end of the video (81.567s) is not explicitly stated in the results. If you need the precise final count, we may need to re-analyze the last few seconds of the video.

This time it got the wrong answer.
It seems better to ask questions in English.

• Trying with a random video on hand
  This is a dumpling-making video used in another blog post.

Question: What is this video about?
Answer: The video is about a person preparing dumplings on a wooden table. The process includes mixing ingredients in a large bowl, shaping dough sheets, adding filling with a spoon, folding and sealing the dumplings, and placing them on a metal tray. The video focuses on the hands and techniques used during dumpling preparation, with static background elements showing the workspace setup.

It responds including a general description of the work being done.

Cleanup

• Once verification is complete, return to the notebook and run through the Code cell for 14. Teardown.
  

• Also delete the GPU instance. Click Delete on the Console screen.

Wait a moment and confirm that the instance has been deleted.

Conclusion

• This time, we actually tried VSS by following the official procedure.
  Since it can be operated via GUI and allows you to easily try it with your own data, it gave us a hands-on sense of its accuracy and speed.