I tried video data search and summarization using VSS AI with Brev

Introduction

Hello! I'm tanaka-takeru from Classmethod's Manufacturing Business Technology Department.

• In factories, it is required to accurately grasp various events occurring on-site, such as equipment operating status, worker movements, and product conditions. One means of doing so is information extraction from video data. Of course, it's not limited to factories — there are many situations where you might want to extract information from videos, such as monitoring long-term experimental processes, tracking logistics, or summarizing web conferences.
• This time, we'll verify how information within videos can be obtained by using NVIDIA VSS (Video Search and Summarization) to give natural language questions and instructions to videos.
• NVIDIA VSS is not a single AI model, but a video analysis pipeline that combines technologies such as VLM (Vision Language Model), LLM (Large Language Model), and RAG (Retrieval-Augmented Generation). By integrating these technologies, it can analyze video content including not just visuals but also audio, and flexibly retrieve necessary information through natural language dialogue.

Execution Environment

Looking at the hardware requirements for VSS, the verified GPUs listed include H100, RTX PRO 6000 Blackwell, L40S, DGX SPARK, IGX Thor, and AGX Thor.
https://docs.nvidia.com/vss/latest/prerequisites.html#development-profile-gpu-requirements

Also, an official blueprint including a GUI tool is available, so this time we'll use it from Brev.
https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization

You can experience everything from logging into Brev to running VSS in about 10 minutes.
※Excluding instance startup and video upload time.
※GPU instance charges will apply.

Preparation

• Create a Brev account and log in to the console　https://developer.nvidia.com/login
• Create an NGC account and issue an API Key
• For those who want to use the official sample data -> Install NGC CLI

This time, the flow is:

1. Create a GPU instance: Create a GPU instance on Brev containing the full VSS resource set
   ↓
2. Deploy VSS: Use the prepared scripts to download containers from NGC
   ↓
3. Launch the application

Steps

1. Create a GPU Instance

• Log in to the Brev console
  https://developer.nvidia.com/login

• Open the Launchable tab, search for video-search-and-summarization-blueprint, and open it.
  

• Click Deploy Launchable to start creating the instance (charges will apply from this point onward, so please confirm the displayed cost. The cost during verification was $8.35/hr).
  

• Wait until the instance creation is complete.
  

2. Deploy VSS

• Once the instance is created, click Open Notebook to access JupyterLab.
  

• Deploy VSS using the prepared scripts.
  Open /deploy/docker/scripts/deploy_vss_launchable.ipynb.
  In the first Code cell, enter the API Key you prepared for NGC_CLI_API_KEY.
  

• Execute through to the Code cell for 10. Access the UI with default settings.
  

After waiting a moment, the application will start, so access the displayed URL.

VSS UI (via Brev secure link): https://7777-<ID>.brevlab.com

It launched successfully.

Trying It Out

• Let's upload some videos and try VSS.
  First, I asked questions about a video of boxes flowing along a conveyor belt.

• Asking what the video is about

Question: What is this video about?
Answer: The video is about a curved conveyor belt system in an industrial setting, specifically designed for transporting items such as cardboard boxes and packaged goods. The system features a durable rubber-like belt supported by blue metal frames with vertical supports. The conveyor operates continuously and efficiently, moving objects from one end to the other through a curved path without disruptions. The environment is a warehouse or factory with a concrete floor, and the items on the belt are labeled with barcodes, indicating a logistics or distribution process.
The video is titled "sample-sim-box-conveyor" and shows packages moving along a conveyor belt, demonstrating its functionality and efficiency in handling various items.

It gives a very detailed answer.

• Asking how many boxes passed (the correct answer is 8)

Question: How many boxes are moving along the conveyor belt?
Answer: The final answer is: There are 8 boxes moving along the conveyor belt.

It got the correct answer.

• Asking in Japanese
  
  Question: いくつの箱がベルトコンベアを流れますか？
  Answer: The number of boxes moving on the conveyor belt increases over time according to the analysis. The maximum observed count was 14 boxes at timestamp 80.1s. However, the exact final count at the end of the video (81.567s) is not explicitly stated in the results. If you need the precise final count, we may need to re-analyze the last few seconds of the video.

This time it got the wrong answer.
It seems better to ask in English.

• Trying with a random video on hand
  It's a dumpling-making video used in another blog post.

Question: What is this video about?
Answer: The video is about a person preparing dumplings on a wooden table. The process includes mixing ingredients in a large bowl, shaping dough sheets, adding filling with a spoon, folding and sealing the dumplings, and placing them on a metal tray. The video focuses on the hands and techniques used during dumpling preparation, with static background elements showing the workspace setup.

It responds including the general content of the work.

Cleanup

• Once verification is complete, return to the notebook and execute through to the Code cell for 14. Teardown.
  

• Also delete the GPU instance. Click Delete on the Console screen.

Wait a moment and confirm that the instance has been deleted.

Conclusion

• This time, we actually tried VSS following the official procedure.
  Since it can be operated through a GUI and you can easily try it with your own data, you can get a hands-on feel for the accuracy and speed.