I thought about the retail store version of NVIDIA FOX Blueprint with VSS Skills and Hermes Agent
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
Previously, I wrote about "Mini-FOX," a concept for scaling down NVIDIA Factory Operations Blueprint (commonly known as FOX) to start small with 1 line, 1 camera, and 1 use case.
FOX is a large-scale AI Blueprint for manufacturing, with assumed hardware at the DGX Station class. The concept was to scale it down to 1 line and start by combining DGX Spark, PC, Jetson, and AWS. After finishing that piece, I thought this idea could apply just as well outside of factories.
Supermarket store operations in particular share similar challenges with factories, in the sense that it's difficult to continuously review hours of footage every day. Shelf shortages, delayed restocking, crowding at checkout counters, carts left in aisles. Situations that are hard to see from POS and inventory data alone accumulate in video footage, and it's quite difficult in practice for store managers and supervisors to review all of it.
In this article, I'll explore a concept called "Mini-Retail FOX," which applies the Mini-FOX idea to retail stores. At the core of the reasoning and natural language interface, I'll place the combination of NVIDIA's official VSS Skills and Hermes Agent.
How FOX Looks When Applied to Retail Stores
Here's the main topic.
FOX was a reference design that integrates factory machine signals, quality systems, work procedures, and operational alerts, with a Factory Manager AI orchestrating specialized agents and machinery. The structure had a central "brain on the floor" with domain-specific agents for safety, quality, maintenance, and operations hanging beneath it.
When translated to retail stores, the equivalent of the Factory Manager AI would be a "Store Operations Support AI." The image is a Store Operations Support AI surrounded by agents that monitor shelf shortages and restocking, agents that watch checkout congestion, agents that observe aisles and customer flow, and agents that write daily reports — all assisting store managers and supervisors in their decision-making.
The granularity of events handled differs considerably between factories and retail. In factories, machine signals and procedure deviations are central, while in retail, the focus is on "changes in the state of people and shelves" read from video and sales data. On the other hand, the idea of "the central brain surveys long-form data comprehensively and narrows it down to candidates for human review" applies naturally to both.
Can FOX for factories be directly placed in retail stores? Not exactly. The official FOX configuration is quite heavy, and attempting to target all stores and all cameras at once has a high chance of stalling before PoC even gets moving. So, just like with Mini-FOX, we start by cutting it down small.
Start Small as Mini-Retail FOX Rather Than Targeting the Entire Store
Using the same framing as calling Mini-FOX a "trial version of the fully equipped FOX," Mini-Retail FOX is organized as a "trial version of the Store Operations Support AI." The form is to start with 1 store, 1 to a few cameras, and 1 use case.
Here's a rough mapping of how FOX scaled down for retail stores looks:
| FOX Configuration | Mini-Retail FOX Starting Approach |
|---|---|
| Factory Manager Agent for the entire factory | Lightweight Store Supervisor Agent for 1 store |
| Many specialized agents | Narrow down to 3–5 agents: shelf, flow, checkout, report, etc. |
| Large-scale local inference on DGX Station | Distribute across DGX Spark, PC GPU, AWS Bedrock, EC2 GPU |
| Operational twin | Start with a store event timeline and simple dashboard |
| Automated retraining and production deployment | Keep to a retraining candidate queue with store manager/SV human review |
Personally, I think this approach to cutting things down is the most realistic for retail as well. The first hurdle for retail AI is whether it can capture just one specific situation that's truly causing problems, rather than "connecting everything." Trying to tackle shelf shortages, congestion, aisles, safety, and promotional effectiveness all in parallel from the start causes the difficulty of operations and data preparation to spike all at once.
The Minimum Configuration of Mini-Retail FOX Starts with Event Extraction
The minimum configuration of Mini-Retail FOX follows nearly the same flow as Mini-FOX. Rather than feeding video directly to a VLM, we thin it out at the edge to create events, then pass those to LLM or VLM reasoning. Here's an overview diagram to give you a feel for it.
What I want to emphasize here is the point of not feeding all frames to the VLM. Streaming store footage from business hours directly into a VLM quickly becomes painful in terms of inference costs and bandwidth. The flow is to first thin out at the edge, use lightweight detection and rule evaluation to convert only frames that "might be candidates for review" into events, and then have the LLM or VLM return situation descriptions, review points, and recommended actions for those events.
As a concrete image of a store event JSON, here's an example where a shelf shortage candidate appeared at the beverage shelf in the evening. The idea is to organize the detector output and the results of simple rule evaluation into JSON and pass them to the reasoning side.
{
"timestamp": "2026-06-17T16:20:30+09:00",
"store_id": "store-01",
"camera_id": "store-01-beverage-aisle-01",
"location": "Beverage Shelf",
"event_type": "shelf_low_candidate",
"confidence": 0.78,
"frame_uri": "s3://example-bucket/events/2026/06/17/store-01/frame-001.jpg",
"detected_state": ["partial_empty_section"],
"rule_triggered": "low_shelf_density_over_60s",
"llm_summary": "A state where products appear sparse on the right side of the beverage shelf has continued for approximately 60 seconds.",
"recommended_action": "Please share with the store manager as a restocking timing candidate before the evening peak.",
"human_feedback": null
}
Keeping data at this level of granularity allows human review results to be written into human_feedback later, making it usable as a dataset for retraining. Even without deciding on a precise schema from the start, having timestamp, store_id, camera_id, event_type, frame_uri, llm_summary, and human_feedback should be sufficient.
Another important point is not to stream raw events directly to the store manager or SV review UI. Passing only through the detector and rule evaluation will surface scenes that aren't operationally problematic — such as carts during restocking operations, items awaiting alignment right after stocking, or momentary queues — as "candidates." Inserting a "curation layer" before feeding into the review UI, which bundles raw events, removes duplicates, and uses rules or a lightweight VLM to filter out operationally unproblematic types, significantly changes the experience on the review side. There's quite a difference between 200 alerts per day and 5 alerts per day in how store managers and supervisors use the system.
Laying Out NVIDIA Official VSS Skills as a Toolkit
When building the reasoning side of Mini-Retail FOX, there's no need to build VSS integration from scratch. Under the NVIDIA official NVIDIA-AI-Blueprints/video-search-and-summarization repository, 10 VSS Skills compliant with the agentskills.io specification are publicly available. Simply distributing these Skills to the agent side enables calling VSS APIs through natural language.
Pulling back one level to organize the structure of VSS Skills: there's a two-layer structure where Skills are the "interface for the user side" and VSS Developer Profile is "how VSS is run." From the agent's perspective, Skills are the drawer handles and VSS Profiles are the contents of the drawers.
There are 10 official Skills in total, but for Mini-Retail FOX in a retail store setting, the ones to engage with first can be narrowed down to about 5.
| Skill | Primarily Combined Profile | Use Case in Retail Stores |
|---|---|---|
video-search |
search |
Search for relevant scenes using natural language queries such as shelf shortages, restocking, congestion, and abandoned carts |
video-understanding |
base |
Confirm "what is happening here" in Q&A format for candidate scenes |
alerts |
alerts (verification) / (VLM) |
Alert management for aisle blockages and safety check candidates, combining CV and VLM verification |
report |
Depends on use case | Generate daily/weekly store operations reports via the /generate endpoint |
rt-vlm |
alerts (VLM) |
Caption/alert for live footage, to be developed into future real-time monitoring and recording |
Skills and Profiles are simply the relationship of "drawer handles and contents," so the table shows common combinations as a guideline. In actual operation, some Skills like report are used without selecting a Profile, while others like alerts are combined with multiple Profiles.
The installation procedure isn't difficult either. According to the NVIDIA Developer Blog explanation, sending a single natural language prompt to the agent creates symbolic links of the entire Skill folder to ~/.claude/skills/<name>/ or ~/.codex/skills/<name>/. For general-purpose hosts without agent-specific paths, ~/.agents/skills/<name>/ following the agentskills.io specification is also provided. Since it's a symbolic link approach, running git pull on the repository side simultaneously updates Skills for all agents.
There's no need to re-implement VSS Skills from scratch in the initial stages of Mini-Retail FOX. Starting by "pulling out drawers that seem relevant to retail store use cases" within the scope of official Skills makes PoC startup dramatically easier.
Placing Hermes Agent as the Natural Language Interface
As the agent side that calls VSS Skills, let's place Hermes Agent here. Regarding Hermes Agent, I've written in the NemoHermes series about using it on the DGX Spark OpenShell sandbox.
Placing Hermes Agent as the natural language interface for Mini-Retail FOX enables a flow where Hermes receives inquiries from store managers and SVs, selects the appropriate VSS Skill, passes it to the VSS Profile, and returns reconstructed results. Running it within the OpenShell sandbox means the network permissions needed for VSS integration, credentials, and sandbox boundary operations can directly reuse the configuration established in the NemoHermes series.
Hermes Agent can separate personas and roles using Profile units. Here's an example of role assignments for Mini-Retail FOX:
| Profile | Role |
|---|---|
| Store Ops | Receives natural language inquiries from store managers/SVs, selects and calls VSS Skills |
| Night Report | Generates daily reports in batch after business hours and distributes to Slack or daily reports |
| Triage | Routes and handles customer interaction and campaign-related inquiries away from store operations inquiries |
For the VSS Skills installation destination, if Hermes Agent is placed inside the OpenShell sandbox, the natural approach is to deploy within the sandbox via NemoClaw or OpenClaw to ~/.openclaw/workspace/skills/ or ~/.agents/skills/. Since Hermes Agent can reference the agentskills.io specification's general-purpose path, Skills can be shared with Codex or Claude Code without increasing agent-specific storage locations.
As an image of the experience, here's just one scene of a store manager talking to Hermes Agent. In actual operation, it would likely be set up so that inquiries can be sent from familiar touchpoints on the store side, such as Slack, in-store chat tools, or a simple web UI.
[Store Manager]
From yesterday's store footage from 15:00 to 19:00, please narrow down to 5 scenes that seem operationally concerning.
In particular, please look for shelves where products appear sparse, whether restocking operations occurred, congestion at checkout counters, and carts blocking aisles.
[Hermes Agent]
I have extracted 5 review candidates.
1. 16:20 Beverage Shelf
Part of the products appear sparse. This is a restocking timing review candidate.
2. 16:37 Beverage Shelf
A staff member appears to be approaching with a cart and conducting restocking operations.
3. 17:45 Checkout Area
Multiple people appear to have formed a waiting line. This is a candidate for reviewing the criteria for calling in additional register staff.
4. 18:10 Aisle
A cart or work items appear to remain near the center of the aisle. This is a candidate for reviewing aisle clearance rules.
5. 18:30 Sales Floor
A customer appears to be standing in front of the shelf. This is a candidate for reviewing product discoverability.
Reference frames are attached to each candidate. If you'd like to examine a specific candidate, please instruct with something like "Show me the basis for number 3."
The key point is that Hermes Agent remains in the position of "narrowing down to candidates for human review and presenting them with supporting evidence," rather than "automatically judging and returning instructions." In the initial stages of launching Mini-Retail FOX, I think it's easier to gain buy-in from the field by not overstepping this boundary.
At this point, all the components of Mini-Retail FOX — agents, VSS Skills, and VSS Profiles — have been laid out. Next, let's look at three configuration options for what hardware and services to actually run these on.
Configuration A: Local Setup Centered on DGX Spark and Hermes
The first configuration option is placing one DGX Spark in the store or in a nearby server room, and running VSS's search profile and Hermes Agent on it. This configuration suits situations where there's a requirement to keep footage within the store or to have everything contained within an internal network.
The advantage of this configuration is that it can capture the essence of the NVIDIA stack while keeping store footage local. Existing DGX Spark verification assets such as Cosmos-based VLMs, Nemotron-based LLMs, VSS Skills, and Hermes Agent can stack directly on top of each other, and it becomes easier to explain to the field that "footage remains within the store."
On the other hand, there are things to watch out for. Increasing the number of cameras or running long-form footage continuously on a single DGX Spark quickly hits the ceiling of GPU memory and inference costs. In the initial stages of Mini-Retail FOX, I think it's more realistic to narrow it down to 1 store, 1–3 cameras, and 1 use case. The scale-out discussion can wait until something visible emerges from the PoC.
Configuration B: Lightweight Detection on PC, Reasoning Delegated to the Cloud
If having a DGX Spark isn't feasible, or if you want to minimize hardware on the store side, a configuration that divides roles between edge and cloud is practical. The store side only handles video capture and lightweight detection with a small PC, and passes reasoning to LLMs or VLMs in the cloud.
The approach is to not continuously send normal footage, but rather send only representative frames or short clips around anomaly candidates to the cloud. This makes it easier to balance bandwidth, cost, and privacy, and also helps avoid the situation where cloud bills grow larger than expected during the PoC stage.
Here's an image of the role division between the edge and cloud sides:
| Role | Store PC | Cloud |
|---|---|---|
| Video capture | RTSP capture, frame thinning | Basically none |
| Lightweight judgment | Shelf shortage rules, checkout congestion counting, dwell detection | Basically none |
| Reasoning | Convert rule output to event JSON | VSS Skills + Bedrock / OpenAI compatible API |
| Storage | Short-term cache | S3, DynamoDB, OpenSearch |
| Notification | Local warning lights, in-store handheld notifications | Slack, Teams, daily reports |
Keeping lightweight judgment on the edge side means first-level alerts can still be issued even when the network goes down. Making only reasoning cloud-dependent allows degraded operation during outages to be organized relatively straightforwardly. Hermes Agent can be placed on either the store side or cloud side, but if you want to keep the store PC resources light, placing it on the cloud side and combining with Bedrock or OpenAI-compatible APIs is more manageable.
Configuration C: Using AWS IoT Greengrass with Multiple Stores in Mind
The final configuration option is for cases where multi-store deployment is considered from the beginning. This uses AWS IoT Greengrass as the edge runtime, receives events via IoT Core, and combines with Bedrock or SageMaker. Greengrass is a management platform that can run Lambda and containers on edge devices, and when combined with IoT Core, it makes it easier to safely operate edge devices across multiple locations. AWS official blogs also introduce a configuration using IoT Greengrass and IoT Core for industrial safety-focused video analysis from existing CCTV and edge gateways, connected to S3 and SageMaker.
In terms of roles, Greengrass handles edge application distribution and management, IoT Core receives events, Lambda and Step Functions advance the workflow, and Bedrock and Hermes Agent + VSS Skills generate descriptions and response options. Centralizing model improvement in SageMaker makes it easier to build in retraining and incorporation of store manager review results afterward.
If there's a premise of multi-store deployment in mind, it's easier to include Greengrass from the start. Even for a single-store PoC, if you're keeping subsequent horizontal expansion in view, placing a control point here means you won't be scrambling later. Conversely, if the initial PoC is completely contained within one store, it's also natural to start with Configuration A or B and shift to Configuration C when horizontal expansion comes into view.
Narrow Down to One Use Case First
For the first topic to run Mini-Retail FOX, I recommend either "detecting shelf shortage candidates" or "detecting aisle blockages."
Detecting shelf shortage candidates is a topic where the explanation can be completed with just the footage, it's easy to convert to events, and it's easy for store managers and SVs to understand as an extension of their daily work. The detection side can also run on simple state evaluation, and the landing point of "restocking timing review candidate before the evening peak" is familiar to the field. False positives can also be discussed with clear examples such as "a shelf that looks sparse but is actually fine right after restocking."
Generative AI material capturing a state where the right side of a dairy shelf is partially sparse
Shelf shortage candidates become more aligned with the store operations context when you can confirm not just a single snapshot but also the sequence through when restocking operations begin and the shelf state recovers.
Generative AI material of a scene where a staff member is approaching with a cart and conducting restocking operations
Detecting aisle blockages is a topic that's easy to connect to 5S activities from a safety perspective. It can be organized straightforwardly with rules such as a cart or cardboard box remaining near the center of an aisle, or dwell time exceeding a certain threshold. The fact that it can be completed with footage alone is also similar to shelf shortage candidates.
Generative AI material of a scene where a cart is left abandoned in the middle of an aisle
The scene images included here are pseudo-CCTV style materials created with generative AI, not actual store footage. When you want to explain the Mini-Retail FOX concept but can't immediately prepare actual store footage, using such pseudo materials to advance discussions about UX and event definitions ahead of time can reduce rework after starting the PoC.
SOP deviations — such as "a register is closed within the scheduled time" or "a cart has remained in an unexpected location for an extended period" — can strongly express the Mini-Retail FOX character. However, since they require organizing store-specific operating rules, they feel somewhat heavy as an initial PoC. I think it's more readable to start with shelf shortage candidates or aisle blockages as the main examples, keeping things to the flow of "what was found, how it was explained, and what judgment the store manager or SV made" before automating anything, and then progressively adding SOP verification as a development.
No Rush to Specialize Agents
Although many specialized agents appear in the FOX concept, it's easier to operate without over-dividing from the very start of Mini-Retail FOX. Start with all processing within a single Store Supervisor Agent, and divide it up only after reviewing logs reveals clearly distinct roles.
Even when dividing, I think about 4–5 of the following is sufficient initially:
| Specialized Agent | Area of Responsibility |
|---|---|
shelf_agent |
State description for shelf shortage candidates, restocking timing, and stockout candidates |
flow_agent |
Description of aisle blockages, dwell detection, and flow obstruction candidates |
checkout_agent |
Organization of checkout congestion, additional register candidates, and self-checkout guidance candidates |
report_agent |
Daily/weekly store operations report generation, distribution to Slack and daily reports |
learning_queue |
Collection of false positives and misses, management of feedback loop from human review results |
Being mindful of operational rule differences between stores as a criterion for dividing agents also makes things easier later. Keeping rules common across the entire chain on the Supervisor Agent side, and making fine store-specific differences switchable via Profile or config, makes inventory management easier when adding target stores later.
Summary
Here's a comparison of the three configurations — DGX Spark-centered, PC and cloud-distributed, and AWS IoT Greengrass-centered — laid out by key considerations.
| Configuration | Suitable Scenarios | Initial Cost | Ease of Externalizing Footage | Key Highlights |
|---|---|---|---|---|
| DGX Spark Local Configuration | Stores where footage is hard to externalize, NVIDIA-context PoC | Medium | Manageable even for stores that can't externalize | Combination of local VSS and Hermes Agent |
| PC + Cloud Reasoning | Low-cost technical verification, minimizing store-side hardware | Low | Design to send only anomaly candidates | Lightweight configuration starting from 1 store, 1 camera |
| AWS IoT Greengrass Configuration | PoC with multi-store deployment in mind | Medium | Easy to govern on the AWS side | Role distribution and governance with Greengrass, Bedrock, and SageMaker |
Thinking about what to expand on from here: actually installing VSS's search profile and video-search Skill on DGX Spark and running shelf shortage detection through one use case end-to-end, trying multi-store operations with AWS IoT Greengrass and Bedrock, building store operations automation according to Hermes Agent Profiles, and digging into checkout congestion analysis with Cosmos Reason or Cosmos 3 all seem like viable directions. Alongside the manufacturing edition of Mini-FOX, I'd like to keep tracking the gradual spread of Physical AI to workplaces beyond factories.
Reference Links
- NVIDIA FOX Blueprint を小さく始める Mini-FOX 構成を考えてみた
- NVIDIA VSS + AI Agents + Skills の身近な現場での使いどころを考えてみた
- NemoHermes で Hermes Agent を DGX Spark の OpenShell に載せてみた
- Transform video into instantly searchable, actionable intelligence with AI Agents and Skills (NVIDIA Developer Blog)
- NVIDIA-AI-Blueprints/video-search-and-summarization (GitHub)
- agentskills.io specification
- Improving industrial safety with video analytics, AWS IoT Core, and AWS IoT Greengrass (AWS Blog)
