Robot Visual Inspection Agent Utilizing Unitree Go2 and G1: Near-Real-Time Path Risk Dashboard for Go2 Achieved with Grounding DINO and Geometric Policy

Introduction

Last week, I built a PoC that periodically captures still images from the Unitree Go2's front camera and continuously analyzes them with a local VLM.

The architecture up to last week was as follows.

Go2 front camera
  ↓
Periodic still image capture
  ↓
Observation by Qwen2.5-VL
  ↓
Structured JSON
  ↓
Risk assessment by Python
  ↓
Markdown report

Being able to generate JSON and reports from real camera images was a major step forward.

On the other hand, as I advanced the evaluation of real images, I also found that there are limits to leaving all of "object recognition," "position understanding," "path determination," and "risk assessment" entirely to the VLM.

For example, there were cases where the VLM recognized a person but could not stably determine that the person was blocking the robot's straight-ahead path. There were also cases where the output fluctuated for nearby doors, boxes, and mixed obstacles.

So this week, I redesigned the live risk assessment path as follows.

Go2 front camera
  ↓
Periodic still image capture
  ↓
Object detection by Grounding DINO Tiny
  ↓
Bounding box
  ↓
Geometric calculation with path ROI
  ↓
Deterministic Python risk policy
  ↓
JSON / overlay PNG
  ↓
Streamlit dashboard on DGX Spark

The central theme of this article is the transition from a VLM-centric architecture to one centered on object position information from Grounding DINO and a deterministic geometry policy.

I also connected the pipeline from ingesting real Go2 camera images into DGX Spark, through GPU inference results, to continuous display on an operator-facing dashboard.

"Near-real-time" in this article means continuously analyzing still images captured every few seconds and displaying the latest results at short refresh intervals.

This is neither real-time video analysis that directly inputs a video stream, nor a safety-certified collision avoidance system.

Demo Video

https://youtube.com/shorts/VX45rbx6deU

What I Accomplished This Week

This week, I mainly implemented and verified the following.

• Building an object detection pipeline using Grounding DINO Tiny
• Deterministic geometric risk assessment using bounding boxes and path ROI
• Path obstruction determination using union occupancy across multiple objects
• Generation of overlay images combining bboxes and path ROI
• Continuous analysis of real Go2 front camera images on DGX Spark
• Mailbox design treating Go2 images as a single latest.jpg
• Operator-facing risk visualization dashboard using Streamlit
• Analysis process control via START / STOP
• Display of analysis logs, inference time, detected objects, and path obstruction status
• Grounding DINO CUDA inference on DGX Spark GB10 GPU
• Reduction of dashboard refresh interval and analysis watcher interval
• Implementation of local warning sounds based on risk state
• End-to-end operation verification from real camera input to dashboard display

The architecture this time also remains an advisory-only system intended solely to assist manual operation confirmation.

It is not connected in any way to Go2's movement, stopping, turning, or posture control.

Why I Decoupled the VLM from the Live Risk Assessment Path

In my verification up to last week, I was using Qwen2.5-VL-3B / 7B to obtain observation information like the following from real Go2 camera images.

{
  "scene": "office",
  "objects": [
    "person",
    "boxes",
    "door"
  ],
  "risk_level": "medium",
  "recommended_action": "operator confirmation"
}

Being able to stably output JSON and correctly judging the state of the physical world were separate problems.

The particularly challenging cases were as follows.

A person is detected, but the path is judged as clear
A closed door occupies the center of the screen, but the risk doesn't become high
Multiple boxes are ahead, but each is individually treated as medium
The difference between nearby and distant objects cannot be stably expressed
Observation results and risk expressions can fluctuate even for the same image

Therefore, rather than concluding that VLMs are entirely unnecessary, I decided to decouple them from the center of the fast, continuously running path risk assessment.

The current division of roles is as follows.

Grounding DINO:
  Detects objects and obtains their positions as bounding boxes

Python geometry:
  Calculates the overlap between bounding boxes and the path ROI

Deterministic policy:
  Determines risk level and path_blocked from occupancy and bbox positions

VLM / LLM:
  To be used in the future for scene description, report generation,
  understanding the meaning of signs and unknown objects,
  and supplementary explanation when requested by operators

Comparing VLM and Grounding DINO for this task of "how much of the forward path is occupied," the differences were as follows.

Based on this comparison, I adopted the following architecture for the current live path.

Fast path:
  Grounding DINO
  → ROI geometry
  → deterministic policy
  → dashboard / advisory

Slow optional path:
  VLM / LLM
  → Japanese explanation
  → report
  → supplementary understanding of ambiguous objects or scenes

In other words, the VLM was not discarded as a failure, but rather separated from the live assessment path where speed, stability, and position information are critical, and reassigned to a role more suited to semantic understanding and explanation.

Grounding DINO and Path ROI

The object detection model used this time is as follows.

IDEA-Research/grounding-dino-tiny

The initial prompt is set as follows.

person. box. chair. cart. door. robot. bag.

Detection results are handled as normalized bounding boxes.

{
  "label": "person",
  "score": 0.92,
  "bbox_normalized": [
    0.31,
    0.42,
    0.69,
    0.98
  ]
}

For the Go2 front camera image, I set a forward path ROI centered on the lower center of the image.

Conceptually, the regions are as follows.

Upper portion of image:
  Distant areas and background

Center of image:
  Space directly ahead

Lower center of image:
  Near-path region important when Go2 travels straight ahead

By intersecting object bounding boxes with this ROI, the following information is calculated.

Whether the object overlaps the ROI
The ratio of the ROI occupied by the object
Whether the object extends to the bottom of the image
The total degree to which multiple objects collectively occupy the path

With this architecture, the reason for risk assessment can be explained.

For example, when a person occupies a large portion of the lower-center path, the result is as follows.

{
  "risk_level": "high",
  "path_blocked": true,
  "primary_obstacle": "person",
  "aggregate_route_occupancy_ratio": 0.5948
}

Union Occupancy Across Multiple Objects

Looking at only a single object was sometimes insufficient for handling scenes with mixed obstacles.

For example, when multiple boxes or objects are on the path, each individual bounding box might be medium-level, but in combination, straight-ahead travel may be difficult.

Therefore, I added occupancy ratio calculation within the ROI across multiple bounding boxes.

The key is not to simply add up bounding box areas.

If duplicate detections occur for the same object, simple addition would overestimate the occupancy ratio.

For this reason, union coverage is calculated for the bounding box regions within the ROI.

Multiple bboxes
  ↓
Unionize overlapping regions within the ROI
  ↓
Calculate occupancy ratio relative to ROI area
  ↓
aggregate_route_occupancy_ratio

The current high-risk aggregation rule is roughly as follows.

2 or more route-relevant objects
AND
aggregate_route_occupancy_ratio >= 0.35
  ↓
high / path_blocked=true

This processing allowed me to correct for the multiple-box case.

Provisional high-risk recall with Grounding DINO alone:
  6 / 10
  60%

After adding multi-object aggregation:
  7 / 10
  70%

However, these results are interim results based on a small, high-risk-heavy, provisionally labeled dataset of 12 images.

They do not represent sufficient evaluation of low-risk scenes or proof of safety.

Connecting the Real Go2 Camera to DGX Spark

This week, I connected Go2 front camera images to the inference environment on DGX Spark.

The overall architecture is as follows.

Go2 front camera
  ↓
Unitree SDK2 Python / CycloneDDS
  ↓
DGX Spark host
  ↓
latest.jpg mailbox
  ↓
GPU container
  ↓
Grounding DINO + geometry policy
  ↓
Streamlit dashboard

For capturing Go2 camera images, I used the official front-camera sample included in Unitree SDK2 Python.

python example/go2/front_camera/capture_image.py enP7s7

The JPEG captured from the Go2 was first verified on the DGX Spark host side.

After that, I prepared a capture loop for periodic capture and configured it to save only the latest image to the dashboard input folder.

latest.jpg Mailbox Design

Initially, I considered saving a timestamp-appended file with each periodic capture.

go2_front_20260723_100001.jpg
go2_front_20260723_100004.jpg
go2_front_20260723_100007.jpg
...

However, this approach has problems.

For example, if only the analyzer is stopped by pressing STOP on the dashboard while the capture process continues running, images accumulate.

When START is pressed afterward, past images are processed all at once, and the dashboard may display old scenes rather than the current one.

Therefore, I changed the input to a single-image mailbox.

samples/local_dashboard_input/
  └── latest.jpg

In the capture loop, the file is updated using the following steps.

Official camera image
  ↓
Copy to .latest.jpg.tmp
  ↓
Atomic replace
  ↓
latest.jpg

Conceptually, this is as follows.

shutil.copyfile(
    official_img_path,
    temp_path,
)

temp_path.replace(final_path)

This reduces the chance of the analyzer reading a partially written JPEG, while always retaining only the latest single image.

The advantages of this design are as follows.

Raw image backlog does not accumulate while STOP is active
After START, only the latest scene is processed
The amount of raw images saved does not increase
Privacy and storage burden in actual operation is easier to control

The policy is to use only latest.jpg for daily input images, and to save only a limited number of overlay PNGs and JSONs in history.

GPU Inference on DGX Spark

The NVIDIA GB10 GPU of DGX Spark is used for inference.

Running Grounding DINO Tiny on CUDA, I was able to generate the following for actual Go2 front camera images.

latest_result.json
latest_overlay.png
history/*.json
history/*_overlay.png

With continuous real-machine input, the model forward inference operated at roughly sub-second.

However, the time until new results appear on screen is not determined solely by model inference.

Go2 camera capture
  ↓
JPEG write
  ↓
Watcher detection
  ↓
Image decode / preprocessing
  ↓
Grounding DINO inference
  ↓
Geometry / JSON / overlay generation
  ↓
Streamlit refresh
  ↓
Browser render

For this reason, it is necessary to separately consider not just inference_sec, but also the end-to-end latency from capture through analysis to dashboard display.

In the adjustments made this time, the following were shortened.

Analyzer poll interval:
  0.5 sec
  ↓
  0.2 sec

Dashboard status refresh:
  1.0 sec
  ↓
  0.5 sec

Dashboard live panel refresh:
  1.0 sec
  ↓
  0.5 sec

This change improved the perceived speed of screen updates.

Streamlit Dashboard

Among the outcomes of this work, the Streamlit dashboard is particularly important.

This is not simply a screen displaying the current risk, but is positioned as a common visualization platform for future robot recognition, verification, and operation projects.

Main Screen

The following is the main screen during analysis of actual Go2 front camera images.

GUI Screenshot 1

On the left, the latest analyzed camera image is displayed.

The red rectangles are bounding boxes detected by Grounding DINO.

The green trapezoid is the forward path ROI when Go2 travels straight ahead.

On the right, the following information is consolidated so that an operator can review it quickly.

ADVISORY RISK
PATH BLOCKED
PRIMARY OBJECT
INFERENCE
ROUTE OCCUPANCY
OBJECTS USED
CAPTURED
ANALYZED

In the screenshot, chairs and other objects are detected and partially overlapping the path ROI, so the following state is displayed.

ADVISORY RISK:
  MEDIUM

PATH BLOCKED:
  NO

PRIMARY OBJECT:
  CHAIR

INFERENCE:
  0.237 sec

ROUTE OCCUPANCY:
  0.149

In this way, it is possible to separately confirm the presence of an object and whether the straight-ahead path is completely blocked.

Even if an object is present in the image, PATH BLOCKED=YES will not necessarily be triggered unless the ROI occupancy ratio or position exceeds the threshold.

The START, STOP, ENABLE AUDIO, and STATUS buttons at the top allow the operator to control and confirm the state of the analysis process.

Detail Screen

Opening DETAILS / SETTINGS allows you to check the detection results and analysis process details.

GUI Screenshot 2

The DETECTED OBJECTS on the left displays the following for each detected object.

label
score
normalized bounding box

This allows you to check the inference results themselves, not just the red rectangles on the screen.

Also, GEOMETRY REASON displays the reason for the judgment made by the Python policy.

Example:

A route-relevant object partially overlaps the immediate forward route.

This indicates that the detected object partially overlaps the current forward path ROI, though it does not immediately imply complete blockage.

The DASHBOARD SETTINGS on the right allows you to check the following.

input directory
output directory
audio state
analyzer process state
recent analyzer log

Since Analyzer process: running is displayed, you can confirm that the analyzer launched from the dashboard is operating.

The analyzer log at the bottom outputs the processing history of the latest image, risk level, path_blocked, primary obstacle, ROI occupancy ratio, inference time, and more.

This detail screen is important not only for visualization, but also as a development and verification screen that can be used for threshold adjustment, model comparison, false positive / false negative review, and future G1 profile adjustment.

Near-Real-Time Visualization

The current dashboard continuously displays the latest images periodically captured from Go2.

Go2 camera
  ↓
latest.jpg update
  ↓
Analyzer detects new image
  ↓
JSON / overlay update
  ↓
Dashboard redisplays at 0.5-second intervals

As a result, operators can check the current detection results and path assessment on screen without continuously monitoring SSH logs.

This is not real-time analysis in the sense of directly processing a video stream.

However, by combining periodic capture, GPU inference, and short-interval refresh, a sufficiently practical update feel for confirmation support during manual operation was achieved.

Value as a Verification Tool for Future Projects

This dashboard is not limited to the current Go2 path risk assessment.

It can be used as a common UI when adding and verifying the following features in the future.

Near-field ROI profile for G1
Distance supplementary information from Depth / LiDAR
Mask refinement using SAM and similar methods
Asynchronous scene description by VLM
Comparison of object detection models
Comparison of thresholds and geometry policies
False positive / false negative review
Operator notes and event history
State visualization for G1 manipulation / beverage handover

For example, even when advancing to G1 beverage handover tasks in the future, the dashboard will serve as a foundation for displaying states such as the following.

TASK STATE
TARGET OBJECT
GRASP STATUS
HANDOVER READY
OPERATOR CONFIRMATION
ABORT STATUS

In this way, the dashboard created this time is not only a Go2 inspection PoC, but also a tool for developing, observing, and verifying future robot AI projects.

Warning Sounds Based on Risk Level

The dashboard also implements a feature that plays short local warning sounds based on the risk level.

Operators can select ENABLE AUDIO on the screen and switch to MUTE as needed.

The current notification policy is as follows.

low:
  Safe
  Notified only when risk state changes to low

medium:
  Caution
  Immediate notification on state change
  Approximately every 5 seconds when medium continues

high:
  Danger
  Immediate notification on state change
  Approximately every 2 seconds when high continues

unknown:
  Confirm
  Immediate notification on state change
  Approximately every 10 seconds when unknown continues

To prevent the same analysis result from being repeatedly announced with every dashboard refresh, analyzed_unix_ms is used to target only new analysis results for notification.

Also, when the risk state changes, the new state is immediately announced without waiting for the debounce time.

low → high:
  Danger

high → medium:
  Caution

medium → low:
  Safe

The current implementation uses espeak-ng in the Linux environment on DGX Spark.

espeak-ng -v ja -s 165 "危険"

However, since no available speaker connected to DGX Spark could be prepared in this demo environment, the actual audio notification is not used in the YouTube demo.

The audio feature implementation and the enable/mute toggle on the dashboard are complete.

In the future, if HDMI audio, a USB speaker, or an audio device on the Spark host can be connected, the current advisory audio can be used as-is.

Importantly, this audio is also purely a cautionary alert for operators, and does not perform stopping or movement control of Go2.

Issues Encountered During Implementation

During this implementation, there were many challenges in the actual runtime integration rather than in the model itself.

Issue Where Streamlit's Transparent UI Layer Covers Buttons

Initially, there was a problem where the START button was visible but could not be clicked.

The cause was not on the Python side, but was the transparent header/toolbar layer of Streamlit remaining over the button.

Even though it is visually transparent, it receives pointer events in the browser, so clicks do not reach the button.

The following countermeasures were taken.

Hide header / toolbar / Deploy / decoration
pointer-events: none
Specify z-index and pointer-events: auto for dashboard buttons

This problem is difficult to recognize from the dashboard's appearance alone, and it was necessary to check the UI layers and z-index in the browser.

Docker Container UID and PyTorch Cache Issue

When running the GPU container with the host user's UID, PyTorch internally could not obtain user information, causing the analyzer to exit immediately after startup.

KeyError:
  getpwuid(): uid not found

This was caused by the absence of a passwd entry for the host UID inside the container.

The solution was ultimately to mount the host's user/group information as read-only into the container runtime.

TorchInductor cache and Hugging Face cache directories were also specified to writable locations inside the container.

While not directly related to robot recognition algorithms, these were important operational challenges when connecting real hardware, GPU, container, and dashboard.

Current Results and Limitations

With Go2 v1 this time, I reached the following state.

Go2 real camera
  ↓
Unitree SDK2 / DDS read-only capture
  ↓
DGX Spark latest.jpg mailbox
  ↓
GB10 CUDA Grounding DINO
  ↓
Deterministic ROI geometry policy
  ↓
JSON / overlay
  ↓
Live Streamlit dashboard

The current visualization is very clear, and when persons, doors, boxes, chairs, etc. enter the path ROI, operators can make judgments while verifying the reasoning.

On the other hand, the following limitations remain.

Mixed lower-center clutter
Bag
Cable
Floor equipment
Objects not included in the prompt
Scenes with unstable detection bboxes

Also, the current evaluation data is small-scale and biased toward high-risk cases.

Therefore, what can be said at this point is the following.

Operated as a grounded visual advisory prototype for real Go2 input

However,
It is not a safety-certified collision avoidance system

Going forward, it will be necessary to create a controlled dataset with a better balance of low / medium / high cases and evaluate false positives and false negatives.

About G1

I did not advance to real G1 verification this time.

Rather than being incomplete, this is a deliberate decision not to directly apply the forward path ROI designed for Go2 to G1.

G1 may differ from Go2 in camera position, field of view, and how near-field areas appear.

Therefore, the following will be prepared in the next stage.

go2_forward_route_v1
g1_near_field_v0

The G1 ROI will initially be treated as a provisional profile for near-field inspection, and will be adjusted after controlled image collection.

For G1 as well, the current target remains read-only camera inspection at this stage, with no connection to walking, arm movement, grasping, or VLA execution.

Summary

This week, I evolved last week's VLM-centric PoC into a Grounded Multimodal Robot Inspection Agent utilizing real Go2 camera hardware and DGX Spark GPU.

The main achievements are as follows.

Read-only capture of real Go2 front camera images
Real hardware camera integration via Unitree SDK2 Python / CycloneDDS
Grounding DINO inference on DGX Spark GB10 CUDA
Geometric risk assessment using bounding boxes and path ROI
Multi-object union occupancy
Backlog avoidance with latest.jpg mailbox
JSON / overlay generation
Streamlit dashboard with START / STOP control
Continuous visualization of real Go2 images
Implementation of local warning sounds based on risk state
GUI foundation usable for future verification projects

The major learning from this work is as follows.

"For problems where position and occupancy ratio are important, such as path risk assessment, it is easier to achieve speed, reproducibility, and explainability by separating object position information from Grounding DINO and a deterministic geometry policy, rather than relying solely on the textual output of a VLM."

At this point, Go2 v1 has become a sufficiently demonstrable prototype connecting real camera hardware, GPU inference, and a dashboard.

In the next stage, the current Go2 v1 will be further stabilized.

Creation of a balanced controlled dataset
Measurement of capture-to-analysis latency
Analysis of hard cases such as mixed clutter
Recording of false positives / false negatives
Stabilization of thresholds and geometry policy
Verification of audio output device on Spark

After that, in order to adapt to G1, which has a different camera position and field of view from Go2, I plan to proceed with designing G1 ROI profiles and performing controlled calibration.

Once stability improvements for Go2 and adaptation to G1 have progressed, inspection agent v1 will serve as a foundation to build upon for the next robot AI project.

Note that this system is an advisory-only prototype intended to assist confirmation during manual operation.

The outputs of the detector, geometry policy, dashboard, and audio notifications are not connected to movement control of Go2 or G1.