Robot Visual Inspection Agent Utilizing Unitree Go2 and G1: Continuously Analyzing Live Camera Feed from Go2 with a Local VLM

Introduction

This week, I connected the Robot Visual Inspection Agent PoC built last week from manually captured images to the actual Unitree Go2 front camera.

At last week's stage, I was using images taken from a low angle with an iPhone, and the following pipeline was working.

Local image
  ↓
Analysis by Qwen2.5-VL
  ↓
Structured JSON output
  ↓
InspectionResult schema validation
  ↓
Markdown report generation

This week, I extended the configuration to periodically capture still images from the Go2 front camera and continuously analyze them with a resident Qwen process.

Go2 front camera
  ↓
Capture still images every few seconds
  ↓
Save as JPEG to watched folder
  ↓
Resident process with Qwen loaded only once
  ↓
Continuously analyze new images
  ↓
Generate JSON / Markdown reports

The current implementation does not feed video files or video streams directly into the VLM.

This is periodic capture-based continuous inspection, which captures still images from the Go2 camera at fixed intervals.

Therefore, in this article I will describe it as "continuous analysis of periodically captured camera images" rather than "real-time video analysis."

What I Accomplished This Week

This week, I mainly worked on the following tasks.

• Verified network connectivity between Go2 and the inference environment
• Investigated the ROS 2 environment and camera device on Go2
• Set up the Unitree SDK2 Python / CycloneDDS environment
• Captured actual images from the Go2 front camera
• Analyzed actual Go2 images with qwen_local
• Implemented a resident watch runner that loads Qwen only once
• Implemented a loop to capture still images from the Go2 camera at fixed intervals
• Implemented a mechanism to analyze only when a new image is saved
• Measured processing time per frame
• Implemented an optional speech queue
• Enhanced JSON repair and alias normalization
• Evaluated risk level judgment with actual images
• Analyzed false detections caused by copying prompt examples
• Analyzed false risk conversions caused by post-processing
• Investigated how to use the Go2's built-in speaker
• Organized the policy of separating VLM observation from Python-based risk judgment

Capturing Images from the Go2 Front Camera

The Go2 front camera could not be accessed from /dev/video0 like a typical USB camera.

Also, since ROS 2 commands were not functioning properly in the environment I confirmed this time, I adopted a method of capturing images via Unitree SDK2 Python / DDS.

Image capture uses the official camera capture sample included in Unitree SDK2 Python.

The execution looks like this:

python example/go2/front_camera/capture_image.py <NETWORK_INTERFACE>

The captured images were in the following format:

format:
  JPEG

resolution:
  1920 x 1080

size:
  approximately 224KB

I had expected around 1280×720 beforehand, but the images I actually captured were 1920×1080.

Analyzing Actual Go2 Images with Qwen

I fed the captured Go2 images into the local Qwen2.5-VL-3B-Instruct.

The application-side execution looks like this:

python app/cli.py \
  --image samples/real_tests/go2_front_001.jpg \
  --robot Go2 \
  --location "office test area" \
  --image-source dds \
  --camera-source go2_front_camera \
  --vlm qwen_local

During inference, images are resized to the following dimensions to reduce processing time:

1920 x 1080
  ↓
768 x 432

Part of the raw output was as follows:

{
  "scene": "office",
  "quality": "clear",
  "objects": [
    "black door",
    "glass wall",
    "stacked boxes",
    "monitor",
    "cable on floor"
  ],
  "hazard": "cable",
  "riskLevel": "medium",
  "action": "Ask an operator to confirm or secure the cable.",
  "confidenceScore": 0
}

Ultimately, I was able to generate normalized JSON and a Markdown report.

Go2 front camera
  ↓
Unitree SDK2 Python / DDS
  ↓
JPEG image
  ↓
Qwen2.5-VL
  ↓
JSON repair / normalization
  ↓
InspectionResult
  ↓
JSON / Markdown report

I was deeply moved that the pipeline I built myself ran all the way through from actual robot camera images.

From Single Image to Periodic Capture-Based Continuous Inspection

In the previous CLI, Qwen was loaded every time a single image was analyzed.

Process start
  ↓
Qwen load: approximately 40–50 seconds
  ↓
Analyze one image
  ↓
Exit

With this configuration, even if camera images are captured every 10 seconds, model loading is required each time.

So I added app/watch_run.py.

Process start
  ↓
Load Qwen only once
  ↓
Watch image folder
  ↓
Analyze new images
  ↓
Watch again
  ↓
Continue until Ctrl+C

Currently, the roles are divided across the following three processes:

Terminal A:
  Load Qwen only once
  Analyze newly saved images
  Save JSON / Markdown

Terminal B:
  Watch for speech request JSON
  Print or speak risk level

Terminal C:
  Save still images from Go2 front camera at fixed intervals

The main commands look like this:

# Analysis process
python app/watch_run.py \
  --image-dir samples/go2_periodic_eval \
  --robot Go2 \
  --location "office test area" \
  --image-source dds \
  --camera-source go2_front_camera \
  --vlm qwen_local \
  --poll-interval 1 \
  --summary-file watch_summary.json \
  --skip-existing

# Periodic camera image capture
python robot_io/go2_capture_loop.py \
  --iface <NETWORK_INTERFACE> \
  --out-dir samples/go2_periodic_eval \
  --interval 10 \
  --count 12

Currently, I am using three separate terminals, prioritizing ease of debugging.

In the future, I would like to use tmux or a supervisor script to allow startup with a single command.

Measuring Processing Time

I added processing to watch_run.py to record the processing time per image.

{
  "image": "samples/go2_periodic_eval/go2_front_xxx.jpg",
  "status": "success",
  "risk_level": "medium",
  "path_blocked": false,
  "processing_time_sec": 5.24
}

The actual measured results were as follows:

Qwen cold start:
  approximately 40–50 seconds

Image processing after model load:
  approximately 4–7 seconds

The heavier process is the initial model load rather than each image analysis.

By keeping Qwen resident, processing at 10-second intervals was possible without generating a significant backlog at this point.

For 5-second intervals, since processing time exceeds 5 seconds for some images, there is a possibility that a backlog may occur.

Optional Risk Level Notification

I also added a mechanism to notify messages like the following based on VLM analysis results:

Risk level, low.
Risk level, medium.
Risk level, high.

A speech queue is used to separate the analysis process from audio output.

watch_run.py
  ↓
speech request JSON
  ↓
speech_watch.py
  ↓
print or audio output

At this point, operation has been confirmed with the print backend.

[SPEAK] Risk level, medium.

I also investigated several methods for the Go2's built-in speaker, but I was unable to establish a way to play arbitrary audio programmatically in the current environment.

Next week, a new external speaker is expected to be provided.

If an external speaker becomes available, a simple configuration like the following would be possible:

speech request JSON
  ↓
speech_watch.py
  ↓
local TTS
  ↓
external speaker

For example, the following command could be executed from the speech watcher:

espeak-ng "Risk level, medium."

Since the external speaker is expected to simplify the implementation of the audio output part, I decided to proceed with the audio feature next week.

Stabilizing JSON Output

During continuous analysis of Go2 camera images, there were cases where Qwen's JSON output became unstable.

The format breakdowns I confirmed were as follows:

single quotes
Python True / False
comments
trailing comma
Markdown code fence
nested scene / image / path
object detection format including bbox_2d

So currently I am parsing in the following order:

strict JSON parse
  ↓
minor JSON repair
  ↓
Python boolean / None conversion
  ↓
ast.literal_eval
  ↓
alias normalization
  ↓
nested structure repair

I also simplified the generation settings.

generated_ids = model.generate(
    **inputs,
    max_new_tokens=384,
    do_sample=False
)

repetition_penalty and no_repeat_ngram_size were removed because they could break the repetitive structures needed in JSON.

In the latest evaluation, JSON parsing succeeded for all 12 images.

JSON parse success:
  12 / 12

Evaluating Risk Judgment

While the JSON format stabilized, issues remain with risk judgment.

Problem of Being Too Sensitive to Cables

With the initial rules, there was a tendency for medium to be triggered simply by a cable being visible in the image.

However, the following types of cables are not necessarily dangerous:

Cables far away
Cables along the wall
Cables beside furniture
Cables not entering the direction of travel

So I changed the prompt to consider distance and position relative to the path.

Problem of Everything Becoming High

When I added specific expressions to the prompt to mark nearby objects as high, Qwen would sometimes copy those expressions regardless of the image.

large box blocking lower-center path
person very close in center path

For example, there were cases where boxes or people not actually detected were output as obstacles.

This is a problem close to prompt example leakage, where specific examples in the prompt leak into the output.

So I removed the specific example sentences and changed to the following rules:

• Output only objects visible in the current image
• Do not treat objects not in detected_objects as obstacles
• Remove person-related hazards when human_presence=false
• Do not copy expressions from the prompt
• Only consider high candidates when an object and a proximity expression are in the same entry

False Judgments from Post-Processing

I also found a problem where raw output of low would become high after normalization.

In the previous post-processing, the following fields were concatenated for keyword search:

detected_objects
obstacles
potential_hazards
recommended_action
summary_ja

For example, the default recommended action is:

No immediate action is required.

The word immediate within this was being mistakenly used as a keyword representing proximity.

Currently, the information used for risk judgment is limited to the following:

Primary evidence:
  obstacles
  potential_hazards

Secondary evidence:
  detected_objects containing proximity expressions in the same string

Do not use:
  recommended_action
  summary_ja
  speech text

Recommended actions and summaries are information generated from risk judgment.

Using them again as the basis for risk judgment creates circular logic.

Latest Evaluation Results

In the latest evaluation, 12 Go2 camera images were processed.

JSON parse success:
  12 / 12

normalized risk:
  low: 10
  medium: 2
  high: 0

processing time:
  average: approximately 5.24 seconds
  minimum: approximately 4.47 seconds
  maximum: approximately 6.46 seconds

The "almost everything becomes high" problem that occurred previously has been resolved.

Also, when there was explicit path information as shown below, it could be normalized to medium:

boxes blocking the path
cables crossing the path
boxes in the way
clutter obstructing the path

On the other hand, even when a person or box was present nearby, high could not be determined when the VLM output contained only generic nouns.

{
  "detected_objects": ["boxes", "person"],
  "obstacles": ["boxes"],
  "potential_hazards": ["person"]
}

From this information alone, the following differences cannot be distinguished:

A person far away
A person right in front

Boxes along the wall
Boxes directly in front of the robot

This is the biggest challenge in the current risk judgment.

Next Design Policy

Going forward, rather than having the VLM directly determine the final risk level, the policy is to have it take on the role of structuring observational information from images.

I am considering a configuration where risk judgment is calculated deterministically in Python.

VLM:
  Observe the scene and objects
  Structure position, distance, and path status

Python Policy:
  Determine risk level from observation information

LLM:
  Explain final results for humans
  Assist with judgment in ambiguous cases

I would like to add the following fields to the observation schema:

{
  "forward_path_status": "partially_obstructed",
  "closest_forward_object": "boxes",
  "closest_forward_object_position": "center",
  "closest_forward_object_proximity": "near",
  "closest_forward_object_size": "large",
  "cable_crosses_forward_path": false
}

On the Python side, risk can be determined with rules like the following:

if forward_path_status == "blocked":
    risk_level = "high"
    path_blocked = True

elif (
    closest_forward_object_position == "center"
    and closest_forward_object_proximity == "very_near"
):
    risk_level = "high"
    path_blocked = True

elif forward_path_status in {
    "caution",
    "partially_obstructed"
}:
    risk_level = "medium"
    path_blocked = False

elif cable_crosses_forward_path:
    risk_level = "medium"
    path_blocked = False

else:
    risk_level = "low"
    path_blocked = False

Using an LLM for every final risk judgment is also an option, but the current VLM processing alone takes approximately 4–7 seconds per image.

Adding LLM processing on top of that could cause backlogs even with 10-second interval periodic capture.

Also, relying solely on an LLM for safety judgment means the output may vary even for the same input.

Therefore, I currently believe the following hybrid configuration is most appropriate:

Normal cases:
  Judge with Python Policy

Ambiguous cases:
  Supplementary confirmation by LLM

Human-readable explanation:
  Generated by LLM or Report Generator

I Also Want to Use Temporal Information

Currently, each image captured at fixed intervals is analyzed independently.

However, results may fluctuate for the same scene as follows:

image 1:
  low

image 2:
  medium

image 3:
  low

In the future, I would like to add temporal smoothing using the results of approximately the last 3 images.

low
medium
low

→ final risk: low

For high risk, a method of only confirming when detected consecutively is also conceivable, in order to err on the side of safety while suppressing false positives.

high
high
high

→ confirmed high risk

Even with periodic capture-based inspection, using recent history rather than judging from a single image alone may help stabilize results.

Next Steps

The following are the tasks I plan to tackle next:

1. Change the VLM output schema to focus on observational information such as object position, distance, apparent size, and path status
2. Deterministically determine risk level with Python Policy based on VLM observation results
3. Use the same evaluation images to compare before / after, and also consider smoothing using the results of the last few images
4. Consolidate the current 3-process configuration with tmux or supervisor to simplify the startup method
5. Connect the external speaker expected to arrive next week and implement risk level audio notification
6. After confirming the accuracy and stability of visual inspection, gradually consider safe movement coordination using VLA

Regarding movement coordination, I will not use a configuration that directly generates walking commands from VLM.

I will use a configuration that goes through a safety policy and pre-defined movement skills, as follows:

VLM observation
  ↓
Python Safety Policy
  ↓
Human approval or safety condition confirmation
  ↓
Pre-defined movement skills
  ↓
Robot movement

I plan to first verify in simulation or a limited safe environment, and proceed step by step with emergency stop and human override mechanisms in place.

Summary

I was able to connect the local image-based PoC created last week to the actual Unitree Go2 front camera.

Furthermore, I extended the configuration from single image processing to continuously analyzing images captured from the Go2 camera at fixed intervals.

The main things accomplished this week are as follows:

Capturing actual camera images from Go2
Analyzing real machine images with Qwen
Making Qwen resident
Continuous analysis of periodically captured images
JSON / Markdown report generation
Measuring processing time per image
Optional speech queue
Enhanced JSON repair
Real image evaluation of risk level policy

On the other hand, using the actual machine camera revealed challenges that were not apparent with only local images.

In particular, when the VLM is responsible for all of object detection, distance estimation, path judgment, and risk judgment, results change dramatically with prompt modifications.

The major learning from this verification is as follows:

"By directing the VLM toward the role of extracting observational information from images, and separating the final risk judgment as a deterministic Python Policy, stability, explainability, and evaluability can be improved."

Also, when handling images captured periodically, temporal consistency is important in addition to judging each image independently.

Next week, I will separate the observation schema and risk policy, and proceed with before / after comparison using the same evaluation images.

Furthermore, I also plan to work on risk level audio notification using the external speaker expected to arrive next week, simplifying the startup method for the 3-process configuration, and designing toward safe movement coordination.