I've started building an MVP of a robot visual inspection agent utilizing Unitree Go2 and G1.

Introduction

In this internship, I am working on developing a Robot Visual Inspection Agent utilizing Unitree Go2 and G1.

This is the result of one night of brainstorming, but I received positive feedback from Atsu-san.
Based on that feedback, I reorganized the direction of the theme slightly.

In this article, I will summarize the theme ideas I am currently considering and the first MVP skeleton I implemented this week.

Theme I Want to Work On

Robot Visual Inspection Agent

The goal is to create a PoC that uses camera images from actual robots to check the status of offices and facilities, and performs the following processing using VLM/LLM.

• Scene understanding
• Anomaly detection
• Risk assessment
• Structured JSON output
• Japanese / English report generation

The overall flow assumes a pipeline like the following.

Camera images from Unitree Go2/G1
        ↓
Scene understanding by VLM
        ↓
Structured JSON output
        ↓
Japanese/English report generation by LLM
        ↓
Output via Markdown/Slack/etc.

Assumed Use Cases

For example, I am assuming instructions like the following.

Please use Go2 to check the area around the entrance and report any abnormalities.

In this case, the image is of VLM / LLM outputting inspection results like the following.

{
  "robot": "Go2",
  "location": "office entrance",
  "status": "warning",
  "observations": [
    "A cardboard box is placed near the corridor",
    "There is a possibility of obstructing passage"
  ],
  "risk_level": "medium",
  "recommended_action": "Check whether it is obstructing the corridor and move the box if necessary",
  "summary_ja": "A cardboard box is placed in the corridor near the entrance, which may obstruct passage. It is recommended to check."
}

Rather than simply describing "what is shown in the image,"
the goal is to assess the situation and generate a report from the perspective of facility inspection and safety confirmation.

MVP Scope

Rather than aiming for autonomous navigation or real-time video analysis from the start, I will first build a minimal MVP.

In this MVP, the goal is to run the following pipeline.

Image input
  ↓
VLM analysis
  ↓
JSON conversion
  ↓
Report generation

The plan is to first build the pipeline with local images, then connect it to images captured by Go2.

Technical Architecture Plan

The technical architecture I am currently considering is as follows.

Expected Deliverables

Ultimately, I would like to leave the following deliverables.

• A working PoC for the image → VLM → JSON → report generation pipeline
• A demo using images captured by Go2
• Sample images and evaluation results
• Compilation of failure cases and areas for improvement
• Setup instructions and technical documentation

What I Accomplished This Week

This week, I started by organizing ideas for the robot visual inspection agent, and ultimately managed to progress to a PoC that works with real image evaluation using a local VLM.

Initially, I considered a configuration that uses camera images from Unitree Go2/G1 to check the status of offices and facilities, and performs anomaly detection, risk assessment, and report generation using VLM/LLM.

After that, rather than jumping straight into real robot integration or real-time processing, I decided to first build a minimal MVP like the following.

Image input
  ↓
VLM analysis
  ↓
Structured JSON output
  ↓
Schema validation
  ↓
Markdown report generation

The main items I implemented and verified are as follows.

• Created project repository structure
• Defined Pydantic schema for inspection results
• Implemented mock VLM client
• Implemented Markdown report generator
• Implemented CLI to generate JSON and Markdown reports from input images
• Created initial inspection prompt
• Organized README and setup instructions
• Built local VLM execution environment with Brev / Docker
• Verified SmolVLM2-2.2B-Instruct
• Verified Qwen2.5-VL-3B-Instruct
• Integrated Qwen2.5-VL into CLI as qwen_local backend
• Improved prompts to suit inspection use cases
• Partially implemented JSON repair / alias normalization
• Started real image evaluation using low-angle iPhone images
• Created evaluation/labels.json
• Organized failure cases and areas for improvement

At this point, the following pipeline is working end-to-end.

Image input
  ↓
Local Qwen VLM inference
  ↓
Structured JSON output
  ↓
InspectionResult schema validation
  ↓
Markdown report generation

I believe the major achievement of this week is not simply getting the model to run, but
building the skeleton of the application from image input to report generation, and also being able to confirm failure cases with real images.

MVP Skeleton Implementation

First, as an initial step, I implemented the MVP skeleton of the robot visual inspection agent.

The repository was set up with mainly the following structure.

robot_inspection_agent/
├── app/
│   └── cli.py
├── evaluation/
│   └── labels.json
├── outputs/
├── reports/
│   └── generate_report.py
├── samples/
│   └── real_tests/
├── vlm/
│   ├── clients/
│   │   ├── base.py
│   │   ├── mock_client.py
│   │   └── qwen_local_client.py
│   ├── prompts/
│   │   ├── inspection_prompt_v1.md
│   │   └── inspection_prompt_v2.md
│   └── schema.py
├── README.md
├── requirements.txt
└── requirements-vlm.txt

At this stage, I prioritized confirming the overall flow of the application rather than the accuracy of the VLM itself.

Therefore, I first implemented mock_client.py and confirmed the following flow with a mock VLM that returns fixed inspection results.

Pass image path to CLI
  ↓
mock backend returns inspection result
  ↓
Validate with InspectionResult schema
  ↓
Save as JSON
  ↓
Generate Markdown report

This allowed me to create a configuration where the CLI, schema, and report generation parts can be shared even when the VLM backend is swapped out.

Inspection Result Schema and Report Generation

In this PoC, rather than having the VLM output free-form text, I decided to have it output structured JSON.

The reason is to make it easier to perform the following in subsequent processing.

• Schema validation
• Risk level comparison
• Cross-referencing with evaluation data
• Markdown report generation
• Future Slack notifications and dashboard integration

The inspection result is defined as InspectionResult, containing information such as the following.

scene_type
image_quality
detected_objects
human_presence
door_state
obstacles
potential_hazards
path_blocked
risk_level
recommended_action
confidence
summary_ja

The Markdown report generator uses this structured JSON to generate a Japanese report that is easy for human operators to read.

At this point, the following division of roles within the application became clear.

VLM: Looks at the image and outputs structured information
Schema: Validates the output format
Normalizer: Corrects inconsistent output
Report Generator: Creates a report for humans
CLI: Connects everything together

Local VLM Backend Verification

After the MVP skeleton was working, I verified local VLMs in the Brev / Docker environment.

First, I confirmed that the Isaac Sim container could access an NVIDIA L40S GPU, and created a Python virtual environment for VLM experiments.

After that, I installed packages such as the following.

• PyTorch
• Transformers
• Hugging Face Hub
• Other dependency packages required for VLM execution

I first tried SmolVLM2-2.2B-Instruct as a lightweight model.

SmolVLM2 loaded successfully locally and was able to generate output from images.
JSON-like output was also obtained.

On the other hand, when viewed as an inspection task, there were challenges with its reasoning capabilities.

For example, while it could recognize objects such as doors and boxes,
the judgment to treat a box on the corridor as an "obstacle" or "hazard" was not very stable.

I then verified Qwen2.5-VL-3B-Instruct.

Qwen2.5-VL showed better results compared to SmolVLM2 in the following areas.

• Can output structured JSON relatively stably
• Can generate Japanese summaries
• Can recognize major objects in images to a certain extent
• Responds well to prompt improvements

Therefore, at this point, I have integrated Qwen2.5-VL-3B-Instruct into the main CLI as the qwen_local backend.

What I Learned from Prompt Improvement

With the initial prompt, even though Qwen could recognize objects in the image, it sometimes made judgments that differed from expectations for inspection purposes.

For example, even with an image of a box on the corridor, there were cases where it would simply describe "there is a box" without treating it as an obstacle.

Also, the recommended_action would sometimes contain robot control-oriented content such as the following.

The robot should move to a different route

However, what I want to build is not an agent that directly controls the robot, but
a visual inspection agent that reports the situation to a human operator.

Therefore, I adjusted the prompt as follows.

• Do not issue instructions to control the robot
• Output recommended actions aimed at humans / operators
• Treat objects on the corridor as obstacle candidates
• Explicitly judge whether the path is blocked
• Do not treat ordinary furniture as obstacles unless it is blocking the path
• Do not hallucinate floor trash or cables without basis
• Output only JSON
• Do not change the specified field names

After the improvement, the prompt was able to treat the box on the corridor as an obstacle and make the following judgments.

{
  "path_blocked": true,
  "risk_level": "high",
  "recommended_action": "Please check the box on the corridor and remove it if necessary."
}

Through this verification, I learned that when using a VLM as an inspection agent, not only model performance, but also
clearly defining the task, output format, and the perspective for recommended actions is important.

Start of Real Image Evaluation

In the latter half of the week, I also started evaluation using indoor images actually taken, not just synthetic or sample images.

Ideally, I would like to use Go2's camera images, but working alone, I avoided forcibly proceeding with DDS/ROS/real robot camera integration.
Therefore, as a safe alternative, I held an iPhone at a low position and took images close to the Go2's perspective.

These manually captured images are handled with the following metadata.

image_source = manual_capture
camera_source = phone_low_angle_go2_like

The images taken are as follows.

manual_entrance_clear_001.jpg
manual_lab_clear_001.jpg
manual_lab_obstacle_001.jpg
manual_seat_clear_001.jpg
manual_seat_obstacle_001.jpg

The images were not committed to Git, but transferred to the Brev environment and evaluated inside the Docker container.

An example of running the CLI using the local Qwen backend is as follows.

python app/cli.py \
  --image samples/real_tests/manual_entrance_clear_001.jpg \
  --robot Go2 \
  --location "office entrance" \
  --image-source manual_capture \
  --camera-source phone_low_angle_go2_like \
  --vlm qwen_local

At this stage, I was able to confirm the entire process from local Qwen inference to JSON generation, schema validation, and Markdown report generation with real images.

Results Confirmed with Real Images

In the real image evaluation, several good results and failure cases became apparent.

Entrance Image

For the entrance image, Qwen was able to understand the scene mostly correctly.

Part of the raw output is as follows.

{
  "scene": "entrance",
  "quality": "clear",
  "objects_detected": ["glass door"],
  "human": false,
  "door_status": "closed",
  "floor_condition": "wooden",
  "potential_risks": ["no specific hazards observed"],
  "obstructions": ["none"],
  "recommended_actions": [
    "Ensure the door is properly locked when leaving the building."
  ],
  "confidence_score": 0,
  "status_message": "The area appears safe and ready for entry."
}

It was able to recognize that it was an entrance, that there was a glass door, and that the corridor was not blocked.

On the other hand, field names different from the expected schema were used.

For example, scene was used instead of the expected scene_type,
quality instead of image_quality,
and potential_risks instead of potential_hazards.

Also, the absence of obstacles was sometimes expressed as a list like ["none"].

For this reason, rather than passing the VLM output directly to the schema, alias normalization and list cleanup are needed.

Lab Image

For images close to a lab or storage area, Qwen was able to detect many objects.

{
  "objectsDetected": [
    "floor",
    "cabinets",
    "cart",
    "box",
    "plastic bag",
    "air purifier",
    "monitor",
    "signboard"
  ],
  "obstacles": ["cabinet"],
  "potentialHazards": ["cable on floor"],
  "pathBlocked": true,
  "riskLevel": "medium"
}

Object detection itself was working to some extent.

On the other hand, for images that could not be clearly called dangerous, there were cases of hallucinating cable on floor or treating ordinary cabinets as obstacles.

From these results, I felt that Qwen tends to slightly overestimate risks for scenes where there are many objects but passage is possible.

Obstacle Image Around Seating Area

For an image showing a cable near the seating area, Qwen was able to recognize the cable as a hazard.

{
  "scene": "office",
  "quality": "clear",
  "objects_detected": ["yellow chair", "desk", "monitor"],
  "obstacle": ["cable on floor"],
  "hazard": ["cables on floor"]
}

However, even in this case, the output used field names different from expected, such as obstacle and hazard.

The expected field names are as follows.

obstacles
potential_hazards

To handle such output inconsistencies, it is necessary to add more aliases on the post-processing side.

Improvement Points Identified from Real Image Evaluation

Through the real image evaluation, the points that need to be improved going forward became quite clear.

Broadly speaking, there are three issues.

1. Schema Inconsistency

Qwen sometimes uses different field names for items with the same meaning.

Examples:

scene
sceneType
quality
imageUrlQuality
objectsDetected
objects_detected
human
humanPresence
door_status
doorState
obstacle
hazard
potentialHazards
potential_risks
pathBlocked
riskLevel
recommendedAction
recommended_actions
confidence_score
summaryJa
status_message

Even when instructed in the prompt to "use the specified keys," it was not fully complied with.

Therefore, in addition to prompt engineering, alias normalization in post-processing is necessary.

2. JSON Format Corruption

Even when instructed to output only JSON, the VLM can sometimes produce format corruption such as the following.

Markdown fences are included
JavaScript-style comments are included
trailing commas are included
single quotes are used
required fields are missing

For this reason, JSON repair processing also needs to be strengthened.

3. Instability in Risk Assessment

While image understanding itself works to a certain extent, there is still instability in judgments as an inspection task.

For example, the following tendencies were observed.

• Treating ordinary furniture as obstacles
• Hallucinating unclear cables or floor trash
• Judging a room that simply has many objects as excessively dangerous
• When the prompt is made stricter, it may conversely miss hazards
• Returning values like confidence: 90 on a 0–100 scale instead of 0–1

Regarding this, it will be necessary going forward to combine improvements to the prompt, normalization, and evaluation rules while increasing the evaluation data.

Fixes to Implement Next

In the next implementation, I plan to first strengthen the alias map in _normalize_parsed_json().

The aliases I want to add are as follows.

alias_map = {
    "scene": "scene_type",
    "human": "human_presence",
    "human_detection": "human_presence",
    "imageUrlQuality": "image_quality",
    "image_url_quality": "image_quality",
    "summaryJa": "summary_ja",
    "summaryJA": "summary_ja",
    "status_message": "summary_ja",
    "statusMessage": "summary_ja",
    "obstacle": "obstacles",
    "hazard": "potential_hazards",
    "potential_risks": "potential_hazards",
    "potentialRisks": "potential_hazards",
    "recommended_actions": "recommended_action",
    "recommendedActions": "recommended_action",
    "confidence_score": "confidence",
    "confidenceScore": "confidence"
}

I also want to add processing to remove strings that mean "no value" from lists, such as the following.

no_value_strings = {
    "none",
    "n/a",
    "na",
    "null",
    "unknown",
    "no",
    "nothing",
    "no hazards",
    "no hazard",
    "no specific hazards observed",
    "no specific hazard observed",
    "no obstacles",
    "no obstacle",
    "not visible"
}

For example, output like the following,

{
  "obstacles": ["none"],
  "potential_hazards": ["no specific hazards observed"]
}

should ultimately be normalized to the following.

{
  "obstacles": [],
  "potential_hazards": []
}

Current Bottleneck

In the current CLI, the Qwen model is loaded every time a single image is processed.

Therefore, when evaluating multiple images, the execution time becomes long.

Going forward, I would like to build a batch runner like the following.

Initialize QwenLocalVLMClient only once
  ↓
Process multiple images in sequence
  ↓
Save JSON / Markdown / evaluation results

For example, I would like to add app/batch_run.py or an evaluation runner to be able to process around 5 or 30 evaluation images all at once.

Next Steps

The items I plan to work on next are as follows.

1. Add alias mappings
2. Normalize list values like ["none"] or ["no specific hazards observed"] to empty arrays
3. Convert recommended_action to a string when it is returned as a list
4. Add aliases for summary_ja
5. Re-run 5 real images
6. Record the normalized JSON
7. Create an evaluation table
8. Add a batch runner
9. Try Go2 camera integration when there is support from a mentor or team

In the evaluation table, I plan to compare the following items.

Summary

This week, I started with considering the theme of the robot visual inspection agent and was able to progress to a locally working VLM PoC.

Specifically, I implemented the CLI, Pydantic schema, mock VLM, Markdown report generation, and local Qwen backend, and built the entire flow from image input to JSON generation, schema validation, and report generation.

I also compared SmolVLM2 and Qwen2.5-VL, and found that at this point Qwen2.5-VL-3B-Instruct seems to be more suitable for this inspection use case.

Today, I started evaluation using real images taken from a low angle with an iPhone.
As a result, while I was able to confirm that the pipeline works with real images, challenges such as schema inconsistency, JSON format corruption, and instability in risk assessment also became apparent.

The current conclusion is as follows.

"The local VLM pipeline from image input to report generation works.
On the other hand, to make it a practical visual inspection agent,
JSON repair / alias normalization / evaluation are important,
not just prompt engineering."

Next, I will strengthen the normalization processing, re-evaluate the real images, create an evaluation table, and more concretely organize the failure cases and improvement directions.