I tried structured analysis of images and videos with NVIDIA Cosmos-Reason2 on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA's Cosmos-Reason2 is a VLM (Vision-Language Model) that takes video and images as input to reason about cause-and-effect relationships and spatial structures in the physical world. It is designed for Physical AI use cases such as robot action planning and autonomous driving safety judgment.

https://github.com/nvidia-cosmos/cosmos-reason2

When you hear "Reason2," you might think of it as "a model that answers questions about video content in natural language." In my previous articles, I only used it that way. However, looking at the repository again, I found 10 types of structured data prompt templates available — returning bounding box coordinates as JSON, timestamping events in video, and outputting robot movement trajectories as pixel coordinates.

This article focuses on such "structured output," selects 6 from the 10 templates, and introduces the results of hands-on verification with both the 2B and 8B models on a DGX Spark.

Organizing Reason2's Prompt Templates

The Reason2 repository provides 10 types of prompt templates in the prompts/ directory. They can be broadly organized into 3 categories.

Here is a summary of each template.

The top 4 (structured output) are the main focus of this article. caption and causal/causal_vqa are types that respond to video in natural language — these are the features I tried with "pouring water" and "cracking eggs" videos in previous Cosmos articles. mvp_bench is a prompt for evaluating the model's physical understanding, not for inference use.

Reason2 is a VLM based on Qwen3-VL, post-trained with Physical AI-oriented SFT and reinforcement learning (RL). In RL, object recognition accuracy, adherence to spatial constraints, and temporal transition accuracy are evaluated with rule-based rewards, and this training appears to contribute to the quality of structured output.

Verification Environment

Setup

Set up following the repository's README. On DGX Spark, specify --extra cu130 for CUDA 13.0.

git clone https://github.com/nvidia-cosmos/cosmos-reason2.git
cd cosmos-reason2
uv sync --extra cu130

There are two inference methods: direct transformers inference and vLLM server, and the README describes the steps for each. The comparative verification in this article was done with transformers, and operation was also confirmed with vLLM.

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "nvidia/Cosmos-Reason2-8B",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
).to("cuda")

When using vLLM, you need to set TRITON_PTXAS_PATH. If you install by following the README procedure with uv sync --extra cu130, an OpenAI-compatible API will start up with vllm serve.

Memory Comparison: 2B vs 8B

Both have plenty of headroom against the 128GB, so you can choose based on your use case.

Outputting Bounding Boxes with 2D Grounding

This was the feature I most wanted to try. The prompt template is very simple — just specify the object name with Locate the bounding box of {object_name}. Return a json.

Trying with a Robot Arm Scene

I tried detecting 3 objects in a sample image included in the repository (a basket, a tape roll, and 2 robot arms on a table).

8B Results

[
  { "bbox_2d": [459, 632, 568, 815], "label": "blue tape roll" },
  { "bbox_2d": [322, 248, 624, 556], "label": "gray basket" },
  { "bbox_2d": [51, 451, 254, 992], "label": "robot arm" },
  { "bbox_2d": [657, 398, 925, 992], "label": "robot arm" }
]

bbox_2d is in [x1, y1, x2, y2] format, returning the top-left and bottom-right coordinates. The values are normalized coordinates from 0-1000, and to convert to actual pixel positions, you scale with value × actual_size / 1000 using the image's width and height. This is a specification common to Qwen-VL-based models. It's impressive that the 2 robot arms are each detected with individual bounding boxes.

Overlaying on the image looks like this.

I was honestly surprised that the base model without fine-tuning could locate positions this accurately.

What Changes with 2B

Testing the 2B model with the same image, the coordinates for the tape roll and basket are nearly the same, but only 1 of the robot arms was detected.

The coordinate differences are only a few pixels, and the bbox accuracy itself is sufficient even with 2B. However, the 8B model appears to have a higher ability to individually detect multiple objects of the same type.

Checking Base Model Capability with PPE Detection

In the LoRA SFT article, I fine-tuned Reason2 to improve PPE violation detection accuracy from 46.7% to 90.0%. So, how well can the base model's 2D Grounding detect bounding boxes for helmets and vests?

I tested with a construction site image (2 people with helmets, 1 with a safety vest).

Image source: Pexels

The 8B model detected both the white and yellow helmets, and correctly enclosed the safety vest and persons. If the base model can detect this much without fine-tuning, there may be cases where just the base model's 2D Grounding is sufficient depending on the use case.

On the other hand, with the 2B model, the helmet bbox included not just the head but also the upper body, resulting in coarser granularity.

The use case distinction I see here is: 2B is sufficient for roughly understanding what is where, while 8B is needed for pixel-accurate position identification.

Adding Timestamps to Video with Temporal Localization

Next is the feature that takes a video as input and returns the time of event occurrence as a timestamp JSON in mm:ss.ff format.

When I input an 18-second video of a worker carrying cardboard boxes in a warehouse, the 8B model captured the entire scene as a single event and returned a detailed caption in JSON including the worker's clothing and surrounding conditions.

[
  {
    "start": "00:00.200",
    "end": "00:01.200",
    "caption": "A man wearing a blue beanie, a brown and beige jacket, and
     white gloves is seen from behind carrying a large cardboard box on his
     shoulder as he walks forward through a bustling warehouse aisle. In the
     background, several other workers in similar attire are moving about,
     some pulling carts loaded with boxes."
  }
]

Looking at frames from the actual video side by side, you can see the flow of the worker walking through the aisle with a box and passing another worker along the way.

Video source: Pexels

The caption content itself is accurate, capturing the person's clothing (blue beanie, brown and beige jacket), surrounding shelves, and the presence of other workers.

On the other hand, there seems to be room for improvement in terms of consolidating the entire 18-second video into a single event and the accuracy of timestamps. Internally, the video is processed by splitting it into temporal patches of approximately 1-second units with temporal_patch_size=2, so timestamp resolution also depends on this granularity. With longer videos or content with higher event density, it might be split into multiple events.

This feature seems applicable to video indexing in video AI pipelines like VSS (Video Search and Summarization). It could be used to automatically extract "what happened" from warehouse or factory surveillance footage in JSON format.

Reasoning About Robot Operation 2D Trajectories

When given the robot arm scene image with the task instruction "pick up the tape roll and place it in the basket," it returns the 2D trajectory the end-effector should follow as a JSON array of pixel coordinates.

8B: Detailed Trajectory with 10 Points

[
  {"point_2d": [688, 530], "label": "gripper trajectory"},
  {"point_2d": [504, 672], "label": "gripper trajectory"},
  {"point_2d": [505, 692], "label": "gripper trajectory"},
  ...
  {"point_2d": [467, 648], "label": "gripper trajectory"}
]

The output shows a trajectory from near the right robot arm (START) toward the tape roll position (END). It's the first half up to grasping the tape roll, but it's interesting that it can indicate "how this hand should move" in pixel coordinates just by looking at an image.

2B: Only 2 Points + Invalid JSON

The 2B model returned only 2 points, and furthermore the JSON format was invalid (duplicate keys within the same object).

{
  "point_2d": [719, 588],
  "label": "gripper trajectory_1",
  "point_2d": [499, 732],
  "label": "gripper trajectory_2"
}

In terms of structured output accuracy, this was the feature where the difference between 2B and 8B was most pronounced. For tasks like trajectory planning, it seems better to use 8B.

Reasoning About Next Actions with Embodied Reasoning

When asked "what should you do next?" for the same robot arm image, it shows the reasoning process in Chain-of-Thought format.

<think>
Okay, let's try to figure out the next immediate action here.
The video is a lab or workshop with a robot arm and hand on a table.
There's a basket in the center and a roll of blue tape below it.
The robot's hand is on the left, palm up, fingers slightly curled.
The setup suggests the robot is about to interact with the tape or the basket.

Since the tape is below the basket and the robot's hand is open and ready,
maybe the next step is picking up the tape. The hand is in a position to
grasp something, and the tape is nearby. Alternatively, the robot might
move the tape into the basket. But since the tape is below the basket,
maybe the first action is to pick it up. The open hand and the tape's
position make me think grasping the tape roll is the next move.
So the predicted action would be the robot hand picking up the blue tape.
</think>

The robot hand will grasp the roll of blue tape to move or place it
into the gray woven basket.

While Robot CoT returns "how to move (coordinates)," Embodied Reasoning returns "what to do (semantics)." I think it's a strength of the design that you can select templates based on the use case, since the granularity of the output changes completely depending on the prompt design even for the same image.

The 2B model also returned the correct reasoning of "grasp the tape and put it in the basket," showing that 2B has sufficient quality for everyday reasoning tasks.

Safety Reasoning for Autonomous Driving Scenes

I also tried the AV CoT prompt that instructs the model to "identify objects that require attention for safe driving" given dashcam footage.

Interestingly, the 2B model returned a more detailed analysis.

<think>
I see the gray Mazda parked to my left, restricting my lateral movement
and requiring me to maintain a safe distance while avoiding any sudden
maneuvers that could endanger the Mazda driver. Vehicles are parked on
both sides of the street, necessitating careful navigation to avoid
collisions. Crossing the intersection requires full attention to ensure
no vehicles are approaching from either direction.
</think>

gray Mazda, parked vehicles on both sides of the street, intersection signs

The 8B model summarized concisely in 35 tokens.

I am turning right, and this black SUV is coming from the opposite direction, so I should be cautious.
→ Vehicles - Light trucks & SUVs

The 2B gave a comprehensive analysis including parked vehicles and intersection signs, while the 8B focused on immediate dynamic threats (oncoming SUV). Which is the "correct answer" depends on the use case, but from a safety monitoring perspective, the 2B seems less likely to miss things.

Generating Structured Captions with Describe Anything

The Describe Anything prompt is a feature that returns structured data with subject_id / category / caption for multiple objects in an image.

[
  {
    "subject_id": "1",
    "category": "robot hand",
    "caption": "Positioned on the left side of the table, this black robot hand
     features four distinct fingers, each with a segmented design that mimics
     human anatomy; it is connected to a sleek, cylindrical base equipped with
     visible joints and a small circular opening at its upper part."
  },
  {
    "subject_id": "2",
    "category": "robot hand",
    "caption": "Located on the right side of the table, this black robot hand
     also has four fingers with a segmented structure, similar in design to the
     left hand; it is attached to a cylindrical base that includes a noticeable
     circular cutout near the top."
  }
]

The 8B described the 2 robot hands in detail, but did not detect the tape roll or basket. The 2B, on the other hand, detected 3 categories: basket / tape / hand.

Combined with the AV CoT results, there seems to be a tendency for 2B to be "broad and shallow" and 8B to be "narrow and deep."

Performance Summary

Here are the results of running all 6 features with both 2B and 8B visualized.

The 2B is faster than the 8B for most features. However, there is a slightly misleading point when comparing only by TPS. For Robot CoT, the 8B's TPS of 4.6 exceeds the 2B's 2.0, but this is because the 8B outputs a detailed trajectory of 10 points and 257 tokens all at once, and the actual inference time is longer for 8B (56.3 sec vs 29.8 sec). Since TPS tends to be higher when the number of output tokens is large, it's better to look at inference time as well as TPS.

The 8B is generally higher in quality, but the 2B was more detailed for Scene Reasoning (AV CoT). The choice may vary depending on whether you prioritize "accuracy" or "comprehensiveness."

Relationship Between Video Length and Inference Time

I measured how inference time for Temporal Localization scales by varying the video length.

Interestingly, the inference time was nearly constant regardless of video length — approximately 6 seconds for 2B and approximately 13 seconds for 8B. Even when the frame count increased from 10 to 36, the generation of output tokens accounts for most of the inference time, so the increase in input frames doesn't seem to affect the overall time much.

As a result, the 2B can complete analysis faster than real-time for videos of 10 seconds or more, and the 8B for videos of 15 seconds or more. While not real-time streaming processing, for the use case of analyzing surveillance footage after the fact, it seems sufficiently practical speed can be achieved with a single DGX Spark.

Thinking About Practical Scenarios

The 6 features I tried seem to show their true value when combined rather than used individually.

For example, in a video AI pipeline, you could first use temporal_localization to decompose long surveillance footage into JSON describing "what" happened "when," then apply 2d_grounding to the relevant frames of events of interest, creating a structured index that includes object locations. Combined with the Event Reviewer from VSS verified in a previous article, it could automate the attachment of bounding boxes to detected events.

In the robotics context, a two-stage reasoning approach is conceivable: use embodied_reasoning to make task-level judgments such as "grasp the tape and put it in the basket," then use robot_cot to translate that into specific pixel coordinate trajectories. Using trajectory coordinates directly for robot control is challenging in terms of accuracy, but it's sufficient as an initial hypothesis for planning.

Combining with the LoRA SFT from the previous article is also an interesting direction. A configuration where the base model's 2d_grounding broadly detects objects and a domain-specific VLM fine-tuned with LoRA analyzes those cropped regions in detail would allow a division of roles: "position identification by the base model, judgment by the custom model."

Summary

I verified 6 structured output features of Cosmos-Reason2 on a DGX Spark.

The bounding box output of 2D Grounding was able to accurately detect PPE helmets and vests at a level achievable by the base model without fine-tuning. In the LoRA SFT article, I trained for text classification, but I found that 2D Grounding can be used as-is when location information is needed.

For Temporal Localization and Robot CoT, the 8B clearly returned higher-quality results. On the other hand, for reasoning tasks like AV CoT, the 2B also performed well, and the relationship between model size and quality is not uniform.

With the 128GB unified memory of the DGX Spark, both the 2B (4.3 GB) and 8B (17.5 GB) operate with plenty of headroom. Having an environment where you can choose based on the purpose is convenient for verification.

Cosmos 3 was also announced at GTC 2026, and I'm curious to see how structured output will change.

Reference Links

• nvidia-cosmos/cosmos-reason2 — Official repository (10 prompt templates)
• Scale Synthetic Data and Physical AI Reasoning with NVIDIA Cosmos World Foundation Models — NVIDIA Developer Blog
• Cosmos-Reason2-8B (HuggingFace) — Model card
• Cosmos-Reason2-2B (HuggingFace) — Model card
• Running NVIDIA Cosmos World Foundation Models on DGX Spark — Previous Cosmos article
• Fine-tuning Cosmos-Reason2-8B for PPE Detection on DGX Spark — Previous LoRA SFT article