I tried structured analysis of images and videos with NVIDIA Cosmos-Reason2 on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA's Cosmos-Reason2 is a VLM (Vision-Language Model) that takes video and images as input to reason about causal relationships and spatial structures in the physical world. It is designed for Physical AI use cases such as robot action planning and autonomous driving safety decisions.

https://github.com/nvidia-cosmos/cosmos-reason2

When you hear "Reason2," you might think of it as "a model that answers questions about video content in natural language." In my previous articles, I only used it that way. However, looking at the repository again, I found that it has 10 prompt templates for returning structured data — including bounding box coordinates in JSON, timestamps for events in video, and robot movement trajectories in pixel coordinates.

In this article, I focus on these "structured outputs," select 6 of the 10 templates, and present the results of hands-on verification with both the 2B and 8B models on a DGX Spark.

Organizing Reason2's Prompt Templates

The Reason2 repository provides 10 prompt templates in the prompts/ directory. They can be broadly organized into 3 categories.

Here is a summary of each template.

The top 4 (structured outputs) are the main focus of this article. The caption and causal/causal_vqa templates return natural language responses to video, which is the functionality I tested in previous Cosmos articles with videos of "pouring water" and "cracking eggs." mvp_bench is a prompt for evaluating a model's physical understanding and is not intended for inference use.

Reason2 is a VLM based on Qwen3-VL, post-trained with SFT and reinforcement learning (RL) for Physical AI. In RL, object recognition accuracy, adherence to spatial constraints, and temporal accuracy are evaluated with rule-based rewards, and this training appears to be what drives the quality of structured outputs.

Verification Environment

Setup

Follow the repository's README for setup. On DGX Spark, specify --extra cu130 for CUDA 13.0.

git clone https://github.com/nvidia-cosmos/cosmos-reason2.git
cd cosmos-reason2
uv sync --extra cu130

There are two inference methods: direct transformers inference and a vLLM server, and the README describes the steps for each. The comparative verification in this article was done with transformers, and operation was also confirmed with vLLM.

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "nvidia/Cosmos-Reason2-8B",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
).to("cuda")

When using vLLM, you need to configure TRITON_PTXAS_PATH. If you install following the README's steps with uv sync --extra cu130, vllm serve will bring up an OpenAI-compatible API.

Memory Comparison: 2B vs 8B

Both have plenty of headroom against 128GB, so you can choose based on your use case.

Outputting Bounding Boxes with 2D Grounding

This is the feature I most wanted to try. The prompt template is very simple — you just specify an object name with Locate the bounding box of {object_name}. Return a json.

Testing with a Robot Arm Scene

I tried detecting 3 objects in the sample image included in the repository (a basket, a tape roll, and 2 robot arms on a table).

8B Results

[
  { "bbox_2d": [459, 632, 568, 815], "label": "blue tape roll" },
  { "bbox_2d": [322, 248, 624, 556], "label": "gray basket" },
  { "bbox_2d": [51, 451, 254, 992], "label": "robot arm" },
  { "bbox_2d": [657, 398, 925, 992], "label": "robot arm" }
]

bbox_2d is in [x1, y1, x2, y2] format, returning the coordinates of the top-left and bottom-right corners. The values are normalized coordinates from 0–1000, and to convert to actual pixel positions you scale with value × actual_size / 1000 for the image's width and height. This is a common specification in Qwen-VL-based models. What's impressive is that both robot arms are detected individually with separate bounding boxes.

Here's what it looks like overlaid on the image.

Honestly, it was surprising to see this level of positional accuracy from a base model without any fine-tuning.

What Changes with 2B

Testing with the same image on the 2B model, the coordinates for the tape roll and basket are nearly the same, but only one robot arm was detected on the right side.

The coordinate differences are only a few pixels, and the bbox accuracy itself is sufficient with 2B. However, the ability to individually detect multiple objects of the same type appears to be higher with 8B.

Checking Base Model Capability with PPE Detection — No Fine-tuning

In the LoRA SFT article, I fine-tuned Reason2 to improve PPE violation detection accuracy from 46.7% to 90.0%. So how well can the base model's 2D Grounding detect bounding boxes for helmets and vests?

I tested with a construction site image (2 workers with helmets, 1 with a safety vest).

Image source: Pexels

The 8B model detected both the white and yellow helmets, and correctly bounded the safety vest and persons as well. If the base model's 2D Grounding can detect this well without fine-tuning, there may be use cases where the base model alone is sufficient.

On the other hand, the 2B model produced bounding boxes for the helmets that included not just the head but the upper body as well, resulting in coarser granularity.

From this, a practical division seems to be: 2B is sufficient for roughly understanding "what is where," while 8B is better for pixel-accurate localization.

Timestamping Video with Temporal Localization

Next is a feature that takes video as input and returns JSON with event timestamps in mm:ss.ff format.

When I input an 18-second video of a worker carrying cardboard boxes in a warehouse, the 8B model captured the entire scene as a single event and returned a detailed caption in JSON, including the worker's attire and surrounding context.

[
  {
    "start": "00:00.200",
    "end": "00:01.200",
    "caption": "A man wearing a blue beanie, a brown and beige jacket, and
     white gloves is seen from behind carrying a large cardboard box on his
     shoulder as he walks forward through a bustling warehouse aisle. In the
     background, several other workers in similar attire are moving about,
     some pulling carts loaded with boxes."
  }
]

Looking at frames from the actual video side by side, you can see the flow of the worker carrying a box through the aisle and passing another worker along the way.

Video source: Pexels

The caption content itself is accurate, capturing the person's clothing (blue beanie, brown and beige jacket), surrounding shelves, and the presence of other workers.

On the other hand, there is room for improvement in terms of grouping the entire 18-second video into a single event and in timestamp precision. Internally, the video is processed by splitting it into temporal patches of approximately 1 second each using temporal_patch_size=2, so the timestamp resolution also depends on this granularity. With longer videos or footage with higher event density, the output might split into multiple events.

This feature seems well-suited for video indexing in AI pipelines like VSS (Video Search and Summarization). It could be used to automatically extract "what happened" from warehouse or factory surveillance footage in JSON format.

Inferring 2D Robot Manipulation Trajectories

Given an image of a robot arm scene with the task instruction "pick up the tape roll and place it in the basket," the model returns the 2D trajectory the end-effector should follow as a JSON array of pixel coordinates.

8B: Detailed 10-Point Trajectory

[
  {"point_2d": [688, 530], "label": "gripper trajectory"},
  {"point_2d": [504, 672], "label": "gripper trajectory"},
  {"point_2d": [505, 692], "label": "gripper trajectory"},
  ...
  {"point_2d": [467, 648], "label": "gripper trajectory"}
]

The output shows a trajectory from near the right robot arm (START) toward the tape roll position (END). This covers the first half of the motion up to grasping the tape roll, but it's fascinating that the model can indicate "how this hand should move" in pixel coordinates just from looking at an image.

2B: Only 2 Points + Invalid JSON

The 2B model returned only 2 points, and the JSON format was invalid (duplicate keys within the same object).

{
  "point_2d": [719, 588],
  "label": "gripper trajectory_1",
  "point_2d": [499, 732],
  "label": "gripper trajectory_2"
}

In terms of structured output accuracy, this is where the gap between 2B and 8B was most pronounced. For tasks like trajectory planning, 8B is clearly the better choice.

Reasoning About the Next Action with Embodied Reasoning

When asked "what should be done next?" about the same robot arm image, the model shows its reasoning process in Chain-of-Thought format.

<think>
Okay, let's try to figure out the next immediate action here.
The video is a lab or workshop with a robot arm and hand on a table.
There's a basket in the center and a roll of blue tape below it.
The robot's hand is on the left, palm up, fingers slightly curled.
The setup suggests the robot is about to interact with the tape or the basket.

Since the tape is below the basket and the robot's hand is open and ready,
maybe the next step is picking up the tape. The hand is in a position to
grasp something, and the tape is nearby. Alternatively, the robot might
move the tape into the basket. But since the tape is below the basket,
maybe the first action is to pick it up. The open hand and the tape's
position make me think grasping the tape roll is the next move.
So the predicted action would be the robot hand picking up the blue tape.
</think>

The robot hand will grasp the roll of blue tape to move or place it
into the gray woven basket.

While Robot CoT returns "how to move (coordinates)," Embodied Reasoning returns "what to do (semantics)." The fact that the same image produces completely different output granularity depending on the prompt design reflects the strength of being able to choose a template based on the use case.

The 2B model also returned correct reasoning — "grab the tape and place it in the basket" — showing that 2B is sufficient quality for everyday reasoning tasks.

Safety Reasoning for Autonomous Driving Scenes

I also tried the AV CoT prompt, which instructs the model to "identify objects that require attention for safe driving" given dashcam footage.

Interestingly, the 2B model returned a more detailed analysis.

<think>
I see the gray Mazda parked to my left, restricting my lateral movement
and requiring me to maintain a safe distance while avoiding any sudden
maneuvers that could endanger the Mazda driver. Vehicles are parked on
both sides of the street, necessitating careful navigation to avoid
collisions. Crossing the intersection requires full attention to ensure
no vehicles are approaching from either direction.
</think>

gray Mazda, parked vehicles on both sides of the street, intersection signs

The 8B model summarized concisely in 35 tokens.

I am turning right, and this black SUV is coming from the opposite direction, so I should be cautious.
→ Vehicles - Light trucks & SUVs

The 2B model gave a comprehensive analysis including parked vehicles and intersection signs, while the 8B model made a quick, focused judgment on the dynamic threat (oncoming SUV). Which is "correct" depends on the use case, but from a safety monitoring perspective, 2B seems less likely to miss things.

Generating Structured Captions with Describe Anything

The Describe Anything prompt returns structured data with subject_id / category / caption for multiple objects in an image.

[
  {
    "subject_id": "1",
    "category": "robot hand",
    "caption": "Positioned on the left side of the table, this black robot hand
     features four distinct fingers, each with a segmented design that mimics
     human anatomy; it is connected to a sleek, cylindrical base equipped with
     visible joints and a small circular opening at its upper part."
  },
  {
    "subject_id": "2",
    "category": "robot hand",
    "caption": "Located on the right side of the table, this black robot hand
     also has four fingers with a segmented structure, similar in design to the
     left hand; it is attached to a cylindrical base that includes a noticeable
     circular cutout near the top."
  }
]

The 8B model described the two robot hands in great detail but did not detect the tape roll or basket. The 2B model, on the other hand, detected 3 categories: basket / tape / hand.

Combined with the AV CoT results, there seems to be a tendency for 2B to go "broad and shallow" and 8B to go "narrow and deep."

Performance Summary

Here are the results of running all 6 features with both 2B and 8B models, visualized.

The 2B model is faster than 8B for most features. However, comparing by TPS alone can be slightly misleading. For Robot CoT, the 8B TPS of 4.6 exceeds 2B's 2.0, but this is because 8B outputs a detailed 10-point, 257-token trajectory all at once, and the actual inference time is longer for 8B (56.3 sec vs 29.8 sec). Since higher output token counts tend to produce higher TPS, it's better to look at inference time alongside TPS.

In terms of quality, 8B is generally higher overall, but 2B was more detailed for Scene Reasoning (AV CoT). The choice may come down to whether you prioritize "precision" or "coverage."

Relationship Between Video Length and Inference Time

I measured how inference time scales with video length for Temporal Localization.

Interestingly, inference time was nearly constant regardless of video length — about 6 seconds for 2B and about 13 seconds for 8B. Even when the frame count increased from 10 to 36, the output token generation accounts for most of the inference time, so the increase in input frames has little overall impact.

As a result, 2B can complete analysis faster than real-time for videos over 10 seconds, and 8B for videos over 15 seconds. While this isn't real-time streaming processing, for analyzing surveillance footage after the fact, DGX Spark shows sufficiently practical speeds for a single machine.

Thinking About Practical Scenarios

The 6 features I tested this time seem to show their true value when combined rather than used individually.

For example, in a video AI pipeline, you could first use temporal_localization to decompose long surveillance footage into a JSON of "what" happened "when," then apply 2d_grounding to the relevant frames of events of interest to create a structured index that includes object positions. Combined with the Event Reviewer from VSS, which I verified in a previous article, it would be possible to automate bounding box assignment to detected events.

In a robotics context, you could use embodied_reasoning to make a task-level decision like "pick up the tape and place it in the basket," then use robot_cot to translate that into a specific pixel-coordinate trajectory — a two-stage reasoning approach. Using the trajectory coordinates directly for robot control would be challenging in terms of accuracy, but they could serve well as initial hypotheses for planning.

Combining with the LoRA SFT from last time is also an interesting direction. You could use the base model's 2d_grounding to roughly detect objects, then have a domain-specialized VLM fine-tuned with LoRA analyze those cropped regions in detail — a "base model for localization, custom model for classification" division of labor.

Summary

I verified 6 structured output features of Cosmos-Reason2 on DGX Spark.

The bounding box output from 2D Grounding was able to accurately detect PPE helmets and vests with the base model without any fine-tuning. In the LoRA SFT article, I trained for text classification, but I've now confirmed that 2D Grounding can be used as-is when positional information is needed.

Temporal Localization and Robot CoT clearly returned higher-quality results with 8B. On the other hand, for reasoning tasks like AV CoT, 2B held its own, so the relationship between model size and quality is not uniform.

With the 128GB unified memory of DGX Spark, both the 2B (4.3 GB) and 8B (17.5 GB) models run comfortably. Having an environment where you can switch between them depending on the purpose is very convenient for verification.

Cosmos 3 was also announced at GTC 2026, and I'm curious to see how structured outputs will evolve.

Reference Links

• nvidia-cosmos/cosmos-reason2 — Official repository (10 prompt templates)
• Scale Synthetic Data and Physical AI Reasoning with NVIDIA Cosmos World Foundation Models — NVIDIA Developer Blog
• Cosmos-Reason2-8B (HuggingFace) — Model card
• Cosmos-Reason2-2B (HuggingFace) — Model card
• Running NVIDIA Cosmos World Foundation Models on DGX Spark — Previous Cosmos article
• Fine-tuning Cosmos-Reason2-8B for PPE Detection on DGX Spark — Previous LoRA SFT article