I tried NVIDIA's agent execution runtime NeMo Relay

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Did you know that a runtime has been added to NVIDIA's NeMo family that takes over the actual execution of agents? It's called NeMo Relay.

https://github.com/NVIDIA/NeMo-Relay

When running agents, it's surprisingly hard to see what's happening inside. Which tools were called how many times, how many round trips were made with the LLM, how many tokens were used in a single turn. Relay captures all of that from outside the agent and streams it in formats like OpenTelemetry. The selling point is that you don't need to modify your existing agents.

When you hear "agent" and "NeMo" together, you might think of NeMo Agent Toolkit, but this is a different thing. While Agent Toolkit is a library for building agent workflows in Python, Relay is a runtime written in Rust that sits in a layer supporting the execution boundary of assembled agents from below. In the official layer diagram, Relay is placed below Agent Toolkit, and there's also a path drawn for frameworks to directly call Relay.

Claude Code was on the list of supported agents, so I tried running it in my local environment.

To share my impressions upfront, while installation takes just one command, what you can get varies considerably depending on the output destination. Token counts don't come out from the OpenTelemetry output destination. Knowing this kind of discrepancy in advance saves you from later realizing "the numbers I wanted aren't included."

I wrote previously about NVIDIA's LLM routing infrastructure (article as of 2026-07-03). Relay sits one layer above the layer that was deciding "which model to send to" back then.

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

This article introduces what NeMo Relay can do, and then describes what I was actually able to capture by piggybacking on Claude Code.

What NeMo Relay Does and Doesn't Take On

NeMo Relay is an OSS that NVIDIA published at the end of March 2026, licensed under Apache-2.0 and written in Rust. The stable version is 0.6.0 (2026-07-22). This article was verified with this version.

It was originally called NeMo Flow, with 0.1.0 and 0.2.0 released under that name. It was renamed at 0.3.0, but the repository was carried over as-is, and old URLs redirect to the current NeMo Relay. The reason migration support features from NeMo Flow remain in the documentation is due to this history.

What struck me about its positioning was this passage in the documentation:

A framework asks, "What should the agent do next?" NeMo Relay asks, "When the agent does work, which scope owns it, which middleware applies, what events are emitted, and which subscribers can consume the result?"

It's a clear separation: making the agent think about "what to do next" is the framework's job, while what Relay takes on is "when work is actually done, who owns it, what gets recorded, and who can receive the results."

What it doesn't take on is also explicitly stated: agent frameworks, model providers, guardrail description layers, deployment infrastructure. It's written that Relay is not a replacement for these, and observation backends like Langfuse and Arize Phoenix are also placed "outside of Relay." The organization is that Relay is the event emitter, while storing and displaying them is the job of a separate layer.

https://docs.nvidia.com/nemo/relay/about-nemo-relay/ecosystem

Division of Roles with Switchyard

Since it's easy to confuse with Switchyard from the same NeMo family, let me clarify upfront.

Since Relay calls Switchyard's Decision API, Relay is the higher layer. However, Switchyard integration is marked as Experimental and not included in the default build, so I didn't touch it this time.

Only 3 Agents Are Supported

What surprised me when looking at the compatibility table was how limited the targets are.

Cursor was on the compatibility table up through the 0.4 series, but disappeared at 0.5. You may sometimes find information online saying Cursor is supported, but that's just picking up pages from older versions.

There's also a caveat for Claude Code — only the CLI is supported.

Claude desktop, web, and application sessions are unsupported unless they expose the same local hook and gateway controls.

This means desktop app or web sessions are outside the scope of observation. Similarly for Codex, it's noted that "cloud executions that don't go through the local machine will have partial or no LLM capture." The constraint that you can only see what's running locally is something worth understanding from the start.

Distribution platforms include Linux x86_64 and ARM64, macOS Apple Silicon, Windows x86_64 and ARM64. There are no pre-built binaries for Intel Mac (a workaround of building from source via cargo install is mentioned). If even one person on your team has an Intel Mac, that person will fall out of observation.

Incidentally, within the scope of my research, I couldn't find any articles in Japanese or English from people who had actually tried this. Since information sources are essentially limited to official documentation and GitHub, please note that there's currently nowhere to cross-reference the numbers in this article.

There Are 4 Telemetry Output Destinations and What You Can Get Differs

Here's the main topic. Relay's observability plugins have 4 output destinations, projecting the same events into different forms. If you build without understanding these differences, you'll later find that "the numbers you wanted aren't included."

First, two are file outputs. ATOF (Agent Trajectory Observability Format) is a raw event stream, JSONL with one event per line. It retains the usage returned by the provider as-is. ATIF (Agent Trajectory Interchange Format) is one file per agent scope, containing a normalized trajectory.

The remaining two are projections that send via OTLP to an external destination: OpenTelemetry and OpenInference. Both can be sent to the same OTLP endpoint.

The official documentation has a table called Exporter Field Mapping, which was the most impactful thing I found this time.

Notice that the OpenTelemetry column has no tokens at all. The documentation states this clearly:

Token counts are not emitted as discrete attributes.

This is intentional behavior, and the release notes also explicitly state "OpenTelemetry emits cost only, not token counts." It was also written that this may change in the future.

ATIF also has quirks. Cache read and cache write are combined into cached_tokens, so if you want to see the breakdown, you need ATOF or OpenInference. Marks (point events) are also dropped in ATIF, so ATOF becomes the authoritative source.

In other words, the practical solution became: pick up cost from OpenTelemetry and token counts from OpenInference. Fortunately, in 0.6, both can coexist as independent sections.

There's a Full Set of Features Beyond Observation

Observation is the main feature, but there are other plugins too. I didn't use them this time, so I'll just mention what's available.

PII redaction is a mechanism for masking personal information before sending. As of 0.6, it's only a deterministic local backend, with model-based judgment reserved for the future. There's also NeMo Guardrails integration, which lets you insert runtime guardrails around managed LLM calls. Writing policies is the job of the upper layer, while Relay hosts them.

The Adaptive category includes Cache Governor and tool parallelism — these fall under the optimization lineage rather than observation. The plugin mechanism itself also comes in two types: a native type that runs in-process, and a gRPC worker type that runs in a separate process.

It's reassuring to have a complete set of extension points, but since it's a 0.x OSS with minor releases coming out monthly, I'd want to be careful about how much to dive in. This time I limited myself to observation only.

Installing and Piggybacking on Claude Code

Installation is a single command.

curl -fsSL https://raw.githubusercontent.com/NVIDIA/NeMo-Relay/main/install.sh | sh

To pin a version, pass it as an environment variable. There's one pitfall here — if you get the position of the variable wrong, it won't take effect.

NEMO_RELAY_VERSION=0.6.0 curl -fsSL https://.../install.sh | sh

curl -fsSL https://.../install.sh | NEMO_RELAY_VERSION=0.6.0 sh

The installer is a 258-line shell script that doesn't call sudo and doesn't rewrite shell configuration files. It downloads a binary from GitHub Releases, verifies SHA-256, and if there's a mismatch, it fails without replacing the existing binary. Only a single 31MB executable is placed in $HOME/.local/bin, so I was comfortable running this as-is.

Choosing Between Persistent and Ephemeral

There are two ways to integrate with Claude Code. This choice has lasting consequences, so let me organize it in a table first.

I chose ephemeral. The reason is described in Wall 2 later, but persistent modifies ~/.claude/settings.json. People who have this settings file as a symlink to a dotfiles repository will find it broken.

I verified empirically that ephemeral truly doesn't touch anything. Adding --dry-run --print shows only what it intends to do.

$ nemo-relay run --dry-run --print -- claude
agent = claude
gateway_url = http://127.0.0.1:65412
anthropic_base_url = https://api.anthropic.com
argv = claude --plugin-dir <temporary-claude-plugin-dir> --settings <temporary-claude-settings>
env.NEMO_RELAY_GATEWAY_URL = http://127.0.0.1:65412
env.NEMO_RELAY_TRANSPARENT_RUN = 1
env.ANTHROPIC_BASE_URL = http://127.0.0.1:65412
note = would generate a temporary Claude Code plugin directory

ANTHROPIC_BASE_URL is only passed to the process's environment variables, and the plugin directory and settings file are created in a temporary area. I took the sha256 of ~/.claude/settings.json before and after execution and confirmed the values hadn't changed. The symlink remained a symlink.

Verifying Connectivity

You can diagnose your configuration with doctor. Here I noticed something else — doctor has two ways to pass arguments.

nemo-relay doctor claude              # positional argument (claude / codex / hermes)
nemo-relay doctor --plugin claude-code  # flag (codex / claude-code / hermes / all)

This looks like inconsistent naming, but the values they take are different (claude vs claude-code). Positional arguments diagnose the agent, while --plugin diagnoses the persistently installed plugin — they're different things. For ephemeral usage, use the former, or run without arguments for a full diagnosis.

Running without arguments outputs the configuration file search results and observability plugin validation together.

  Observability
    ✓  Plugin validation       validation passed
    ✓  ATOF file sink          sinks[0]: ~/relay-poc/atof (appears writable)
    ✓  ATIF dir                ~/relay-poc/atif (appears writable)
    !  OpenTelemetry endpoint  http://<collection-server>:4318/v1/traces (HTTP 405)
    !  OpenInference endpoint  http://<collection-server>:4318/v1/traces (HTTP 405)

The HTTP 405 on endpoints appears as a warning, but this is because doctor probes with GET. Since OTLP only accepts POST, it returns 405. The connection itself is working, so this warning can be ignored.

Configuration files go in ~/.config/nemo-relay/plugins.toml. The search order is a bit unusual: system → project → user, with user being the strongest. This is the reverse of typical configuration systems where the closer one wins, so items you want to change per project will lose if written on the user side.

Setting Up a Receiver on DGX Spark and Viewing Traces

I set up the OTLP receiver on my local DGX Spark. Three components: OpenTelemetry Collector, Tempo, and Grafana, with actual memory usage of about 220MB combined. Light enough to run as a background service without concern.

On the Relay side, you just write the endpoint in plugins.toml. In 0.6, OpenTelemetry and OpenInference are independent sections, so both can be enabled simultaneously.

[components.config.opentelemetry]
enabled = true
transport = "http_binary"
endpoint = "http://<collection-server>:4318/v1/traces"
service_name = "claude-code"

[components.config.openinference]
enabled = true
transport = "http_binary"
endpoint = "http://<collection-server>:4318/v1/traces"
service_name = "claude-code-openinference"

I'm sending to the same endpoint, but separating the service_name lets me filter by TraceQL later.

The Collector and Tempo compose setup is almost the same flow as the article I wrote about setting up Langfuse on DGX Spark (article as of 2026-05-02), so I'll defer to that.

https://dev.classmethod.jp/articles/langfuse-self-host-llm-observability-handson/

After running Claude Code for one turn, traces become visible in Grafana.

One user turn becomes a single trace called claude-code-turn. Of the 11.95 seconds total, you can see the breakdown: first LLM call 4.25 seconds, Read 21.88ms, Bash 367.54ms.

I observed 4 types of span names. The trace root claude-code-turn, LLM calls as anthropic.messages, and tool executions use the tool name directly as the span name. Point events like session.start appear as zero-length spans named mark:session.start.

Since the official documentation doesn't list span names, this is from actual measurement. However, I haven't triggered sub-agents or context compression yet, so there should be more types.

Note that enabling both OpenTelemetry and OpenInference causes the same span to appear twice in the same trace. That's why anthropic.messages and Read appear duplicated in the screen above. Since they share the same trace ID, Grafana mixes them into a single tree. You need to filter by service.name when reading.

Not a Single Token Comes Out of the OpenTelemetry Projection

Now for the key part.

When I listed the attributes in Tempo, I found 3 cost-related ones.

nemo_relay.llm.cost.total = 0.0541685
nemo_relay.llm.cost.currency = USD
llm.cost.total = 0.0541685

As for tokens, those only exist on the OpenInference side.

llm.token_count.prompt                     = 2
llm.token_count.completion                 = 4
llm.token_count.total                      = 6
llm.token_count.prompt_details.cache_read  = 108117
llm.token_count.prompt_details.cache_write = 0

As you can tell from the attribute name prefixes, llm.token_count.* is what the OpenInference projection emits. The nemo_relay.* side has no token counts. This matched the documented behavior.

What's interesting here is the attribute nemo_relay.end.data.usage, which contains the raw usage as a complete JSON string.

{
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 108117,
  "input_tokens": 2,
  "output_tokens": 4,
  "output_tokens_details": { "thinking_tokens": 0 }
}

So the information itself is being sent over OTLP. It's just not decomposed into individual attributes. If you want to write conditions like llm.token_count.prompt > 1000 in TraceQL, you'll need to look at the OpenInference side or process the attributes in the Collector.

A single turn of just asking "please answer ok" resulted in 108,117 cache read tokens. That's because CLAUDE.md, skills, and tool definitions are all loaded into the cache. Seeing this number immediately made it clear that cache is the main front in cost analysis.

There's Nowhere to Write the Price of Cache Writes

I said cost attributes were appearing, but that's after configuring a pricing table. The situation was different before configuring it.

Relay doesn't bundle a pricing table. The documentation also states "if no configured source exists, all models are treated as unknown." While it prioritizes provider-returned amounts when they're in the response, Anthropic's usage has no price field. Looking at Claude Code session logs, only 4 types of token counts are included, so when going through Anthropic, it always falls back to estimation from the pricing table.

In fact, when I counted attributes in Tempo before adding a pricing table, there were 230, with 0 containing cost. After adding the pricing table, it became 233, with those 3 added.

Writing the Pricing Table

The catalog is written in JSON and registered via CLI.

{
  "version": 1,
  "entries": [
    {
      "provider": "anthropic",
      "model_id": "claude-opus-5",
      "currency": "USD",
      "unit": "per_token",
      "rates": {
        "input_per_million": 5.0,
        "output_per_million": 25.0,
        "cache_read_per_million": 0.5
      },
      "prompt_cache": { "read_accounting": "separate" },
      "pricing_as_of": "2026-08-04"
    }
  ]
}

nemo-relay model-pricing validate claude-pricing.json
nemo-relay model-pricing add-source claude-pricing.json --user
nemo-relay model-pricing resolve claude-opus-5 --provider anthropic \
  --prompt-tokens 1000 --completion-tokens 500

resolve returned estimated_total = 0.0175. Input 1000 tokens at $0.005, output 500 tokens at $0.0125. The math checks out.

You may have noticed that rates only accepts 3 entries: input, output, cache read. There's nowhere to write the unit price for cache writes.

It's not that Relay is unaware of cache writes. Reading the codec documentation, it states that Usage.cache_write_tokens is correctly mapped from Anthropic's cache_creation_input_tokens, and the cost structure also has a cache_write category. It has the tokens, but there's just no path to tell it the price.

What's more, even if you add cache_write_per_million as a key on your own, validation still passes.

$ nemo-relay model-pricing validate test.json
Valid model pricing catalog: test.json (2 entries)

No error, no warning — silently ignored. The tricky part is that you can think you wrote it correctly.

Measuring What Actually Happens

I ran the same prompt twice with different cache states.

First, a turn where the cache was hit:

Raw usage  : cache_creation 0 / cache_read 108,117 / input 2 / output 4
ATIF       : total_cost_usd 0.0541685

Checking: 2×$5/M + 4×$25/M + 108,117×$0.50/M = 0.0541685. Exact match. Cache reads are correctly accounted for.

Next, a turn where cache writes occurred. I ran it in a different directory with a different prompt.

Raw usage  : cache_creation 59,017 / cache_read 0 / input 2 / output 4
ATIF       : total_cached_tokens 59,017 / total_cost_usd 0.00011

0.00011 is the cost for just 2 input tokens and 4 output tokens. The 59,017 cache write tokens have completely disappeared from the cost. The token count appears in total_cached_tokens, but it's not reflected in the price.

According to Anthropic's pricing, cache writes with 1-hour TTL are 2x the input unit price. Claude Code uses 1-hour TTL (visible in the raw usage as ephemeral_1h_input_tokens). At Opus 5's input unit price of $5/M, that means $10/M, which would amount to about $0.59.

The 5,366x figure is impactful, but this is for a single turn where only cache writes occurred, so it doesn't directly translate to monthly costs. To get a sense of real-world impact, let me compare with my actual measurements.

Aggregating a fully active month in June 2026 with ccusage, I had 79.77 million cache write tokens. At Opus 5 pricing, that's equivalent to $798. The total for that same month was $3,008, meaning about 20% of costs would be missing from the aggregate. It's not an order-of-magnitude difference, but it's not negligible either.

Trying to look at costs for cache-heavy agents using only Relay seems tough at this point. Since token counts are captured accurately, calculating costs yourself seems like the practical approach.

Prompt Content Reaches the Center by Default

One more thing that directly impacts adoption decisions.

When sending observability data from my local machine to an external destination, I had the premise that "prompt content and code content wouldn't be sent." Having the actual work content flowing out goes beyond the purpose of observation.

When I checked what was actually flowing, it was being sent by default.

nemo_relay.start.data.prompt = Reply with exactly: ok
input.value                  = user: <system-reminder> As you answer the user's ...
output.value                 = ok
llm.input_messages.0.message.content = ...

The prompt I sent, the entire conversation including system reminders, and the response body are all in the span attributes.

Expanding a span shows input.value containing user: <system-reminder> As y… with the conversation content. The service.name on the Resource attributes side is claude-code-openinference.

Furthermore, tool definition JSON schemas from llm.tools.0 through llm.tools.116 — 117 tool definitions — were loaded as attributes. The ATIF file being 1.25MB per turn is mainly due to this.

To configure a setup that doesn't send content, you explicitly configure sanitizers or PII redaction. This is not "safe by default," so it's worth checking before distributing to a team. It doesn't matter when running only in your own environment, but when someone else's screen content starts flowing into your Grafana, the situation changes.

Walls I Hit Before Getting It to Piggyback

I said installation is a single command, but I hit 3 walls getting there. Since all of them could occur in a reader's environment, I'll keep the details collapsed.

curl -fsSL https://.../install.sh | NEMO_RELAY_VERSION=0.6.0 sh

{
  "env": { "ANTHROPIC_BASE_URL": "http://127.0.0.1:47632" },
  "enabledPlugins": { "nemo-relay-plugin@nemo-relay-local": true },
  "extraKnownMarketplaces": { "nemo-relay-local": { "source": { "path": "...", "source": "directory" } } }
}

Before install:  lrwxr-xr-x  .claude/settings.json -> dotfiles/settings.json
After install:   -rw-r--r--  .claude/settings.json

Install state: failed (missing or invalid state at .../claude-code.json)
Host registration: failed (plugin or marketplace registration is incomplete)
claude provider routing: failed (not configured)
installer error: Claude Code plugin doctor checks failed; remediation: nemo-relay install claude-code --force

Things I'm Curious About

After trying everything out, here's a summary of the points that caught my attention at this stage.

The biggest one is the observation scope. As noted in the official Known Issues, Relay can only capture traffic that passes through local hooks and the gateway — it cannot capture execution on remote or cloud environments. Claude Code sessions outside the CLI are also out of scope. This matters when you want to have a complete picture of how it's being used. The same applies to the lack of pre-built binaries for Intel Mac.

The version pace is also fast. The stable version is 0.6.0, but 0.7.0 RCs are coming out almost daily (while writing this article, it progressed from rc.4 to rc.6). In 0.7, the observability plugin configuration has been bumped from version 2 to 3, and it's explicitly stated that "the version 2 OTLP section format is rejected in version 3." Configuration files written today will not work as-is once 0.7 goes GA. It seems that the transition from 0.5 to 0.6 also required migrating 5 areas, so putting configurations under version control looks like something you'll need to do every month.

Finally, let me also touch on the reliability of the numbers in this article. I only validated on a single machine and with a few dozen turns' worth of sessions. I didn't run long sessions involving sub-agents or context compression, so there should be more span types than what I observed. I also wasn't able to validate the path through which providers return cost figures. Since Anthropic doesn't return them, this setup will always fall back to estimates from the pricing table.

Summary

I tried running NeMo Relay alongside Claude Code to see what could be observed.

Just by routing through a single local gateway, you can capture the agent's entire lifecycle. It was genuinely useful to have traces at the turn level with tool calls and LLM calls visible in a hierarchy. Being able to keep everything local without sending data to an external SaaS is also helpful when thinking about internal use.

On the other hand, you need to know in advance that what you can capture differs by output. Token counts don't come through from OpenTelemetry, and cache write costs are missing from the amount. Prompt text flows through by default. If you go in with the premise of combining OpenInference or ATIF with that in mind, I think it's well within practical use.

I only tried Claude Code this time, but the supported agents also include Codex CLI and Hermes Agent. Since a persistent installation is designed to share the same gateway, I'm curious about what it looks like when observing multiple agents together. Once 0.7 goes GA and configuration migration becomes necessary, I'll take another look at that point.

Reference Links

• NVIDIA/NeMo-Relay — Apache-2.0, Rust implementation
• Ecosystem — An overview of what it does and doesn't handle
• Support Matrix — List of supported agents and platforms
• Provider Response Codecs and Model Pricing — Exporter Field Mapping is here
• Model Pricing — Pricing catalog schema
• OpenTelemetry exporter — Configuration reference for the 0.6 series
• Release Notes — Breaking changes in 0.7 and Known Issues
• Prompt caching — Cache write unit price multiplier (Anthropic)