I tried running Nemotron 3.5 Lightning with Hermes Agent and measured the effective cost

Introduction

Hello, I'm Shimada from the Classmethod Manufacturing Business Technology Department.

Today, NVIDIA released Nemotron 3.5 Lightning, and it has also been added to Fireworks AI.

https://app.fireworks.ai/models/fireworks/nemotron-lightning-3p5-30b-a3b

nemotron-lightning-3p5-30b-a3b is a model aimed at long-running agents, and what catches the eye is its pricing.
$0.05 per million input tokens, $0.20 per million output tokens.
It sits in the lowest-cost tier among Fireworks serverless models.

Previously, I wrote an article about running the resident-type agent Hermes Agent on Fireworks open models.

https://dev.classmethod.jp/articles/reona-hermes-agent-fireworks-slack/

In the conclusion of that article, I wrote that I hadn't measured costs.
The decision to assign a cheap model to routine morning processing should be evaluated after seeing actual consumption.
Since a model marketed on "low cost" has just appeared, I decided to settle this homework.

Regarding the same model, two articles have already been published from the angle of local execution on DGX Spark.

https://dev.classmethod.jp/articles/dgx-spark-nemotron-3-5-lightning-first-touch/

https://dev.classmethod.jp/articles/nvidia-nemotron-3-5-lightning-dspark-dflash-mtp-dgx-spark/

This article looks at whether it can be used as a resident model for an agent via the serverless API.
To state the conclusion upfront, low unit price is not the same as low effective cost.
Depending on the nature of the task, results ranged from being cheaper than the model already in use, to costing 9.7 times more.

Positioning of Nemotron 3.5 Lightning

Let me summarize what the model page says.

• MoE with 3B active out of 32B parameters. NVIDIA Nemotron-H family
• Mamba-Transformer hybrid architecture with speculative decoding heads
• Generates reasoning traces before returning the final response
• 262k context, tool call support

The effect of speculative decoding is best covered in local environment measurements, so I'll defer to the prior articles.
What matters from the perspective of using it via API is that it is a model that thinks before answering, and that the context is 262k.
Many of Fireworks' major models have 1M, so this falls on the modest side.

Here is a comparison of unit prices. Values are in USD per million tokens at the Standard tier.

Compared to deepseek-v4-flash-0731, which I've been using as the default model since last time, the input is 1/2.8 and the output is 1/1.4.
I'll be comparing these two models going forward.

Connecting It

First, I verified connectivity with the bare API.

One thing caught my attention here.
The GET /v1/models response returns supports_chat: false for this model.

{
  "id": "accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b",
  "kind": "HF_BASE_MODEL",
  "supports_chat": false,
  "supports_tools": true,
  "context_length": 262144
}

However, actually hitting chat/completions returns a 200.

$ curl -s https://api.fireworks.ai/inference/v1/chat/completions \
    -H "Authorization: Bearer $FW" -H 'Content-Type: application/json' \
    -d '{"model":"accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b",
         "messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":256}'

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "PONG",
        "reasoning_content": "Here's a thinking process:\n\n1. **Analyze User Input**..."
      }
    }
  ],
  "usage": { "prompt_tokens": 22, "completion_tokens": 184 }
}

The response came in 1.0 seconds.
If you're automating model selection based on catalog flags, this model will be left out of candidates.
You can use it by specifying it explicitly.

What I want to note is the usage.
The content is just the 4 characters PONG, but completion_tokens is 184.
The difference is the reasoning that went into reasoning_content.
In other words, reasoning tokens are billed as output tokens.
Tool calls also succeeded in the same format, returning finish_reason: tool_calls with valid JSON.

On the Hermes side, I pass the provider and model separately, same as last time.
Since the Fireworks model ID itself contains slashes, the concatenated format doesn't work.

$ hermes -z 'Reply with exactly: PONG' \
    --provider fireworks -m accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b
PONG

Preparing to Measure Costs

Hermes has a global option called --usage-file that writes session usage to a JSON file.

$ hermes -z '...' --usage-file usage.json

{
  "estimated_cost_usd": 0.0,
  "cost_status": "unknown",
  "cost_source": "none",
  "input_tokens": 15067,
  "output_tokens": 93,
  "cache_read_tokens": 14992,
  "reasoning_tokens": 0,
  "api_calls": 2
}

I got the token counts, but the cost is 0.
The cost_status is showing unknown.

Investigating, I found that Hermes has a built-in Fireworks price table.
agent/usage_pricing.py has a dictionary keyed by ("fireworks", <model name>), with pricing_version set to fireworks-pricing-2026-07.

This means it's a snapshot as of July, so the model released today isn't in it.
The Nemotron model was also absent from the Fireworks entries (17 entries) in ~/.hermes/models_dev_cache.json.

Here I noticed that deepseek-v4-flash-0731, which is my comparison target, also shows unknown.
This is a model that has existed since before July.

The cause was the suffix at the end of the ID.
The key in the price table is deepseek-v4-flash, and deepseek-v4-flash-0731 is treated as a different entry.
Model name normalization is only applied to Anthropic and Bedrock; Fireworks uses exact matching only.

When I removed the suffix and ran it, values appeared.

model = accounts/fireworks/models/deepseek-v4-flash
cost_status = estimated
cost_source = official_docs_snapshot
estimated_cost_usd = 0.00200718

In my previous article, I chose the version-pinned -0731 to make behavior reproducible.
That choice came with the tradeoff of losing cost display.

So I decided to calculate it myself.
Usage is also recorded in the session DB (~/.hermes/state.db) in the sessions table, so I read from there.

PRICE = {
    "nemotron-lightning-3p5-30b-a3b": (0.05, 0.01, 0.20),
    "deepseek-v4-flash-0731": (0.14, 0.028, 0.28),
}

rows = con.execute("""
    SELECT id, model, api_call_count, input_tokens, cache_read_tokens, output_tokens
      FROM sessions WHERE started_at > strftime('%s','now') - ?
""", (7200,)).fetchall()

for sid, model, calls, inp, cache, outp in rows:
    pin, pcache, pout = PRICE[model.rsplit("/", 1)[-1]]
    cost = (inp * pin + cache * pcache + outp * pout) / 1_000_000

One note of caution.
input_tokens represents only the input that did not hit the cache.
It is additive with cache_read_tokens, and the relationship total_tokens = input + cache_read + output holds.
You need to calculate both at their respective unit prices.

Note that reasoning_tokens was always 0.
This is because Fireworks does not report reasoning tokens separately; they are included in the output tokens.

Measurement 1: Single Tool Call

First, the same task as last time: counting .md files in a directory.

$ hermes -z 'Use your tools to count how many .md files are in the current directory. Reply with just the number.'
2

Both answered correctly on the first try with nearly identical consumption.
The unit price difference came through directly, with Nemotron costing 1/2.6 as much.
All yen conversions going forward use ¥150/USD.

Measurement 2: Daily Routine Job

Next, I tried a task closer to actual production use.
Using the same prompt as the cron job I created last time, it fetches 3 RSS feeds via curl and summarizes them into 5 items in Japanese.

I ran each model twice.

Nemotron was 20–30% cheaper and finished faster.

What's interesting is that Nemotron's output tokens are 2–3 times higher.
This is expected for a model that thinks before answering.
How much it's thinking can be compared by the character count of reasoning_content stored in the session DB.

It thinks about 3 times as much.
Yet the total cost is still lower, because while the output unit price difference is small (0.20 vs. 0.28), the input unit price difference is large (0.05 vs. 0.14).
In an agent's consumption, the input side—where history is resent every turn—tends to dominate.
The disadvantage in reasoning volume was offset by the advantage in input unit price.

Measurement 3: The Result Reverses for Exploratory Tasks

Here the results flipped.

I gave it the same research task I requested via Slack in the previous article.
It involves looking up NVIDIA-related articles posted today on DevelopersIO and answering with their titles and URLs in Japanese.
Since no search API key is configured, the model has to find them on its own using browser operations or curl.

Deepseek finished in 3 turns.
It cited 3 items from RSS feeds, and even added a note saying "The NeMo Switchyard article doesn't have NVIDIA in its title, but I included it as related—this may change depending on your criteria."

Nemotron spent 22 turns, and 51 turns in the next attempt.
The cost was 2.5 to 6.5 times that of deepseek.

The number of tool calls reflects the difference more directly than the monetary amount.

The cause is the number of turns.
Every time an agent calls a tool, it resends the entire history up to that point.
When the model wanders and the turn count increases, input tokens accumulate quadratically.

In the 51-turn run, ¥2.79 of the ¥3.79 total came from cached input.
The unit price of $0.01 per 1M tokens—which looks negligible in the table—became the dominant term.
The contribution of output tokens was only ¥0.53.

Unit price tables only show the cost per token; the cost of an agent is determined by the number of turns.

Quality Issues Also Emerged

More concerning than the cost was the following.
In the 51-turn run, Nemotron output a URL that does not exist.

- **Canonical URL:** `https://dev.classmethod.jp/guri/hajime-oguri/nvidia-nemotron-3-5-lightning-3-speculative-decoding-dsg-dflash-mtp-dgx-spark`
  (based on author slug structure)

It even wrote "estimated based on the author slug structure" itself.
Furthermore, it duplicated the same URL across 2 entries.

Even though the instructions said to answer in Japanese, some runs started in English.
For use cases involving URLs in unattended jobs, this cannot be ignored.

Reducing Reasoning Doesn't Help

Thinking that reducing reasoning might help if turn count is the problem, I tried --reasoning low.
The result was 90 turns, ¥5.61—the worst outcome.
The end_reason was a normal completion, so it wasn't cut off by a limit.

It seems that taking away thinking room from a model designed to think caused exploration to diverge.
The article that tested this model on DGX Spark also noted that it requires a sufficient token budget (around 16,000).
The same characteristic appeared when using it via API.

Restricting Tools Does Help Speed

Removing the browser tools with -t terminal,file improved things from 51 turns to 19, and from 131.8 seconds to 35.2 seconds.
The Japanese language instruction was also followed.

However, URL fabrication still occurred.
The turn count problem and the factuality problem are separate issues; solving only the former leaves the latter unresolved.

Running It in a cron Job in Production

Since routine tasks showed favorable results, I put it in cron to verify.
The model can be fixed per job.

$ hermes cron create '0 6 * * *' "<prompt>" --name nvidia-daily-nemotron \
    --deliver slack:C0BPS3PR13K --provider fireworks \
    --model accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b
Created job: 3d8c095ef30e

I ran it manually and confirmed delivery to Slack.

INFO cron.scheduler: Job '3d8c095ef30e': delivered to slack:C0BPS3PR13K

Note that manual execution is also rejected when a job is in the hermes cron pause state.

Job is paused/disabled; resume it before running.

Paused jobs disappear from the hermes cron list output.
Without --all, they become invisible entirely, so be careful not to forget test jobs left in a paused state.

Here is a comparison with the existing job.

It completed in 32.3 seconds, showing the same trend as the CLI measurements.
At once a day, that's about ¥28 over 30 days.

However, there was an error in a proper noun in the output.
Where it should have written "Apollo, BlackRock, et al.", it wrote "Apache, BlackRock, et al."
There was also a line with one English sentence mixed in.
The deepseek version of the same job had no such errors.

It's cheaper, but proper nouns need to be treated with skepticism.
For a morning news summary you skim through, this may be acceptable, but it's not suited for use cases where the output is quoted directly.

Conclusion on How to Divide Usage

Based on the measurements in this article, I've decided to divide usage as follows.

• Routine cron jobs go to Nemotron. For tasks with a fixed procedure and predictable turn count, it's 20–30% cheaper and faster.
• Conversation and exploration go to deepseek-v4-flash. For tasks with unpredictable turn counts, the low unit price is negated by the increase in turns, making it more expensive in the end.

To generalize, I think it can be said this way.
What's appropriate to delegate to a cheap model is a task with little room left for the agent to think.
Tasks where the procedure can be fixed in the prompt, the tools can be narrowed down, and the output format is predetermined.
Conversely, for tasks where how to investigate is left up to the model, the ranking in the unit price table is meaningless.

Conclusion

I connected Nemotron 3.5 Lightning to Hermes Agent via Fireworks and measured the effective cost.

Here is a summary of findings.

• Unit price is among the lowest in serverless. For routine tasks, actual measurements also showed 20–30% lower cost and faster execution.
• For multi-step exploratory tasks, the turn count ballooned from 3 to 22–51, and the effective cost reversed by up to 9.7 times.
• Reducing reasoning volume doesn't improve things. Restricting tools reduces turn count, but factuality does not improve.
• Errors appear in URLs and proper nouns. The use of output in unattended jobs needs to be chosen carefully.
• Hermes' built-in price table is a snapshot as of July, so cost display doesn't work for new models. It also doesn't work for model IDs with version suffixes.

In the previous article, I covered getting a resident agent up and running.
This time, a cost estimate has been added to that picture.
With a figure of about ¥1 per day in sight, it becomes easier to decide whether to add more jobs.

There are also things I haven't been able to verify.
I haven't looked at behavior when a Slack conversation session is kept running for a long time.
The frequency of fabrications also can't be called more than a tendency with just 2 exploratory task attempts.
It seems worth verifying how evaluation changes with long-term operation.

References

• NVIDIA Nemotron 3.5 Lightning Announcement Blog
• Nemotron Lightning 3.5 30B A3B (Fireworks AI)
• Fireworks AI Serverless Pricing
• Hermes Agent Official Documentation
• NousResearch/hermes-agent