I tried running Nemotron 3.5 Lightning with Hermes Agent and measured the effective cost

Introduction

Hello, I'm Shimada from Classmethod's Manufacturing Business Technology Department.

Today, NVIDIA released Nemotron 3.5 Lightning, and it has also been added to Fireworks AI.

https://app.fireworks.ai/models/fireworks/nemotron-lightning-3p5-30b-a3b

nemotron-lightning-3p5-30b-a3b is a model aimed at long-running agents, and what catches the eye is its price.
$0.05 per million input tokens, $0.20 for output.
It ranks among the cheapest in Fireworks' serverless tier.

I previously wrote an article about running the resident-type agent Hermes Agent with open models on Fireworks.

https://dev.classmethod.jp/articles/reona-hermes-agent-fireworks-slack/

At the end of that article, I noted that I had not measured costs.
The judgment to assign cheap models to routine morning processing should be evaluated after seeing actual consumption.
A model that advertises "cheapness" as its selling point has just appeared, so I decided to follow up on this homework.

Incidentally, two articles have already been published about the same model, focusing on local execution on DGX Spark.

https://dev.classmethod.jp/articles/dgx-spark-nemotron-3-5-lightning-first-touch/

https://dev.classmethod.jp/articles/nvidia-nemotron-3-5-lightning-dspark-dflash-mtp-dgx-spark/

This article examines whether it can be used as a resident model for an agent via the serverless API.
To state the conclusion upfront, low unit price is not the same as low effective cost.
Depending on the nature of the task, results ranged from being cheaper than the model I was already using to costing 9.7 times more.

Positioning of Nemotron 3.5 Lightning

Here is a summary of what the model page states.

• MoE with 3B active out of 32B parameters. NVIDIA Nemotron-H family
• Mamba-Transformer hybrid architecture with speculative decoding heads
• Generates a reasoning trace before returning the final response
• Context 262k, supports tool calling

The effect of speculative decoding is best covered in the preceding articles since it becomes the main topic for local environment measurements.
What matters from the perspective of using it via API is that it is a model that thinks before answering, and that the context is 262k.
Since many of Fireworks' major models have 1M, this is relatively modest.

Here are the unit prices. In USD per million tokens, Standard tier values.

Compared to deepseek-v4-flash-0731, which I have been using as the default model since the last article, input is 1/2.8 and output is 1/1.4.
I will compare these two models going forward.

Connecting It

First, I confirmed connectivity with the bare API.

There was one thing that caught my attention here.
The GET /v1/models response returns supports_chat: false for this model.

{
  "id": "accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b",
  "kind": "HF_BASE_MODEL",
  "supports_chat": false,
  "supports_tools": true,
  "context_length": 262144
}

Yet when you actually call chat/completions, you get a 200 response.

$ curl -s https://api.fireworks.ai/inference/v1/chat/completions \
    -H "Authorization: Bearer $FW" -H 'Content-Type: application/json' \
    -d '{"model":"accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b",
         "messages":[{"role":"user","content":"Reply with exactly: PONG"}],"max_tokens":256}'

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "PONG",
        "reasoning_content": "Here's a thinking process:\n\n1. **Analyze User Input**..."
      }
    }
  ],
  "usage": { "prompt_tokens": 22, "completion_tokens": 184 }
}

The response came back in 1.0 seconds.
If you automate model selection by trusting the catalog flags, this model will be excluded from candidates.
Specifying it explicitly works fine.

What I want to highlight is the usage field.
The content is just the 4 characters PONG, but completion_tokens shows 184.
The difference is the reasoning captured in reasoning_content.
In other words, reasoning tokens are billed as output tokens.
Tool calling also worked in the same format, returning finish_reason: tool_calls with a correct JSON.

On the Hermes side, I pass the provider and model separately, same as last time.
Since the Fireworks model ID itself contains slashes, the concatenated format does not work.

$ hermes -z 'Reply with exactly: PONG' \
    --provider fireworks -m accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b
PONG

Preparing to Measure Costs

Hermes has a global option called --usage-file that writes session usage to a JSON file.

$ hermes -z '...' --usage-file usage.json

{
  "estimated_cost_usd": 0.0,
  "cost_status": "unknown",
  "cost_source": "none",
  "input_tokens": 15067,
  "output_tokens": 93,
  "cache_read_tokens": 14992,
  "reasoning_tokens": 0,
  "api_calls": 2
}

The token counts were captured, but the cost is 0.
The cost_status shows unknown.

Upon investigation, I found that Hermes has a built-in price list for Fireworks.
agent/usage_pricing.py contains a dictionary keyed by ("fireworks", <model name>), with a pricing_version of fireworks-pricing-2026-07.

This means it is a snapshot as of July, so a model released today is not included.
The Fireworks entries (17 entries) in ~/.hermes/models_dev_cache.json did not contain Nemotron either.

At this point, I noticed that deepseek-v4-flash-0731, the comparison target, also shows unknown.
This is a model that has existed since before July.

The cause was the suffix at the end of the ID.
The key in the price list is deepseek-v4-flash, and deepseek-v4-flash-0731 is treated as something different.
Model name normalization is only applied to Anthropic and Bedrock; Fireworks uses exact match only.

When I ran it with the suffix removed, values appeared.

model = accounts/fireworks/models/deepseek-v4-flash
cost_status = estimated
cost_source = official_docs_snapshot
estimated_cost_usd = 0.00200718

In my previous article, I chose the version-pinned -0731 to make behavior easier to reproduce.
That choice came with the tradeoff of losing the cost display.

So I decided to calculate it myself.
Usage is also recorded in the sessions table of the session DB (~/.hermes/state.db), so I read from there.

PRICE = {
    "nemotron-lightning-3p5-30b-a3b": (0.05, 0.01, 0.20),
    "deepseek-v4-flash-0731": (0.14, 0.028, 0.28),
}

rows = con.execute("""
    SELECT id, model, api_call_count, input_tokens, cache_read_tokens, output_tokens
      FROM sessions WHERE started_at > strftime('%s','now') - ?
""", (7200,)).fetchall()

for sid, model, calls, inp, cache, outp in rows:
    pin, pcache, pout = PRICE[model.rsplit("/", 1)[-1]]
    cost = (inp * pin + cache * pcache + outp * pout) / 1_000_000

One point to note.
input_tokens represents only the input that did not hit the cache.
It is additive with cache_read_tokens, and the relationship total_tokens = input + cache_read + output holds.
You need to calculate both at their respective unit prices.

Note that reasoning_tokens was always 0.
This is because Fireworks does not report reasoning tokens separately; they are included in the output tokens.

Measurement 1: Single Tool Call

First, the same task as last time: counting .md files in a directory.

$ hermes -z 'Use your tools to count how many .md files are in the current directory. Reply with just the number.'
2

Both gave the correct answer on the first try with nearly identical consumption.
The difference in unit prices showed up directly, making Nemotron 1/2.6 of the cost.
All subsequent yen conversions use ¥150/USD.

Measurement 2: Routine Morning Job

Next, I tried a task closer to real-world operation.
Using the same prompt as the cron job I created last time, it curls 3 RSS feeds and summarizes them into 5 items in Japanese.

I ran each model twice.

Nemotron was 20–30% cheaper and finished faster.

What's interesting is that Nemotron's output tokens are 2–3 times higher.
This is to be expected for a model that thinks before answering.
How much it is thinking can be compared by looking at the character count of reasoning_content stored in the session DB.

It thinks about 3 times as much.
Yet the total cost still comes out lower because the difference in output unit price is small (0.20 vs. 0.28), while the difference in input unit price is large (0.05 vs. 0.14).
Agent consumption is dominated by the input side, where history is resent on every turn.
The disadvantage from reasoning was offset by the advantage in input unit price.

Measurement 3: Tasks Involving Exploration Reverse the Result

Here, the results flipped.

I gave the same research task I had requested from Slack in the previous article.
It was to look up NVIDIA-related articles posted on DevelopersIO today and answer with the titles and URLs in Japanese.
Since no search API key was configured, it would have to navigate on its own via browser operations or curl.

deepseek finished in 3 turns.
It listed 3 items from the RSS feed and even added a note saying: "The NeMo Switchyard article doesn't have NVIDIA in the title, but I included it as related — this may vary depending on criteria."

Nemotron took 22, 51, and 90 turns respectively.
These were 3 runs with the same prompt and same settings.
The cost ranged from 2.5 to 9.7 times that of deepseek.

The number of tool calls represents the difference more straightforwardly than the monetary amount.

The cause is the number of turns.
Each time an agent calls a tool, it resends the entire history up to that point.
When it wanders and the turn count grows, input tokens accumulate quadratically.

In the 51-turn run, ¥2.79 of the total ¥3.79 came from cached input.
The $0.01 per 1M unit price, which looks negligible in the table, became the dominant term.
The contribution of output tokens was only ¥0.53.

What a unit price table shows is only the price per token; the cost of an agent is determined by the number of turns.

Quality Issues Also Emerged

This was more concerning than the cost.
In the 51-turn run, Nemotron output a URL that does not exist.

- **Canonical URL:** `https://dev.classmethod.jp/guri/hajime-oguri/nvidia-nemotron-3-5-lightning-3-speculative-decoding-dsg-dflash-mtp-dgx-spark`
  (based on author slug structure)

It wrote itself that it was "estimated from the author slug structure."
On top of that, it duplicated the same URL across two entries.

Even though I instructed it to answer in Japanese, there were runs where the opening was in English.
For use cases involving unattended jobs that handle URLs, this cannot be ignored.

High Variance

The three runs took 22 / 51 / 90 turns.
The prompt and settings were identical; the only thing that changed was the time of execution.
In terms of cost, that's ¥1.44 / ¥3.79 / ¥5.61, meaning the price for the same job varies by a factor of 3.9.

The end_reason was normal completion in all cases — not termination due to hitting a limit.
In other words, it judged "done" by itself each time, yet the path it took to get there differed by a factor of 4.
For unattended jobs, you need to estimate based on the upper bound, not the average.

Restricting Tools Does Help with Speed

Using -t terminal,file to remove the browser tools improved things from 51 turns to 19 turns, and from 131.8 seconds to 35.2 seconds.
The instruction to answer in Japanese was also followed.

This flag does have an effect.
The difference shows up clearly when you have the agent list the tools it has available.

$ hermes -z '... list the exact names of every tool you can call' -t terminal,file
NO, patch, process, read_file, search_files, terminal, write_file

$ hermes -z '... list the exact names of every tool you can call'
YES
browser_back,browser_click,browser_console,browser_get_images,browser_navigate,
browser_press,browser_scroll,browser_snapshot,browser_type,clarify,delegate_task,
execute_code,memory,patch,process,read_file,search_files,session_search,
skill_manage,skill_view,skills_list,terminal,text_to_speech,todo,write_file

It was narrowed down from 25 to 7, with browser_* removed (the YES/NO at the beginning is the answer to the question "do you have browser tools?").

However, URL fabrication still occurred.
The turn count problem and the factual accuracy problem are separate issues; solving only the former does not fix the latter.

Running It in a cron Job in Real Operation

Since the results showed an advantage for routine tasks, I put it in a cron job to verify.
You can fix the model per job.

$ hermes cron create '0 6 * * *' "<prompt>" --name nvidia-daily-nemotron \
    --deliver slack:C0BPS3PR13K --provider fireworks \
    --model accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b
Created job: 3d8c095ef30e

I ran it manually and confirmed delivery to Slack.

INFO cron.scheduler: Job '3d8c095ef30e': delivered to slack:C0BPS3PR13K

Note that manual execution is also rejected when a job is in the hermes cron pause state.

Job is paused/disabled; resume it before running.

Paused jobs disappear from the hermes cron list listing.
Since they are invisible without the --all flag, be careful not to forget about test jobs you left paused.

Comparison with the existing job:

It completed in 32.3 seconds, showing the same trend as the CLI measurements.
At once a day, that's about ¥28 over 30 days.

However, there was an error in a proper noun in the output.
Where it should have written "Apollo, BlackRock, et al.", it wrote "Apache, BlackRock et al."
There was also a line with one English sentence mixed in.
The deepseek version of the same job had no such errors.

It may be cheaper, but you need to be skeptical about proper nouns.
For a news summary you skim through each morning, this may be acceptable, but it is not suitable for use cases where you quote the output directly.

Conclusion on How to Use Each

Based on the measurements this time, I decided to divide usage as follows.

• Routine cron jobs: Nemotron. For tasks where the procedure is fixed and the number of turns is predictable, it is 20–30% cheaper and faster.
• Dialogue and exploration: deepseek-v4-flash. For tasks where the number of turns is unpredictable, the cheaper unit price is negated by the increase in turns, and it ends up costing more.

To generalize, I think it can be put this way.
Delegating to a cheaper model is appropriate for tasks that leave little room for the agent to figure things out.
Tasks where the procedure can be fixed in the prompt, the tools can be narrowed down, and the output format is predetermined.
Conversely, for tasks where how to investigate is left up to the model, the ranking in the unit price table is meaningless.

Conclusion

I connected Nemotron 3.5 Lightning to Hermes Agent via Fireworks and measured the effective cost.

Here is a summary of what I found.

• The unit price is among the cheapest in the serverless tier. In actual measurements for routine tasks, it is also 20–30% cheaper and faster.
• For multi-step tasks involving exploration, the number of turns ballooned from 3 to 22–90, reversing the effective cost to up to 9.7 times higher.
• Even under the same conditions, the number of turns varied between 22 / 51 / 90. For unattended operation, you need to estimate based on the upper bound.
• Restricting tools reduces the number of turns, but does not improve factual accuracy.
• --reasoning is silently discarded on Fireworks. The amount of reasoning cannot be controlled.
• Errors in URLs and proper nouns occur. You need to be selective about how the output is used in unattended jobs.
• The price list built into Hermes is a snapshot as of July, so cost display does not work for new models. It also does not work with version-suffixed model IDs.

In the previous article, I wrote about setting up and running a resident agent.
This time, a cost estimate has been added to that.
With a figure of just under ¥1 per day now in view, it has become easier to make decisions about adding more jobs.

There are also things I have not verified.
I have not looked at the behavior when continuing a dialogue session via Slack for an extended period.
The frequency of fabrication cannot be described as a trend with only 3 exploration task runs.
Whether restricting reasoning effort changes the number of turns also cannot be measured without providing a way to pass reasoning_effort.
It seems worthwhile to verify how evaluations change after long-term operation.

References

• NVIDIA Nemotron 3.5 Lightning announcement blog
• Nemotron Lightning 3.5 30B A3B (Fireworks AI)
• Fireworks AI serverless pricing
• Hermes Agent official documentation
• NousResearch/hermes-agent