I summarized the NeMo Agent Toolkit as an introductory edition and a practical operation edition
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.
NVIDIA's NeMo Agent Toolkit (package name nvidia-nat, hereafter NAT) — many of you may be interested but feel unsure "where to start" or find it "hard to reproduce in your own environment since the official documentation assumes NIM cloud." In reality, while NAT's official documentation is very comprehensive as a feature catalog, it felt like there was a gap in Japanese resources that let first-timers follow the journey from "Hello Agent to production deployment" in one continuous flow.
I started writing to fill that gap, but it grew far beyond what a single blog post could contain, so I split it into two Zenn Books. The introductory volume is "a hands-on guide to exploring NAT from scratch without a GPU," and the practical operations volume is "an end-to-end guide to elevating a working agent to production-ready quality." This article serves as a reading guide for those two books, pulling out the "thinking" behind choices made within them — such as the reasoning behind the four pillars — into a form you can take away as a blog post.
A broader map of the NeMo Framework as a whole is covered in a separate article, NVIDIA NeMo Framework を俯瞰してみた, so referring to that alongside this article should make it easier to grasp where NAT sits within the overall ecosystem.
Scope Covered in the Two Books
First, let's get a bird's-eye view of the relationship between the two books in a single diagram. The idea is that both use the same NAT but are split into two volumes by layer.
The introductory volume is a foundational guide to assembling a "working agent" in a local environment, while the practical operations volume systematizes the "four pillars for improving operational quality" using the same NAT as its subject.
What I Wanted to Deliver with the Introductory NIM + Docker Hands-On
The title of the introductory volume is "NeMo Agent Toolkit Hands-On: Getting Started with NIM + Docker." The goal was to remove the very first barrier.
NAT's official tutorials are often written assuming a NIM API key, and when you try to run things locally, it's easy to fall into the trap of "spending half a day just on environment setup." The introductory volume is committed to a configuration that requires no GPU, making it possible to go all the way from Hello Agent to the final application using only Colima + Docker and the free tier of NIM on build.nvidia.com.
The concrete path starts with the shortest possible Hello Agent, then covers reading each section of workflow.yml, agent patterns bundling built-in tools, observability with Phoenix, MCP server integration, RAG with Milvus, multi-agent setups using Router and the A2A protocol, scoring with nat eval, Web API exposure via nat serve, and finally wrapping up the sample application. It's structured so you can complete one full pass through NAT's representative features in a natural order.
The book repository is at zenn-contents/books/nemo-agent-toolkit-nim-handson, and sample code is available under Apache 2.0 at nemo-agent-toolkit-book. Working samples are provided at each chapter checkpoint, so they can also serve as answer keys when you get stuck while following along.
If you want to run NAT with DGX Spark + vLLM for local inference, please also check out NeMo Agent Toolkit を DGX Spark + vLLM のローカル構成で動かしてみた as a separate entry point. The book is NIM free-tier based, while that article is fully local — a clear division of roles.
The Four Pillars of Operational Quality Covered in the Practical Operations Volume
The title of the practical operations volume is "NeMo Agent Toolkit Practical Operations — Guardrails × Langfuse." It's positioned as a book that fills the next step for readers who built a "working agent" in the introductory volume.
The sample application has been switched to an internal document Q&A agent. The reason for changing the subject is "to make it easier to experience the context of production deployment." When you start with the premise of handling confidential documents and samples containing PII, the necessity of guardrails and observability shifts from "something you wouldn't understand without explanation" to "something the subject matter demands." The sample is anonymized using Example Co., Ltd. (example.com) in accordance with RFC 2606, so you can run the book + sample repository as-is.
Here's the "four pillars of operational quality" covered in the practical operations volume in a single diagram.
Why LangGraph Was Chosen for Orchestration
The first pillar is Orchestration — how to design the agent's "behavior." NAT is built to incorporate a wide range of frameworks via optional extras, including LangChain / LangGraph / CrewAI / Semantic Kernel / Google ADK / Strands / AutoGen, making this an area where too many choices can actually lead to confusion.
In the practical operations volume, the implementation side is narrowed down to a single framework, LangGraph, with CrewAI / AutoGen treated as comparison tables and sidebars. There were two deciding factors: one is that NAT's _type: langgraph integration is cleanly structured, allowing behavior defined in LangGraph to be declared directly as a NAT function. The other is that for applications with clear state transitions like "retrieve → check → synthesize" — as in an internal Q&A system — expressing things as a state machine in LangGraph felt more natural than CrewAI's role-based model.
This doesn't eliminate use cases where CrewAI or AutoGen are the better choice — in role-based workflows with multiple roles (sales / development / reviewer), CrewAI's expressiveness can really shine. Chapter 3 of the practical operations volume lays out the selection criteria along the axis of "which framework suits which type of flow."
Why NemoGuard Safety Guard Multilingual v3 Was Chosen as the Guardrail LLM
The second pillar is Guardrails. NAT doesn't come with an official Guardrails middleware, so NeMo Guardrails is attached externally. The book covers both the pattern of calling LLMRails.check_async() manually and the pattern of wrapping with LangChain's RunnableRails.
What's quietly important here is the choice of Guardrail LLM. When applying safety judgments to Japanese input, Llama Guard 3 / 4 has no official Japanese support, and there's a risk of misses with its English-keyword-fixed parser. In the book, this problem is addressed by adopting NVIDIA's NemoGuard Safety Guard Multilingual v3 (nvidia/llama-3.1-nemotron-safety-guard-8b-v3). It uses a multilingual guarding approach called CultureGuard with cultural adaptation training for 9 languages including Japanese, achieving an average accuracy of 85.32% across those 9 languages (NVIDIA published figure). It can be integrated into NeMo Guardrails by adding just a few lines to the models block with engine: nim, and it works within build.nvidia.com's free NIM tier (40 RPM), making it quite an accessible first step.
The structure of NeMo Guardrails' five rail layers (Input / Output / Dialog / Retrieval / Execution) itself is also covered in a separate article, NeMo Guardrails で日本語 LLM に安全装備を載せてみた. Chapters 8-9 of the practical operations volume go one step deeper from there, exploring "integrating it into a NAT flow" and "deploying it for multilingual operations."
Why Phoenix Was Replaced with Langfuse Self-Hosted for Observability
The third pillar is Observability. The introductory volume uses Arize Phoenix as an entry-level tool, and its appeal was the ease of achieving the minimum bar of viewing traces with a single block of configuration.
However, with production operations in mind, you start wanting to consolidate the "surrounding areas of observability" — prompt management, cost tracking, evaluation datasets — somewhere. The reason the practical operations volume switches to Langfuse self-hosted was that it can cover all of this in a single stack. Langfuse v3 has an OTLP endpoint (/api/public/otel) and can receive traces directly via NAT's _type: langfuse exporter. Furthermore, prompt management (versioning + A/B comparison), cost and token tracking, and dataset management via the Datasets API are all connected seamlessly through a UI and API.
A migration guide from Phoenix to Langfuse is included in Appendix A, so readers familiar with Phoenix from the introductory volume should be able to pick up just the necessary differences and move on to the practical operations volume.
Why Eval Was Consolidated into Langfuse Datasets
The fourth pillar is Eval Dataset — how to incorporate evaluation into operations. NAT includes a scoring CLI called nat eval, which is also covered in Chapter 13 of the introductory volume.
In the practical operations volume, the management of evaluation datasets themselves is shifted to the Langfuse Datasets side. The reason is that in real operational settings, having evaluation results displayed alongside the traces, prompts, and costs consolidated in Observability on the same screen is genuinely useful. When you update a prompt, you can immediately re-evaluate it against the dataset and compare the diff with the previous version in the UI — a workflow that becomes naturally achievable.
This doesn't mean abandoning nat eval — it remains effective for batch scoring in CI and JSONL-based evaluation. Chapter 13 outlines the division of roles as: nat eval for CI batches, and Langfuse Datasets for operational feedback. For a broader map of NeMo's evaluation library ecosystem, also see LLM 評価基盤「NeMo Evaluator」を DGX Spark で試してみた.
The book repository is at zenn-contents/books/nemo-agent-toolkit-production-ops, and sample code is at nemo-agent-toolkit-production-ops. The Langfuse compose setup and Guardrails Colang configuration are all included, so feel free to use it as a scaffold when reproducing things locally while reading the book.
Criteria for Deciding Which Book to Read First
Since the question "which of the two books should I read first?" is likely to come up, here's a rough decision flow and comparison table.
Here's a table of the correspondence between the two.
| Axis | Introductory Volume - NIM + Docker Hands-On | Practical Operations Volume - Guardrails × Langfuse |
|---|---|---|
| Target audience | NAT beginners | Readers who completed the introductory volume / those considering production operations |
| Inference backend | NIM free tier | NIM free tier |
| GPU / Environment | Not required / Colima + Docker | Not required / Langfuse self-hosted with 6 self-managed services |
| Core technologies | YAML / ReAct / Milvus RAG / MCP / A2A | LangGraph / NeMo Guardrails / Langfuse / Datasets |
| Subject | Hello Agent → final application | Internal document Q&A |
For those new to NAT, starting with the introductory volume is the natural path. For those who already have experience roughly equivalent to the introductory volume and are curious about "how to incorporate observability / safety / evaluation into operations," jumping into the practical operations volume is the straightforward route.
Navigation Routes Between Existing DevIO Articles
Including those referenced so far, here's a single diagram of where this article sits among DevIO articles on NeMo-related topics.
The first entry point is NVIDIA NeMo Framework を俯瞰してみた, which covers brand organization for NeMo / Nemotron / NIM / Cosmos / NemoClaw and a DGX Spark compatibility matrix. The introduction to running Agent Toolkit on actual hardware is NeMo Agent Toolkit を DGX Spark + vLLM のローカル構成で動かしてみた, the Guardrails introduction is NeMo Guardrails で日本語 LLM に安全装備を載せてみた, and the evaluation side is LLM 評価基盤「NeMo Evaluator」を DGX Spark で試してみた. This article adds "an entry point for people who want to dive deeper chapter by chapter in Zenn Books" to that flow, with internal links structured so you can get the big picture from the articles and then dig in by following along in the books.
Conclusion
NeMo Agent Toolkit is less a single framework and more a foundation for layering observability, evaluation, and safety on top of multiple frameworks. While writing the practical operations volume, the division of labor — "write behavior in LangGraph, protect with Guardrails, observe with Langfuse" — became clear, and the meaning of choosing NeMo Agent Toolkit gradually came into focus.
Here are the books — please pick them up if you're interested.
📕 NeMo Agent Toolkit Hands-On: Getting Started with NIM + Docker
📗 NeMo Agent Toolkit Practical Operations — Guardrails × Langfuse
Sample code is on GitHub and corresponds to each chapter of the books.
Introductory Volume Sample Repository
Practical Operations Volume Sample Repository
