I tried summarizing NeMo Agent Toolkit as an introductory guide and a practical operations guide
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.
NVIDIA's NeMo Agent Toolkit (package name nvidia-nat, hereafter NAT) — many of you may be interested but feel unsure about "where to start" or find that "the official documentation assumes NIM cloud, making it hard to reproduce in your own environment." In fact, while NAT's official documentation is very comprehensive as a feature catalog, I felt there was a noticeable gap in Japanese-language resources that guide first-timers all the way from "Hello Agent to production deployment" in one continuous flow.
When I started writing to fill that gap, the content grew far beyond what could fit in a single blog post, so I organized it into two Zenn Books. The introductory volume is "a hands-on guide to exploring NAT from scratch without requiring a GPU," and the practical operations volume is "a comprehensive guide for elevating a working agent to production-ready status." This article serves as a reading guide for those two books, extracting the "thinking" aspects covered within them — such as the reasoning behind the four-pillar selection — into a form you can take away as a blog post.
A broader map of the NeMo Framework as a whole is covered in a separate article, A Bird's-Eye View of NVIDIA NeMo Framework, so referring to that alongside this content should help you grasp where NAT fits within the overall ecosystem.
Scope Covered in the Two Books
Let's start with a single diagram giving an overview of how the two books relate to each other. The idea is that both use NAT, but they're divided into two books by layer.
The introductory volume is a foundational volume for building a "working agent" in a local environment, while the practical operations volume systematizes "four pillars for improving operational quality" using the same NAT as the subject.
What the Introductory Volume's NIM + Docker Hands-On Aimed to Deliver
The title of the introductory volume is "NeMo Agent Toolkit Hands-On Starting with NIM + Docker." The goal was placed on removing the very first barrier.
NAT's official tutorials are often written assuming a NIM API key, and when you actually try to run things locally, you tend to get caught in the "spent half a day on environment setup" trap. The introductory volume is committed to a configuration that doesn't require a GPU, making it possible to go all the way from Hello Agent to the final application using only Colima + Docker and the free tier of NIM on build.nvidia.com.
The specific path you'll travel starts with the shortest Hello Agent, then progresses through reading each section of workflow.yml, agent patterns bundling built-in tools, observation with Phoenix, MCP server integration, RAG using Milvus, multi-agent setups with Router and A2A protocol, scoring with nat eval, Web API deployment with nat serve, and finally a polished version of the sample application. It's structured so you can make one complete pass through NAT's representative features in a natural order.
The book repository is at zenn-contents/books/nemo-agent-toolkit-nim-handson, and the sample code is available under Apache 2.0 at nemo-agent-toolkit-book. Since there are working samples for each checkpoint in each chapter of the book, I hope they can also serve as answer keys when you get stuck while copying the code.
For those who want to run NAT with DGX Spark + vLLM local inference, please also check out Running NeMo Agent Toolkit in a Local Configuration with DGX Spark + vLLM as a separate entry point from the books. The distinction is that the books are NIM free tier-based, while that article is fully local.
The Four Pillars of Operational Quality Covered in the Practical Operations Volume
The title of the practical operations volume is "NeMo Agent Toolkit Practical Operations — Guardrails × Langfuse." It's positioned as a book that fills the next step for readers who have built a "working agent" in the introductory volume.
The sample application has been switched to an internal document Q&A agent. The intent behind changing the subject is "to make it easier to experience the context of production deployment." When you have the premise of handling confidential documents and sample data containing PII, the necessity of guardrails and observability shifts from "something you won't understand unless explained" to "something the subject itself demands." The subject is anonymized as Example Corporation (example.com) in accordance with RFC 2606, so you can run the book + sample repository as-is.
Here are the "four pillars of operational quality" covered in the practical operations volume shown in a single diagram.
Why LangGraph Was Chosen for Orchestration
The first pillar is Orchestration — that is, how to design the agent's "behavior." NAT is built to incorporate a wide range of frameworks via optional extras, including LangChain / LangGraph / CrewAI / Semantic Kernel / Google ADK / Strands / AutoGen, making this an area where having too many options can actually cause more confusion.
In the practical operations volume, I narrowed the implementation side down to LangGraph alone, treating CrewAI / AutoGen with comparison tables + sidebar columns. There were two deciding factors: first, NAT's _type: langgraph integration is cleanly organized, allowing behavior built with LangGraph to be declared directly as a NAT function. Second, for applications like internal Q&A where there's a clear state transition such as "retrieve → check → synthesize," I felt that LangGraph, which can be written as a state machine, expresses this more naturally than CrewAI's role-assignment model.
This doesn't mean use cases for CrewAI or AutoGen disappear — in role-based workflows with multiple roles (sales / development / reviewer), there are situations where CrewAI's expressiveness really shines. Ch 3 of the practical operations volume lines up these selection criteria along the axis of "what kind of flow is each framework suited for."
Why NemoGuard Safety Guard Multilingual v3 Was Chosen as the Guardrail LLM
The second pillar is Guardrails. Since NAT doesn't have an official Guardrails middleware, you'll be attaching NeMo Guardrails externally. The book covers both the pattern of manually calling LLMRails.check_async() and the pattern of wrapping with LangChain's RunnableRails.
What's subtly important here is the selection of the Guardrail LLM. When performing safety judgments on Japanese-language input, Llama Guard 3/4 has no official Japanese language support, and there's a known issue where it tends to miss things due to an English-keyword-fixed parser. In the book, this problem is avoided by adopting NVIDIA's NemoGuard Safety Guard Multilingual v3 (nvidia/llama-3.1-nemotron-safety-guard-8b-v3). Using a multilingual guard method called CultureGuard, it has culturally adapted learning for 9 languages including Japanese, achieving an accuracy rate of 85.32% averaged across 9 languages (NVIDIA-published value). It can be integrated into NeMo Guardrails by adding just a few lines with engine: nim to the models block, and it can be tried with the build.nvidia.com NIM free tier (40 RPM), so I think it's a fairly approachable first step.
The structure of NeMo Guardrails' 5 rail layers (Input / Output / Dialog / Retrieval / Execution) itself is also covered in a separate article, Adding Safety Features to Japanese LLMs with NeMo Guardrails. Chapters 8-9 of the practical operations volume go one step deeper from there, focusing on the perspectives of "integrating into NAT's flow" and "applying to multilingual operations."
Why Langfuse Self-Hosted Replaced Phoenix for Observability
The third pillar is Observability. The introductory volume adopted Arize Phoenix for introductory purposes, and its appeal was the ease of achieving the minimum baseline of viewing traces with just one block of configuration.
However, looking ahead to production operations, you start wanting to consolidate "peripheral areas of observation" such as prompt management, cost tracking, and evaluation datasets into one place. The reason I switched to Langfuse self-hosted in the practical operations volume was that it could handle all of this with a single stack. Langfuse v3 has an OTLP endpoint (/api/public/otel), and traces can be sent directly with NAT's _type: langfuse exporter. Furthermore, prompt management (versioning + A/B comparison), cost and token tracking, and dataset management via the Datasets API are all connected in one flow through both the UI and API.
A migration guide from Phoenix to Langfuse is summarized in Appendix A, so readers who became familiar with Phoenix in the introductory volume should be able to pick up just the necessary differences and proceed to the practical operations volume.
Why Eval Was Shifted to Langfuse Datasets
The fourth pillar is Eval Dataset — that is, "how to incorporate evaluation into operations." NAT comes bundled with a scoring CLI called nat eval, which is also covered in Ch 13 of the introductory volume.
In the practical operations volume, I shifted the management of evaluation datasets itself to the Langfuse Datasets side. The reason is that I felt it would be effective in actual operations to have evaluation results visible alongside the traces / prompts / costs aggregated in Observability on the same screen. When you update a prompt, you can immediately re-evaluate against the dataset and compare the differences with the previous version in the UI — this enables a natural workflow cycle.
This doesn't mean abandoning nat eval — it remains effective for batch scoring in CI and JSONL-based evaluation. The division of roles, as written in Ch 13, is: nat eval for CI batches, Langfuse Datasets for operational feedback. For a broader map of NeMo's evaluation library ecosystem, see Trying Out the LLM Evaluation Infrastructure "NeMo Evaluator" on DGX Spark as well.
The book repository is at zenn-contents/books/nemo-agent-toolkit-production-ops, and the sample code is at nemo-agent-toolkit-production-ops. Since the complete Langfuse compose setup and Guardrails Colang configuration are included together, please use it as a scaffold for reproducing things locally while reading the book.
Decision Criteria for Which Book to Read First
Since the question "which of the two books should I read first?" is likely to come up, I'll provide a rough decision flow and comparison table.
Here's a side-by-side comparison of the contents.
| Axis | Introductory Volume - NIM + Docker Hands-On | Practical Operations Volume - Guardrails × Langfuse |
|---|---|---|
| Target audience | NAT beginners | Introductory volume completers / production operations planners |
| Inference backend | NIM free tier | NIM free tier |
| GPU / Environment | Not required / Colima + Docker | Not required / Langfuse self-hosted with 6 self-managed services |
| Core technologies | YAML / ReAct / Milvus RAG / MCP / A2A | LangGraph / NeMo Guardrails / Langfuse / Datasets |
| Subject | Hello Agent → Final application | Internal document Q&A |
For those new to NAT, the introductory volume is the natural starting point. For those who already have experience equivalent to the introductory volume level and are curious about "how to incorporate observation / safety / evaluation into operations," the practical operations volume is the direct route.
Cross-Reference Routes with Existing DevIO Articles
Including those referenced so far, here's a single diagram showing where this article sits among the DevIO articles around NeMo.
The first entry point is A Bird's-Eye View of NVIDIA NeMo Framework, where I summarize the brand organization of NeMo / Nemotron / NIM / Cosmos / NemoClaw and a DGX Spark compatibility matrix. The introduction to running Agent Toolkit on actual hardware is Running NeMo Agent Toolkit in a Local Configuration with DGX Spark + vLLM, the Guardrails introduction is Adding Safety Features to Japanese LLMs with NeMo Guardrails, and the evaluation system is Trying Out the LLM Evaluation Infrastructure "NeMo Evaluator" on DGX Spark. This article adds one more entry point in that flow — "for those who want to dive deeper chapter by chapter in a Zenn Book" — with internal links set up so you can grasp the overall picture from the articles and then work through the books while copying code.
Conclusion
NeMo Agent Toolkit is less a single framework and more of a foundation for overlaying layers of observation, evaluation, and safety on top of multiple frameworks. As I wrote the practical operations volume, the division of "writing behavior with LangGraph, protecting with Guardrails, watching with Langfuse" became clear, and the meaning of choosing NeMo Agent Toolkit gradually came into focus.
Here are the books — please feel free to pick them up.
📕 NeMo Agent Toolkit Hands-On Starting with NIM + Docker
📗 NeMo Agent Toolkit Practical Operations — Guardrails × Langfuse
The sample code is on GitHub, corresponding to each chapter of the books.
Introductory Volume Sample Repository
Practical Operations Volume Sample Repository
