I tried running NeMo Agent Toolkit on a local configuration with DGX Spark + vLLM

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

NVIDIA NeMo Agent Toolkit (package name nvidia-nat, formerly AIQ Toolkit / Agent Intelligence Toolkit) is a meta-layer toolkit that wraps existing agent frameworks such as LangChain and CrewAI horizontally to observe, evaluate, and optimize them. It is designed with integration with NIM, NeMo Guardrails, and NemoClaw in mind.

https://github.com/NVIDIA/NeMo-Agent-Toolkit

Many official tutorials are written assuming NIM API keys, and Japanese introductory articles similarly focus on using cloud inference. This article documents the steps to complete agent execution, OpenTelemetry observation, and MCP server deployment "without cloud NIM API" using NGC vLLM container + Nemotron 3 Nano 30B-A3B-FP8 on DGX Spark as the backend.

NeMo Agent Toolkit

The first thing to understand is that NeMo Agent Toolkit is NOT "yet another agent framework." It incorporates LangChain, LangGraph, CrewAI, Semantic Kernel, Google ADK, and others via optional extras, and overlays a meta-layer for observation, evaluation, and optimization on top of them.

Developers basically only need to write workflow.yml, declaring what to attach to the 4 sections: functions / llms / embedders / workflow. It's a straightforward DSL that switches implementations via the _type field, so swapping from NIM to vLLM requires only a few line changes.

The official rebrand from the old name (Agent Intelligence Toolkit / AIQ Toolkit) took place in June 2025, with the core technology and roadmap carried forward. Backward-compatible packages aiqtoolkit / agentiq remain but are scheduled for future removal, so use nvidia-nat for new installations.

Key Features Overview

In the spirit of a first-steps article, here is a table of what's included.

Meta-layer use cases are envisioned, such as "delegate only the observation of an agent written in LangChain to NAT" or "optimize a CrewAI flow with evaluation-driven development."

Considerations for Running on DGX Spark

The official tutorial starts with an example calling model_name: meta/llama-3.1-70b-instruct via NIM API using _type: nim. This is the shortest path for verification, but when considering actual on-site use, there are situations where you want to complete everything with local inference.

From the NeMo Agent Toolkit side, whether it's NIM, vLLM, or Ollama, as long as it can be called as an OpenAI-compatible API, you can connect in the same way with _type: openai + base_url. The fact that switching backends is just one config line is also one reason to choose NAT as an agent infrastructure.

Verification Environment

Installation and Pitfalls

The nvidia-nat package itself is distributed as a pure Python wheel, so it installs on ARM64 without additional compilation. The smoothest approach is to create a venv with uv and then install.

mkdir -p ~/works/nemo-agent-toolkit && cd ~/works/nemo-agent-toolkit
uv venv --python 3.12
source .venv/bin/activate
uv pip install 'nvidia-nat[langchain,mcp]'
nat --version
# nat, version 1.6.0

Available extras include langchain / crewai / mcp / eval / opentelemetry / langsmith / phoenix and others. This time, since I'm aiming for "local LLM with ReAct + MCP + observation," I only specified langchain and mcp. Looking at nat info components after installation, nvidia-nat-eval and nvidia-nat-opentelemetry are also installed as part of dependency resolution. For introductory purposes, it seems fine to proceed without specifying additional extras.

Hello World with vLLM Local Inference

Starting the vLLM Container

The startup command this time is based on the DGX Spark Playbook procedure with one additional --trust-remote-code line. Nemotron 3 Nano uses the nemotron_h (Mamba and Transformer hybrid) architecture, which requires custom code bundled in the repository rather than a standard HF transformers implementation, so without this flag it crashes before the API Server starts with a pydantic ValidationError.

docker run -d \
  --name vllm-nat \
  --gpus all \
  --shm-size=16g \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  nvcr.io/nvidia/vllm:26.01-py3 \
  vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
    --trust-remote-code \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.85

Model loading took 30.5 GiB / approximately 4 minutes, torch.compile took approximately 2 minutes, after which the KV cache was allocated at 34.79 GiB, resulting in a configuration capable of handling up to 362x parallelism with an 8K context.

Verify it's running by hitting /v1/models.

curl -s http://localhost:8000/v1/models | python3 -m json.tool

Generating a Template with nat workflow create

NAT includes a nat workflow create command that generates project templates. This is the official flow to start from.

mkdir -p workflows
nat workflow create --no-install --workflow-dir ./workflows hello_local \
  --description "Hello world workflow"

The generated output takes the form of a Python package, with custom functions and registration code under src/hello_local/, and the NAT declarative workflow definition placed in configs/config.yml.

workflows/hello_local/
├── pyproject.toml
└── src/hello_local/
    ├── hello_local.py      # Custom tool implementation sample
    ├── register.py         # Registration to NAT Type Registry
    └── configs/config.yml  # Workflow definition

The default config.yml is configured to call Llama 3.1 70B Instruct assuming NIM.

functions:
  current_datetime:
    _type: current_datetime
  hello_local:
    _type: hello_local
    prefix: 'Hello:'

llms:
  nim_llm:
    _type: nim
    model_name: meta/llama-3.1-70b-instruct
    temperature: 0.0

workflow:
  _type: react_agent
  llm_name: nim_llm
  tool_names: [current_datetime, hello_local]

Rewriting for Local vLLM

Since I want to run this on DGX Spark's vLLM, I'll replace _type: nim in the llms section with _type: openai + base_url. To focus on Hello World as an introduction, I simplified the configuration to only use the built-in current_datetime without the custom tool (hello_local). I also replaced it with a workflow.yml placed directly under the project root for easier handling.

functions:
  current_datetime:
    _type: current_datetime

llms:
  local_vllm:
    _type: openai
    base_url: http://localhost:8000/v1
    api_key: EMPTY
    model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8
    temperature: 0.0

workflow:
  _type: react_agent
  llm_name: local_vllm
  tool_names: [current_datetime]
  verbose: true

Since the vLLM side doesn't require an api_key, a dummy string is fine. Omitting the field entirely will fail validation, so enter a string like EMPTY.

Running with nat run

nat run --config_file workflow.yml --input "今の日時を教えてください。日本語で回答してください。"

When executed, the ReAct Agent runs one cycle of think → tool call → final answer.

[AGENT]
Agent's thoughts:
Question: 今の日時を教えてください。日本語で回答してください。
Thought: I need to obtain the current date and time and then present it in Japanese.
Action: current_datetime
Action Input: None
------------------------------
Calling tools: current_datetime
Tool's response:
The current time of day is 2026-04-19 07:01:43 +0000
------------------------------
Agent's thoughts:
Thought: I now know the final answer
Final Answer: 現在の日時は 2026年4月19日 07:01:43（UTC）です。

The ReAct template is composed in English so the thinking process is in English, but when user instructions are in Japanese, the final answer is returned in Japanese. The inference latency per query was around 13 seconds for ReAct + 1 tool call, which was within an acceptable range for local inference.

Using nat serve, you can expose the same workflow.yml directly as a FastAPI server, so there's almost no rewriting needed when bridging "from command line to API" in an in-house PoC.

Observing Existing Frameworks (Phoenix Integration)

One of the benefits of NeMo Agent Toolkit is that you can add an OpenTelemetry-based observation layer with just one configuration block. Simply declaring an exporter under general.telemetry.tracing emits each step of the ReAct Agent, LLM calls, and tool executions as spans.

This time, as an easy option for browsing data, I'll set up Arize Phoenix locally.

docker run -d --name phoenix-nat \
  -p 6006:6006 -p 4317:4317 \
  arizephoenix/phoenix:latest

Using NAT's standard general-purpose OTLP exporter (_type: otelcollector), I point it to Phoenix's OTLP HTTP endpoint /v1/traces.

general:
  telemetry:
    tracing:
      phoenix:
        _type: otelcollector
        endpoint: http://localhost:6006/v1/traces
        project: nat-hello-local

functions:
  current_datetime:
    _type: current_datetime

llms:
  local_vllm:
    _type: openai
    base_url: http://localhost:8000/v1
    api_key: EMPTY
    model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8
    temperature: 0.0

workflow:
  _type: react_agent
  llm_name: local_vllm
  tool_names: [current_datetime]
  verbose: true

Running nat run again in this state and checking the /v1/projects/<id>/spans API on the Phoenix side, a CHAIN span named <workflow> was recorded, capturing the input text, final answer, and elapsed time.

{
  "name": "<workflow>",
  "span_kind": "CHAIN",
  "attributes": {
    "input.value": "What is today's date?",
    "output.value": "Today's date is 2026-04-19."
  },
  "status_code": "OK"
}

Accessing Phoenix's Web UI (http://localhost:6006), you can see the trace tree view, elapsed time for each span, and even the prompt/completion token counts for LLM calls.

Opening a single trace row, you can see all at once: the nested structure of the ReAct Agent (the outer <workflow> contains the inner <workflow>, which calls the current_datetime tool), input/output text, elapsed time, and the status of each span.

A sufficient setup for browsing agent behavior was assembled with just one Docker command and one config block.

One note: even when specifying project: nat-hello-local, traces are aggregated under the default project in Phoenix. NAT sends it as a service.name resource attribute, but Phoenix's project routing requires a different header, so setting this up properly becomes the next task for production use. At the introductory stage, being able to "see that it's working" is sufficient, so I'll defer further investigation to a separate article.

Publishing as an MCP Server

NeMo Agent Toolkit can publish your written workflow.yml directly as an MCP (Model Context Protocol) server. With the single nat mcp serve command, it becomes a tool that can be called from Claude Code, Claude Desktop, and other MCP clients.

nat mcp serve --config_file workflow.yml \
  --name "NAT Hello Local" \
  --host 0.0.0.0 \
  --port 9901

The default transport is streamable-http, with protocol version 2025-11-25. You can verify connectivity using the MCP client bundled with NAT.

nat mcp client tool list --url http://localhost:9901/mcp --direct
# current_datetime
# react_agent

Not only the built-in current_datetime, but the workflow itself (react_agent) is also automatically exposed as one tool. The argument schema can be inspected with the --detail option.

nat mcp client tool list --url http://localhost:9901/mcp --direct --detail
# Tool: react_agent
# Description: ReAct Agent Workflow
# Input Schema: { "properties": { "query": { "type": "string" } }, ... }

You can handle the entire ReAct loop as a single tool call by simply passing query. Let's try calling it directly.

nat mcp client tool call react_agent \
  --url http://localhost:9901/mcp --direct \
  --json-args '{"query": "What is the current time?"}'

The response returned is an OpenAI ChatCompletion-compatible object, with content: "2026-04-19 07:08:21 +0000" and token usage of prompt 5 / completion 3 / total 8.

At this point, a route becomes visible for calling "a custom agent running on DGX Spark" as an MCP tool from agents within NemoClaw's sandbox or from Claude Code / Claude Desktop. This seems like the lightest entry point for a hybrid configuration using a local LLM on the backend while fronting with Claude. I'd like to dig deeper into the actual MCP client-side configuration and how to expose multiple workflows from a single server in a separate opportunity.

What I Learned About Strengths and Weaknesses

Here is a summary of my impressions from exploring this as a first step.

The suitable use cases are fairly clear: situations where you already have an agent built with LangChain or CrewAI and want to "observe it," "improve it with evaluation-driven development," or "expose it as an MCP server." Even without writing any NAT-specific code, the ability to use OpenTelemetry, MCP, and A2A with just one configuration block makes it feel like an easy-to-integrate meta-layer design.

Conversely, if you just want to "run a single agent + one simple tool call," plain LangChain or directly using the OpenAI SDK is faster. NAT is the type of foundation that shows its true value only when you need to span multiple frameworks or standardize the observation and evaluation layer, so the practical starting point is to think about "how to overlay it on existing assets."

Summary

I walked through the first steps of NeMo Agent Toolkit in a purely local configuration with DGX Spark + NGC vLLM.

• nvidia-nat==1.6.0 installs smoothly as a pure Python wheel even on ARM64
• Using the NGC vLLM container from the DGX Spark Playbook as the backend, switching to local inference is just 2 lines: _type: openai + base_url
• Adding just one block of general.telemetry.tracing sends traces to Arize Phoenix
• A single nat mcp serve turns the workflow into an MCP server, transforming it into a tool callable from Claude Code and others

Having found that the combination of local LLM × agent infrastructure × observation × MCP comes together with less code than expected, please start by writing one workflow.yml on your local DGX Spark.

I'd also like to explore running evaluation-driven agent development with nat eval, and expanding to multi-agent setups with the A2A protocol.

Reference Links

• GitHub - NVIDIA/NeMo-Agent-Toolkit
• NeMo Agent Toolkit Documentation
• NeMo Agent Toolkit Developer Page
• NVIDIA Developer Forums - Rebrand Announcement
• NVIDIA Technical Blog (Japanese) - ReAct Agent with LangGraph + NeMo Agent Toolkit
• DGX Spark Playbooks - vLLM
• Arize Phoenix
• Model Context Protocol