I tried NVIDIA's new agent framework NOOA with a local LLM on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

The article "Six Agent Harness Capabilities for Higher Model Performance" published by NVIDIA on July 27, 2026 has caught many people's attention. It covers how simply changing the design of the execution system (harness) surrounding an agent can significantly affect benchmark scores for the same model.

https://developer.nvidia.com/blog/six-agent-harness-capabilities-for-higher-model-performance/

Alongside the article, a framework called NVIDIA-labs OO Agents (abbreviated NOOA) was released on GitHub. It's an agent-building SDK for embedding into applications, where you write agents as Python classes, and if you put ... as a method body, the LLM fills it in at runtime — a quite bold design choice. It's now available on PyPI as v0.0.8, and can be installed with a single line: uv add nooa.

https://github.com/NVIDIA-NeMo/labs-OO-Agents

However, neither the paper nor the blog contains numbers measured on anything other than hosted API models. For those of us running local LLMs on a DGX Spark, the most pressing question is "will it work with the models I have on hand?"

This article covers an overview of what kind of framework NOOA is, along with results from running v0.0.8 on a local LLM on DGX Spark and reproducing some of the published capability tests. I hope it's useful to anyone who wants to run an agent framework locally.

What is NOOA?

Before getting into verification, let me organize what kind of tool NOOA is. NOOA is not an inference server like vLLM or Ollama — it's a client of those. It's also not an agent product you "launch and use" like Claude Code. It's an agent-building SDK that you import into your application, and in terms of layers, it sits in the same place as LangChain, LangGraph, Mastra, and Pydantic AI.

The agent-building SDK layer that sits between the application and model API. NOOA belongs to the same layer as LangChain / LangGraph / Mastra / Pydantic AI, connecting to inference servers via litellm in the lower layer.

The difference within the same layer lies in the direction of abstraction. In LangChain-based frameworks, adding a single tool requires defining and registering a schema, prompts are split into templates, and the flow of processing is assembled from framework-specific components like Chain, Graph, and Node. What you want to write is agent logic, but you tend to spend more time first learning the framework's vocabulary. What NOOA is trying to solve is exactly this — it goes all-in on eliminating framework vocabulary and reducing things back to plain Python. Agent definitions are your own classes, tool registration is unnecessary — just write a method, prompts are docstrings, and control flow is just plain if and for.

This design has a few close relatives. There's smolagents, which follows the CodeAct approach of having the model write code and using its execution to handle tool operations; Pydantic AI with its type contract approach; and Marvin, which has the LLM fill in function bodies. NOOA can be understood as bundling the elements of these three lineages into a single object model. The paper's self-assessment that "the 6 capabilities are not individually new inventions, but we are the first to integrate them on a single surface" is precisely a declaration of this positioning.

Another differentiating axis is that existing software engineering tools work as-is. If an agent is "just a class," you can write unit tests with pytest, and tracing and refactoring all fit into your usual workflow. Not introducing unique structures like Chain or Graph — I personally think this is where the practical benefit is actually the greatest.

One more thing worth touching on is the origin. NOOA is less a product and more a framework created to demonstrate the research claim that "harness design can significantly change a model's performance with the same model." What it's trying to prove will be organized together with the numbers after looking at the code.

Finally, let's also note its maturity. The license is Apache 2.0, supported Python is 3.12–3.13, and the development status on PyPI is Alpha. NVIDIA itself positions it as a research preview, so it's more accurate right now to think of it as an experimental platform where you can try out the harness design philosophy along with the code, rather than a production component.

Writing Agents as Python Objects

Now that we understand the positioning, let's look at the code. At the heart of NOOA is the single idea that "an agent is a Python object." The code placed at the beginning of the README is the design philosophy itself.

from nooa import Agent

class SupportAgent(Agent):
    """You are a support agent."""

    # State lives on the object. Fields are typed.
    order_db: OrderDB

    # Ordinary method. Just Python.
    def is_refund_eligible(self, order: Order) -> bool:
        return order.delivered and order.days_since_delivery <= 30

    # Agentic method: the runtime hands this to an LLM.
    async def triage(self, message: str, order: Order) -> Ticket:
        """Create a typed support ticket."""
        ...

The class is the agent, fields are state, docstrings are prompts, and type annotations are contracts. And methods whose body is ... are filled in by the LLM at runtime. "Filled in" doesn't mean a single text generation — an iterative loop runs where the model writes code, executes it, looks at the results, and decides the next move, known as an agentic loop. Methods with a body written out run as ordinary deterministic Python.

What's interesting is that the concept of "tools" doesn't exist. Everything attached to self is visible to the model, so adding a tool is the same as writing a method, and removing a tool is the same as removing a method. There's no schema definition, decorators, or registration process. The "reduction to plain Python" described in the previous chapter is implemented exactly as-is.

Another characteristic is that method names become part of the prompt. The examples note that renaming analyze_feedback to analyze_feedback_briefly changes the output. I actually verified this behavior in the second half of this article.

What Was It Built to Prove?

The interesting part of NOOA is not its features, but its motivation. The starting point is the same claim as the blog at the beginning: the "harness hypothesis" that "even without changing the model, changing the harness design can significantly affect performance." NOOA is an experimental platform prepared by NVIDIA to demonstrate this hypothesis, which is why capability tests and an evaluation harness are bundled in the repository.

https://arxiv.org/abs/2607.20709

So what design choices bring out the power of a model? The technical report organizes those conditions into 6 capabilities: typed input/output, pass-by-reference, code as action, Python loop control, explicit state on objects, and a harness API callable from the model. The "reduction to plain Python" seen in the previous chapter is the technical approach for satisfying all 6 in a single object model.

Why does plain Python bring out the model's power? The paper's explanation is straightforward: models have read massive amounts of Python during training. Reading object documentation, calling methods with typed arguments, using return values, and updating state — current-generation models can handle this sequence of operations without additional training, so inserting framework-specific vocabulary in between itself becomes interface friction. It's worth noting that in the comparison table with 14 frameworks, only NOOA has checkmarks for all 6 items, but that should be read as NVIDIA's own evaluation. The paper itself acknowledges that "the community is already converging on several of these items."

The most persuasive evidence for the hypothesis is the ARC-AGI-3 numbers. While ARC Prize's evaluation of vanilla GPT-5.6-sol on the same 25 games averaged 13.3%, placing the same model in NOOA's harness reportedly achieved 85.1%. The paper describes this as a 6.4x harness improvement. This is the direct basis for the claim that scores can change by double digits through harness design.

And what became the starting point for this article was Table 2 of the technical report — a stress test that extracts 6 of the capability tests that are "closest to real agent work," aggregated by model size.

NVIDIA itself is publishing a degradation curve where small models drop to 70.8%.

What's curious here is what "small" refers to. The 4 models are Claude Haiku 4.5, Gemini 3.5 Flash, Nemotron 3 Nano 30B, and GPT-5.4 Mini — all hosted API models. Not a single Qwen model is included. Neither degradation from quantization nor local inference server behavior has been measured.

In other words, where on this curve a quantized model running on DGX Spark falls, and whether the harness hypothesis holds locally, is not written anywhere. And the stress test code is bundled in the repository, so it can be run as-is. This is definitely worth trying out.

Getting It Running on DGX Spark

I used my DGX Spark for verification. It's a GB10 with 121GB unified memory, running Linux on aarch64.

For use as a library, installation from PyPI is a single line. Peripheral features like CLI and memory are split into subpackages and can be installed together with extras.

uv add nooa                   # Core only
uv add "nooa[cli]"            # + nooa command (trace viewer · eval runner)
uv add "nooa[cli,memory]"     # + long-term memory subsystem

However, examples and capability tests are not included in the wheel, so you'll need to clone the repository to reproduce them. Since the version is determined from git tags using uv-dynamic-versioning, checking out a tag lets you pin to the same point as the PyPI release.

git clone https://github.com/NVIDIA-NeMo/labs-OO-Agents.git nooa
cd nooa && git checkout v0.0.8
uv sync --group dev

Since the dependencies are pure Python centered on pydantic and litellm, there were no wheel issues on aarch64. Since requires-python is >=3.12,<3.14, uv automatically fetches Python 3.13 and creates a virtual environment.

I also ran the framework's own tests.

uv run pytest tests/ -q --ignore=tests/test_mcp

6348 passed, 5 skipped, 282 deselected, 21 warnings in 161.83s (0:02:41)

6348 tests pass on aarch64. This confirms that NOOA itself works on DGX Spark.

Next is connecting to a local LLM. The README includes connection examples for both Ollama and vLLM, and it's as simple as passing api_base to get_llm_client. The framework itself already supports local connections.

from nooa.unifiedllm.registry import get_llm_client

llm = get_llm_client("ollama_chat/qwen3.6:35b", api_base="http://localhost:11434")
llm = get_llm_client("hosted_vllm/qwen36-35b", api_base="http://localhost:8000/v1")

One thing I got stuck on here: the samples in examples/quickstart/ receive an LLM client from a helper called nooa.util.quickstart, but its internals check environment variables and branch between NVIDIA and OpenAI API keys, with no way to inject a local api_base. The examples can't be run locally as-is. I had no choice but to write a shim that only swaps out llm.

import os

from nooa.util.quickstart import *  # noqa: F401,F403
from nooa.unifiedllm.registry import get_llm_client

MODEL = os.environ.get("NOOA_LOCAL_MODEL", "ollama_chat/qwen3.6:35b")
API_BASE = os.environ.get("OLLAMA_API_BASE", "http://localhost:11434")

llm = get_llm_client(MODEL, api_base=API_BASE)

It just inherits everything with import * and then reassigns llm. If you redirect the examples' from nooa.util.quickstart import * to point to this module, they'll run as-is.

Note that this inconvenience has already been resolved upstream. In main, simply setting environment variables NEMO_OO_MODEL and NEMO_OO_API_BASE makes examples point to a local server, so the shim itself should be unnecessary from the next release. Please treat this as a temporary workaround when reproducing with the v0.0.8 PyPI release.

As a more proper approach, there's also a path to register aliases via YAML in the registry. If you place it in .nooa/llm_config.yaml or point to it with NEMO_OO_LLM_CONFIG, you can call models by name alone.

models:
  qwen3.6-35b-local:
    model_name: ollama_chat/qwen3.6:35b
    api_base: http://localhost:11434
    context_window: 262144
    drop_params: true

All 10 Quickstart Examples Worked with the 35B Model

First, I ran through the 10 examples in examples/quickstart/ one by one using qwen3.6:35b on Ollama. The time budget per example was set to 600 seconds.

All 10 worked. However, as noted in the table, the 3 examples with long agentic loops (04, 09, 10) didn't finish within the 600-second budget, and they completed on a re-run with the limit extended to 1800 seconds. Since how long a loop the agent chooses varies between runs, it's worth keeping in mind that if you set too tight a timeout with slow local inference, examples that should work will appear to fail.

Just having the process terminate normally isn't meaningful, so I verified calculations for examples with definite answers.

03_codeact_tools.py is an inventory-check agent. Apples cost $0.75 and have 50 in stock, bananas cost $0.50 and have 30 in stock, and oranges are out of stock. It asks whether these 3 items can be ordered on a $5 budget.

Can fulfill: False
Total cost: 1.25
Unavailable items: ['orange']

0.75 + 0.50 = 1.25, and since oranges are out of stock, the order cannot be fulfilled. All fields are correct. The model called self.get_stock() and self.get_price() inside CodeAct to perform the calculation.

What impressed me even more was 05_progressive_disclosure.py. Progressive disclosure is a design where rather than cramming all information into the prompt up front, the model is made to look things up when needed. In this sample, 4 different classes each have different evaluation methods, and there's no type description in the system prompt. The model needs to call doc(obj) at runtime to find out.

  ART-001 (Artwork): $15,000.00
  STK-001 (StockHolding): $87,550.00
  JWL-001 (Jewelry): $20,000.00
  COL-001 (Collectible): $4,250.00

All 4 were correct. The stock holding is 100 shares × $875.50, jewelry is 2.5 carats × $8,000, and the collectible is the base amount of $5,000 multiplied by a condition factor of 0.85. It discovered the different accessors for each type at runtime and called them correctly. It's genuinely interesting that this works with a local 35B model.

Stress Test Numbers Came Out, but Measuring Effects is Still Unstable

With the samples confirmed working, I next ran the same tests as the paper's Table 2 on local models. The backend was Qwen3.6-35B-A3B-FP8 on vLLM, with 5 runs per test. I narrowed the scope to 4 tests that don't require a judge. The remaining 2 use an LLM judge for scoring, and that judge references NVIDIA-internal model aliases that can't be resolved externally. I tested two conditions for temperature: 0 (prioritizing reproducibility) and the model default. The paper-side values are recalculated by extracting only those same 4 tests, so they can't be directly compared to the 70.8% and 93.9% in the body text.

First, there are 2 things I was able to confirm. The first is the numbers themselves — in both temperature conditions, the aggregate exceeded the paper's small API model group of 63.8%. A quantized local 35B lands above the small hosted models on this playing field. This is the first clue that the foundation of the harness hypothesis doesn't crumble locally either.

The second is the compatibility between the agent's parallel execution and the backend. In the 50-item batch classification sentiment_batch, the model wrote a method that classifies one item at a time, then fired off 50 items simultaneously with asyncio.gather. Counting with the OpenTelemetry traces that NOOA outputs by default, there were 51 LLM calls. Ollama with default settings couldn't handle this fan-out and timed out at 600 seconds, while vLLM launched with --max-num-seqs 64 absorbed it with continuous batching and completed in 35.7–129.0 seconds. This mechanism reproduced stably no matter how many times I changed conditions and ran it again. Ollama is GGUF and vLLM is FP8, so the quantization isn't matched, meaning this isn't a story about which backend is better — what the measurements show is that "NOOA's fan-out assumes a server with concurrent execution capacity matching the batch size." For a batch of 50 items, setting --max-num-seqs to 50 or higher is the reliable approach based on actual measurements.

classify took 602.56s with Result as None. The 51 generation spans lined up in the event panel on the left are the source of the saturation.

Even with the same 51 generation spans, classify completes in 35.74s and the Result returns 50 classification results.

On the other hand, effects beyond these couldn't be confirmed at this scale of measurement. For 3 of the 4 tests, temperature made no difference, and the one test where it did — refinement — showed temperature 0 at 60% versus default at 20%, which doesn't allow for a simple interpretation. In fact, even at temperature 0, results weren't the same every time: refinement went 3 wins and 2 losses across 5 runs under identical conditions. Since vLLM's continuous batching can change the order of numerical computation depending on which other requests are co-located, output varies even with greedy decoding. One example of a close failure: this single line of generated code caused a runtime error that crashed the run. Which run trips over it can't be controlled even with temperature 0.

qty = await self.check_availability({alt: 110}).get(alt, 0)

The case where renaming a method to analyze_feedback_briefly changes the output, and the case where increasing Ollama's parallelism caused the agent to abandon fan-out and switch to a single batch response (faster but incorrect), all follow the same pattern. There are runs where effects appear, but the differences visible in about 5 runs are adjacent to the margin of error, and there's no material to definitively assert whether the effects are real or not. Understanding that the paper averages multiple runs per test because this variability is a given was a key insight. For NOOA running locally, "it definitely works, but the effectiveness of tuning approaches remains unknown" is the accurate reading for now.

Common Sticking Points When Running Locally

Here's a summary of issues I ran into during verification. I hope this saves some time for anyone who gets stuck in the same places.

When running NOOA with vLLM, connecting to a plain vllm serve instance results in an immediate rejection.

BadRequestError: "auto" tool choice requires --enable-auto-tool-choice
and --tool-call-parser to be set

This happens because NOOA sends tool_choice="auto", so tool calling needs to be explicitly enabled on the server side. For the Qwen3 family, the hermes parser worked.

--enable-auto-tool-choice \
--tool-call-parser hermes

The api_base format differs between Ollama and vLLM. For Ollama, use http://localhost:11434 without a trailing /v1; for vLLM, use http://localhost:8000/v1 with it. Getting this wrong only produces a 404 page not found error, which makes the cause hard to track down.

When co-locating Ollama and vLLM on a unified memory machine, there's a trap in the startup order. vLLM measures available memory at startup before allocating it, but if the Ollama model unloads due to keep_alive expiration during that process, the freed memory gets interpreted as "interference from another process" and vLLM fails to start.

AssertionError: Error in memory profiling. Initial free memory 73.16 GiB,
current free memory 77.31 GiB. This happens when other processes sharing
the same container release GPU memory while vLLM is profiling

On the GB10, since CPU and GPU share memory, Ollama's load and unload operations show up directly in vLLM's observations. Running ollama stop <model> before launching vLLM to explicitly unload it provided stability. When testing a second Ollama instance with a different parallel count, you can launch it on a different port using OLLAMA_HOST.

For pytest, if ripgrep isn't installed, the 150 shell tool tests are skipped with "needs rg and grep on PATH." If your test count seems oddly low, check for the presence of rg. Also, since testpaths includes tests/test_mcp but the mcp extra isn't in the dev group, either add --ignore=tests/test_mcp when running or run uv sync --extra mcp beforehand.

The trace viewer launches on port 5001 with uv run nooa start-dev, and completed traces can be loaded in afterward with nooa import-traces <dir> --batch-id <id>. Being able to compare results organized by experiment was handy when running the same tests many times under different conditions, as I did here. One note: the URL format for individual traces is /traces/view?session_id=<id>. Also, the v0.0.8 viewer blocks non-loopback access with 403 if no token is set, so SSH port forwarding is the quickest way to view the viewer running on a remote machine in a local browser.

This is a bit off the main thread of the article, but the long-term memory subsystem alone can be tried without an LLM. examples/quickstart/12_memory.py runs with FakeLLMClient and offline embeddings, so no API key or local model is needed. The backend is SQLite with just 3 tables: memories, memory_edges, and maintenance_log. Contents are stored in plain text, so you can read them normally with sqlite3. One note: even if edges_added in the maintenance report is 0, rows may still be in the memory_edges table. The refines edges created when memories are inserted don't appear in the reflect step's counter — this is a difference in what's being counted, not a bug.

Finally, let me also quote a warning that the README itself writes. NOOA is a framework that executes code generated by an LLM, and while it has AST inspection and a module blocklist, the README explicitly states that these are "a layer of defense, not a containment boundary." It can access arbitrary files with open() and load directly from paths with importlib, so containment can't be guaranteed by Python static analysis alone. The README states that the containment boundary is OS-level isolation, and recommends running inside a container, VM, or NVIDIA OpenShell.

https://github.com/NVIDIA/OpenShell

What Is Upstream Building Right Now?

Development of main has been active even after the v0.0.8 release, with over 60 commits piling up as of 2026-08-18. Here are the items likely to affect the procedures in this article.

The biggest change is that the local support for examples — which I had to work around by writing a shim — has been merged into the main codebase. In addition, on the tracing side, a file-based journal format has been added. The main branch version of 06_tracing.py has been rewritten into a 4-step procedure that writes traces locally using a journal exporter and then loads them into the viewer afterward with nooa import-traces. Remote viewer publishing is also being developed, with token and session cookie authentication, and shareable URL generation now working in main. The parts that were blocked by 403 in v0.0.8 look likely to be resolved in the next release.

A security evaluation agent implementation called CyberGym has been added to examples. By mid-August, a portfolio-style agent and an evaluation report with per-model cost breakdowns were published, giving it enough volume to explore on its own. On the feature side, fixes continue for loading and reloading skills per agent, infrastructure for sharing sessions across multiple agents, and making the long-term memory's SQLite thread-safe. The CHANGELOG also records a security change: MCP server configuration no longer expands ${VAR} placeholders in environment variables. The release process itself has been scripted, so it looks like regular releases to PyPI will continue.

Summary

I ran NOOA v0.0.8 on a local LLM on DGX Spark. Starting with what's certain: all 10 quickstart examples worked on the 35B model, and I confirmed it discovering and calling different accessors per type at runtime. The design of "reducing agents to plain Python" works as advertised even against a locally quantized model. The 4 stress tests not requiring a judge aggregated to 85.0%, exceeding NVIDIA's published small API model group of 63.8% but falling short of the large model group at 91.7%. The compatibility between fan-out and backend concurrent execution count also reproduced stably as a mechanism, so knowing that this is a framework that fires 50 parallel requests at a batch of 50 items, setting --max-num-seqs to at least the batch size in vLLM is the reliable approach based on actual measurements.

On the other hand, the effects of "tuning approaches" like temperature, strategy selection, and method names had no stable sign even in 5-run measurements and remained unknown. Even with temperature 0, results change due to continuous batching non-determinism. There's simply not enough material yet to either confirm or deny the harness hypothesis locally — that's the honest situation, and it's important to read agent measurement values with the assumption that they fluctuate.

I'll also note what I couldn't do. The 2 tests requiring a judge depend on NVIDIA-internal model aliases and couldn't be run. The sample size is also only 5 runs per test, which isn't comparable to the paper's 10 models × 5 runs. The case where changing parallel count seemed to change the agent's strategy also wasn't traced to a root cause.

Next, I'd like to try the journal tracing and CyberGym example being developed in main, as well as the behavior inside OpenShell's sandbox. The ARC-AGI-3 world model solver is also bundled in the repository, so I'd like to revisit that as well.

Reference Links

• Six Agent Harness Capabilities for Higher Model Performance — The NVIDIA Technical Blog article that was the starting point for this article
• NVIDIA-labs OO Agents: Native Python Object-Oriented Agents — Technical report summarizing design principles and evaluation results
• NVIDIA-NeMo/labs-OO-Agents — The framework itself. Capability tests are also included
• nooa - PyPI — v0.0.8. nooa-cli / nooa-memory / nooa-bench can be added with extras
• NVIDIA OpenShell — The sandboxed execution environment recommended by the README
• LiteLLM Documentation — The model name format follows this specification