I tried NVIDIA's new agent framework NOOA with a local LLM on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Many of you may have been curious about the article "Six Agent Harness Capabilities for Higher Model Performance" published by NVIDIA on July 27, 2026. The content explains that simply changing the design of the execution system (harness) surrounding an agent can significantly move benchmark scores for the same model.

https://developer.nvidia.com/blog/six-agent-harness-capabilities-for-higher-model-performance/

Simultaneously with the article, a framework called NVIDIA-labs OO Agents (abbreviated NOOA) was published on GitHub. It's an agent-building SDK that you embed into your application—you write agents as Python classes, and if you put ... as the method body, the LLM fills it in at runtime. This is quite a bold design. It is now available on PyPI as v0.0.8, and can be installed with a single line: uv add nooa.

https://github.com/NVIDIA-NeMo/labs-OO-Agents

However, neither the paper nor the blog contains any numbers measured against anything other than hosted API models. As someone running local LLMs on a DGX Spark, what I most want to know is "will this work with models I have on hand?"

In this article, I'll summarize what kind of framework NOOA is and share the results of running v0.0.8 on a local LLM on DGX Spark, reproducing some of the published capability tests. I hope this resonates with people who want to run agent frameworks locally.

What Is NOOA?

Before getting into the verification, let me organize what kind of tool NOOA is. NOOA is not an inference server like vLLM or Ollama—it's their client. It's also not an agent product you "launch and use" like Claude Code. It's an agent-building SDK that you import into your application, and as a layer, it sits in the same place as LangChain, LangGraph, Mastra, and Pydantic AI.

The agent-building SDK layer that sits between the application and the model API. NOOA belongs to the same layer as LangChain / LangGraph / Mastra / Pydantic AI, and connects to inference servers via litellm below.

The difference within the same layer lies in the direction of abstraction. In LangChain-type frameworks, every time you add a tool, you define and register a schema, split prompts into templates, and assemble the processing flow using framework-specific components like Chain, Graph, and Node. Even though what you want to write is the agent's logic, you tend to spend more time learning the framework's vocabulary first. What NOOA is trying to solve is exactly this—it goes all-in on eliminating the framework's vocabulary and reducing everything back to plain Python. Agent definitions are your own classes, tool registration is unnecessary since you just write methods, prompts are docstrings, and control flow is just plain if and for.

This design has a few close relatives. There's smolagents, which follows the CodeAct approach of having the model write code and using its execution to handle tool operations; Pydantic AI, which follows a type-contract approach; and Marvin, which has the LLM fill in function bodies. NOOA is best understood as bundling the elements of these three lineages into a single object model. The paper's self-assessment that "none of the 6 capabilities are new inventions, but we are the first to integrate them on a single surface" is precisely a declaration of this positioning.

Another axis of differentiation is that existing software engineering tools work as-is. If an agent is "just a class," you can write unit tests with pytest, and tracing and refactoring fit into your usual workflow. The practical benefit of not introducing proprietary structures like Chain or Graph is, I personally think, actually the biggest win here.

Let me also touch on its origins. NOOA was created not so much as a product but as a framework to demonstrate the research claim that "the same model's performance can change significantly depending on harness design." What it's trying to prove will be organized together with the numbers after looking at the code.

Finally, let me also note its maturity. The license is Apache 2.0, supported Python is 3.12–3.13, and the development status on PyPI is Alpha. Since NVIDIA itself positions this as a research preview, it's most accurate right now to think of it as an experimental ground where you can try out the entire way of thinking about harness design, rather than as a production component.

Writing Agents as Python Objects

Now that we understand the positioning, let's look at the code. At the heart of NOOA is the single point that "an agent is a Python object." The code placed at the beginning of the README directly embodies the design philosophy.

from nooa import Agent

class SupportAgent(Agent):
    """You are a support agent."""

    # State lives on the object. Fields are typed.
    order_db: OrderDB

    # Ordinary method. Just Python.
    def is_refund_eligible(self, order: Order) -> bool:
        return order.delivered and order.days_since_delivery <= 30

    # Agentic method: the runtime hands this to an LLM.
    async def triage(self, message: str, order: Order) -> Ticket:
        """Create a typed support ticket."""
        ...

The class is the agent, the fields are the state, the docstring is the prompt, and the type annotations are the contract. And methods with ... as their body are filled in by the LLM at runtime. "Filled in" doesn't mean a single text generation—an agentic loop runs where the model writes and executes code, observes the results, and decides the next move. Methods that have a body run as ordinary deterministic Python.

What's interesting is that the concept of "tools" doesn't exist. Everything on self is visible to the model, so adding a tool is the same as writing a method, and removing a tool is the same as deleting a method. No schema definitions, no decorators, no registration process. The "reduction to plain Python" described in the previous chapter materializes directly as the implementation.

Another characteristic is that the method name becomes part of the prompt. The examples note that renaming analyze_feedback to analyze_feedback_briefly changes the output. I actually tried to verify this behavior in the second half.

What Was It Built to Prove?

What makes NOOA interesting is its motivation more than its features. The starting point is the same claim as the introductory blog—the "harness hypothesis" that "even without changing the model, changing the harness design can significantly change performance." NOOA is the experimental ground NVIDIA prepared to demonstrate this hypothesis, which is also why the repository includes capability tests and an evaluation harness.

https://arxiv.org/abs/2607.20709

So what kind of design brings out the model's capabilities? The technical report organizes those conditions into 6 capabilities: typed input/output, pass-by-reference, code as action, Python for loop control, explicit state on objects, and a harness API callable by the model. The "reduction to plain Python" we saw in the previous chapter is the technical approach for satisfying all 6 together in a single object model.

Why does plain Python bring out capabilities? The paper's explanation is straightforward: models have read massive amounts of Python during training. Reading object documentation, calling methods with typed arguments, using return values, updating state—current-generation models can do all of this without additional training, so inserting framework-specific vocabulary in between itself becomes interface friction. It's worth noting that in the comparison table with 14 frameworks, only NOOA has checkmarks for all 6 items, but that should be read as an evaluation by NVIDIA itself. The paper itself acknowledges that "the community is already converging on several of these items."

The most convincing evidence for the hypothesis is the ARC-AGI-3 numbers. While ARC Prize's evaluation of vanilla GPT-5.6-sol on the same 25 games averaged 13.3%, putting the same model inside NOOA's harness reportedly reached 85.1%. The paper describes this as a 6.4x harness improvement. This is the direct basis for the claim that harness design can change scores by double digits.

And the starting point for this article was Table 2 of the technical report. It's a stress test that extracts 6 of the capability tests that are "closest to real agent work," aggregated by model scale.

NVIDIA itself is publishing a degradation curve showing small models drop to 70.8%.

What I find curious here is the contents of "small." It consists of 4 models: Claude Haiku 4.5, Gemini 3.5 Flash, Nemotron 3 Nano 30B, and GPT-5.4 Mini—all hosted API models. Not a single Qwen model is included. Degradation from quantization isn't measured, and neither is the behavior of local inference servers.

In other words, where a quantized model running on DGX Spark falls on this curve, and whether the harness hypothesis holds locally, is not written anywhere. And the stress test code is included in the repository, so you can run it as-is. There's no choice but to try it out.

Getting It Running on DGX Spark

I used my own DGX Spark for verification. It's a GB10 with 121GB unified memory, running Linux on aarch64.

To use it just as a library, installation from PyPI is a single line. Peripheral features like CLI and memory are split into subpackages, which can be installed together with extras.

uv add nooa                   # Core only
uv add "nooa[cli]"            # + nooa command (trace viewer · eval runner)
uv add "nooa[cli,memory]"     # + long-term memory subsystem

However, the examples and capability tests are not included in the wheel, so you need to clone the repository to reproduce them. Since the version is determined from git tags by uv-dynamic-versioning, checking out the tag lets you pin the same point as the PyPI version.

git clone https://github.com/NVIDIA-NeMo/labs-OO-Agents.git nooa
cd nooa && git checkout v0.0.8
uv sync --group dev

The dependencies are pure Python centered on pydantic and litellm, so there were no wheel issues on aarch64. Since requires-python is >=3.12,<3.14, uv automatically fetches Python 3.13 and creates a virtual environment.

I also ran the framework's own tests.

uv run pytest tests/ -q --ignore=tests/test_mcp

6348 passed, 5 skipped, 282 deselected, 21 warnings in 161.83s (0:02:41)

6348 tests pass on aarch64. This confirms that NOOA itself runs on DGX Spark.

Next is connecting to the local LLM. The README includes connection examples for both Ollama and vLLM, and it's as simple as passing api_base to get_llm_client. This means the framework itself already supports local use.

from nooa.unifiedllm.registry import get_llm_client

llm = get_llm_client("ollama_chat/qwen3.6:35b", api_base="http://localhost:11434")
llm = get_llm_client("hosted_vllm/qwen36-35b", api_base="http://localhost:8000/v1")

There was one snag here. The samples in examples/quickstart/ receive the LLM client from a helper called nooa.util.quickstart, but its internals branch by looking at environment variables to choose between NVIDIA or OpenAI API keys, with no way to inject a local api_base. This means you can't run the examples locally as-is. I had no choice but to write a shim that only swaps out llm.

import os

from nooa.util.quickstart import *  # noqa: F401,F403
from nooa.unifiedllm.registry import get_llm_client

MODEL = os.environ.get("NOOA_LOCAL_MODEL", "ollama_chat/qwen3.6:35b")
API_BASE = os.environ.get("OLLAMA_API_BASE", "http://localhost:11434")

llm = get_llm_client(MODEL, api_base=API_BASE)

It just inherits everything with import * and then reassigns llm. If you redirect the examples' from nooa.util.quickstart import * to this module, it works.

Note that this inconvenience has already been resolved upstream. In main, simply setting the environment variables NEMO_OO_MODEL and NEMO_OO_API_BASE will direct the examples to your local server, and the shim itself should become unnecessary from the next release. Please treat this as a temporary workaround when reproducing with the v0.0.8 PyPI version.

As a more proper method, there's also a route for registering aliases in YAML with the registry. Placing it in .nooa/llm_config.yaml or pointing to it with NEMO_OO_LLM_CONFIG allows you to refer to models by name alone.

models:
  qwen3.6-35b-local:
    model_name: ollama_chat/qwen3.6:35b
    api_base: http://localhost:11434
    context_window: 262144
    drop_params: true

All 10 Quickstart Scripts Ran Successfully on 35B

First, I ran through all 10 scripts in examples/quickstart/ one by one with qwen3.6:35b on Ollama. The time budget per script was set to 600 seconds.

All 10 ran successfully. However, as noted in the table, 3 scripts with long agentic loops (04, 09, 10) didn't fit within the 600-second budget and completed on a rerun with the limit extended to 1800 seconds. Since how long a loop the agent chooses varies per run, it's worth noting that cutting the timeout too short with slow local inference can make working samples appear to fail.

Just seeing the process exit normally isn't meaningful, so I verified answers for items with known correct outputs.

03_codeact_tools.py is an inventory check agent. 50 apples at $0.75, 30 bananas at $0.50, oranges out of stock. It asks whether these 3 items can be ordered within a $5 budget.

Can fulfill: False
Total cost: 1.25
Unavailable items: ['orange']

0.75 + 0.50 = 1.25, and oranges are out of stock so the order cannot be fulfilled. All items are correct. The model called self.get_stock() and self.get_price() inside CodeAct to perform the calculation.

I was even more impressed with 05_progressive_disclosure.py. Progressive disclosure is a design where instead of cramming all information into the prompt upfront, the model is made to look things up when needed. In this sample, 4 different classes each have different evaluation methods, and there's no type description in the system prompt. The model needs to call doc(obj) at runtime to find out.

  ART-001 (Artwork): $15,000.00
  STK-001 (StockHolding): $87,550.00
  JWL-001 (Jewelry): $20,000.00
  COL-001 (Collectible): $4,250.00

All 4 were correct. Stocks are 100 shares × $875.50, jewelry is 2.5 carats × $8,000, and collectibles are the base amount of $5,000 multiplied by a condition factor of 0.85. The model discovered and called the different accessors per type at runtime. It's genuinely interesting to see this work on a local 35B model.

Stress Test Numbers Came Out, But Effect Measurement Still Fluctuates

Now that I confirmed the samples work, I ran the same tests as the paper's Table 2 against local models. The backend was Qwen3.6-35B-A3B-FP8 on vLLM, with 5 runs per test. I narrowed the scope to the 4 tests that don't require a judge. The remaining 2 use an LLM judge in their configuration that points to a NVIDIA-internal model alias, which cannot be resolved externally. Two temperature conditions were used: 0 (prioritizing reproducibility) and the model default. The paper-side values are recalculated by extracting only the same 4 tests, so they cannot be directly compared to the 70.8% and 93.9% in the main text.

First, there are 2 things that were confirmed. The first is the numbers themselves—the aggregate for both temperature conditions exceeded the paper's small API model group at 63.8%. A quantized local 35B tops the small hosted model group on this benchmark. This is the first clue that the foundation of the harness hypothesis doesn't collapse locally.

The second is the compatibility between agent fan-out and backend concurrency. In the 50-item batch classification sentiment_batch, the model wrote a method to classify one item at a time and fired all 50 at once with asyncio.gather. Counting via the OpenTelemetry traces that NOOA outputs by default, there were 51 LLM calls. Ollama with default settings couldn't handle this fan-out and timed out at 600 seconds, while vLLM launched with --max-num-seqs 64 absorbed it with continuous batching and completed in 35.7–129.0 seconds. This reproduces stably as a mechanism regardless of how many times conditions are changed. Ollama is GGUF and vLLM is FP8, so the quantization isn't uniform—this isn't a story about backend superiority. What can be said from actual measurements is that "NOOA's fan-out assumes a server with concurrent capacity matching the batch size." For a batch of 50 items, it's safe to set --max-num-seqs to 50 or more.

classify times out at 602.56s and Result is None. The 51 generation spans lined up in the left event panel are the source of saturation.

Even with the same 51 generation spans, classify completes in 35.74s and the Result returns 50 classification results.

On the other hand, the effects beyond that could not be confirmed with measurements at this scale. Temperature had no effect in 3 of 4 tests, and the one test where it did—refinement—showed temperature 0 at 60% against default at 20%, which doesn't lend itself to straightforward interpretation. In fact, even with temperature 0, results were not the same every time, with refinement going 3 wins and 2 losses across 5 runs under identical conditions. vLLM's continuous batching can change the order of numerical computations depending on co-resident requests, so output can fluctuate even with greedy decoding. One example of a near-miss failure: this one line of generated code crashed the run with a runtime error. Which run hits it cannot be controlled even with temperature 0.

qty = await self.check_availability({alt: 110}).get(alt, 0)

The observation that renaming a method to analyze_feedback_briefly shortened the output, and the observation that increasing Ollama's concurrency caused the agent to abandon fan-out and switch to a bulk answer (faster but incorrect)—these share the same pattern. While there are runs where an effect seems apparent, differences visible across ~5 executions are adjacent to the margin of error, and I don't have material to definitively assert whether effects are present or not. Seeing that the paper also averages over multiple runs per test validates my understanding that this fluctuation is assumed. For NOOA running locally, reading it as "it definitely works; how well the effects can be leveraged is unknown" is currently the accurate framing.

Common Pitfalls When Running Locally

Let me summarize what I stumbled into during verification. I hope this saves some time for anyone who gets stuck in the same places.

When running NOOA with vLLM, connecting to a vanilla vllm serve immediately results in rejection.

BadRequestError: "auto" tool choice requires --enable-auto-tool-choice
and --tool-call-parser to be set

This is because NOOA sends tool_choice="auto", so you need to explicitly enable tool calling on the server side. For Qwen3-series, the hermes parser worked.

--enable-auto-tool-choice \
--tool-call-parser hermes

The api_base format differs between Ollama and vLLM. Ollama uses http://localhost:11434 without a trailing /v1, while vLLM uses http://localhost:8000/v1 with it. Getting this wrong only outputs 404 page not found, making it hard to trace the cause.

There's a trap in the startup order when co-locating Ollama and vLLM on a unified-memory machine. vLLM measures available memory at startup before allocating, but if the Ollama-side model unloads due to keep_alive expiration during that measurement, the increase in free memory gets flagged as "interference from another process" and vLLM fails to start.

AssertionError: Error in memory profiling. Initial free memory 73.16 GiB,
current free memory 77.31 GiB. This happens when other processes sharing
the same container release GPU memory while vLLM is profiling

On the GB10, since CPU and GPU share memory, Ollama's loading and unloading shows up directly in vLLM's observations. Explicitly unloading with ollama stop <model> before starting vLLM stabilized things. When trying a second Ollama instance with different concurrency settings, you can start it on a different port using OLLAMA_HOST.

On the pytest front, if ripgrep isn't installed in your environment, 150 shell tool-related tests will be skipped with "needs rg and grep on PATH." If the test count seems mysteriously low, suspect the absence of rg. Also, while testpaths includes tests/test_mcp, the mcp extra isn't in the dev group, so either add --ignore=tests/test_mcp when running, or do uv sync --extra mcp beforehand.

The trace viewer starts on port 5001 with uv run nooa start-dev, and completed traces can be loaded in afterwards with nooa import-traces <dir> --batch-id <id>. Being able to compare by experiment unit was very useful when running multiple times with varying conditions as I did. One note: individual trace URLs are in the format /traces/view?session_id=<id>. Also, the v0.0.8 viewer rejects access from non-loopback addresses with 403 when no token is set, so if you want to view the viewer running on a remote machine in your local browser, SSH port forwarding is the quickest approach.

This is off the main topic of the article, but the long-term memory subsystem alone can be tested without an LLM. examples/quickstart/12_memory.py runs with FakeLLMClient and offline embeddings, requiring neither API keys nor local models. The internals use SQLite with just 3 tables: memories, memory_edges, and maintenance_log. The contents are stored in plain text and can be read normally with sqlite3. One note: even if edges_added in the maintenance report is 0, rows may still exist in the memory_edges table. The refines edges created when memories are ingested don't count toward the reflect step's counter—this is a difference in accounting scope, not a bug.

Finally, let me also quote a caveat that the README itself writes. NOOA is a framework that executes code generated by an LLM, and while it has AST inspection and a module blocklist, the README explicitly states that these are "one layer of defense, not a containment boundary." It can touch arbitrary files with open() and load directly from paths with importlib, so containment cannot be guaranteed with Python-level static analysis. OS-level isolation is stated to be the containment boundary, and running inside a container, VM, or NVIDIA OpenShell is recommended.

https://github.com/NVIDIA/OpenShell

What Is Being Added Upstream Right Now

Development on main has been active even after the v0.0.8 release, with over 60 commits accumulated as of 2026-08-18. Let me highlight things that may affect the procedures in this article.

The biggest is that the local support fix for examples—which I worked around by writing a shim—has been merged into the main codebase. On the tracing side, a file-based journal format was added. The main branch's 06_tracing.py has been rewritten into a 4-step procedure that writes traces locally with a journal exporter and loads them into the viewer afterwards with nooa import-traces. Remote viewer access has also been improved, with token and session cookie authentication and shareable URL generation now working in main. The parts that were blocked by 403 in v0.0.8 look like they'll be resolved in the next release.

A CyberGym security evaluation agent implementation was added to examples. By mid-August, a portfolio-type agent and evaluation reports with per-model cost breakdowns were also published, giving you enough to play with independently. On the feature side, fixes allowing skills to be loaded and reloaded per agent, a foundation for sharing sessions across multiple agents, and a fix to make the long-term memory SQLite thread-safe have continued to land. The CHANGELOG also records a security change: MCP server configuration no longer expands ${VAR} placeholders in environment variables. The release process itself has been scripted as well, so it looks like regular releases to PyPI will continue.

Summary

I ran NOOA v0.0.8 on a local LLM on DGX Spark. Starting with what's certain: all 10 quickstart scripts ran successfully on a 35B model, and I confirmed it working all the way to discovering and calling different accessors per type at runtime. The design of "reducing agents to plain Python" functions as advertised even against a locally quantized model. The 4 judge-free stress tests aggregated to 85.0%, exceeding NVIDIA's published small API model group at 63.8% but not reaching the large model group at 91.7%. The compatibility between fan-out and backend concurrency reproduced stably as a mechanism, so knowing that this is a framework that fires 50-item batches as 50 concurrent requests, the safest approach in practice is to set --max-num-seqs to at least the batch size when using vLLM.

On the other hand, the effects of "levers" like temperature, strategy selection, and method names produced unstable signs even across ~5 runs, and remain unknown quantities. Even with temperature set to 0, results change due to non-determinism from continuous batching. I don't have enough material to either confirm or deny the harness hypothesis itself, and the honest state of affairs is that agent measurement values should be read with the assumption that they fluctuate.

Let me also note what I couldn't do. The 2 judge-requiring tests depend on NVIDIA-internal model aliases and couldn't be run. The sample count is also small at 5 runs per test, which is a different population from the paper's 10 models × 5 runs. And I couldn't identify the root cause of the observation that changing concurrency appeared to change agent strategy.

Next, I'd like to try the journal traces and CyberGym example being developed in main, and also the behavior inside the OpenShell sandbox. The ARC-AGI-3 world model solver is also included in the repository, so I'm thinking of taking a closer look at that as well.

Reference Links

• Six Agent Harness Capabilities for Higher Model Performance — The NVIDIA Technical Blog article that was the starting point for this article
• NVIDIA-labs OO Agents: Native Python Object-Oriented Agents — Technical report summarizing design principles and evaluation results
• NVIDIA-NeMo/labs-OO-Agents — The framework itself. Capability tests are also included
• nooa - PyPI — v0.0.8. Can add nooa-cli / nooa-memory / nooa-bench with extras
• NVIDIA OpenShell — Sandboxed execution environment recommended by the README
• LiteLLM Documentation — Model specification format follows this reference