I tried the LLM evaluation platform NeMo Evaluator on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Many people may be wondering what to use for LLM evaluation. MMLU alone can measure reasoning but not safety, and if you want to evaluate code generation, you'll want HumanEval. Benchmarks are scattered by purpose, each with different repositories and APIs. Honestly, having to look things up again every time you evaluate a model—"which one should I run this time?"—is a bit of a draining process.

That's where NVIDIA came in with an "we'll handle everything" approach, and what they produced is NeMo Evaluator. It's an evaluation platform that can call multiple harnesses from a single configuration file and evaluate them side by side under the same conditions.

However, "all-in-one" tools like this tend to become black boxes internally, and when trouble arises, it's easy to get lost thinking "I don't know where to start." This article focuses specifically on dissecting the internals of NeMo Evaluator to understand how it works. A measurement report with actual evaluations run across multiple models will be published in a separate article.

The verification environment is my DGX Spark (GB10, ARM64, 128GB unified memory). To state the conclusion upfront: NeMo Evaluator has a 2-layer structure of "core evaluation engine + launcher," and the major harnesses run as-is on DGX Spark's ARM64. The BYOB mechanism for adding custom benchmarks has also become considerably simpler in recent versions.

What is NeMo Evaluator

NeMo Evaluator is an OSS LLM evaluation platform published by NVIDIA. The repository is at NVIDIA-NeMo/Evaluator, and two packages are lined up as a monorepo.

One is nemo-evaluator, which serves as the core engine for evaluation. This is designed to run inside a container. The other is nemo-evaluator-launcher, commonly called nel, which is a launcher that acts as the orchestrator called from outside the container.

To use a cooking analogy for the division of roles: nel is the "head chef who reads the recipe," and the container is the "kitchen with all the knives and ingredients ready." The head chef doesn't pick up a knife themselves—they trust the kitchen and delegate the work.

In NVIDIA's positioning, NeMo Evaluator sits right between NeMo RL (post-training) and NeMo Agent Toolkit. It's the middle piece in the flow of evaluating a trained model and using the results for agent construction.

To organize who this tool is for, the following are situations where it should make your life easier:

• You want to run multiple benchmarks (MMLU, HumanEval, BFCL, safety-related, etc.) all at once
• You want to compare multiple models side by side under the same conditions
• You want to integrate your own custom benchmarks into the same platform as other benchmarks

The third point in particular—being able to integrate custom benchmarks into the same platform—seems like it would be effective in situations where you want to run evaluations on internal data in a reproducible way.

The Architecture is Built on "Containers × Pipelines"

NeMo Evaluator is divided into two layers: a launcher that orchestrates from the outside, and a container that actually runs the evaluations inside. The overall picture looks like this:

As you can see, nel is the orchestrator that pulls and launches the container, while the nemo-evaluator core inside the container is what actually calls the LLM endpoint and runs the evaluation.

What's surprising here is that even though they're packages sitting in the same repository, nel does not directly call the core's Python API. The nel side does not import and use functions from nemo-evaluator. It always launches a container and keeps everything self-contained within it.

This is a subtly thoughtful design, because even if you run different versions of nel and the core simultaneously, there's no worry about dependency conflicts. The Python environments on the evaluation job side, even if the required versions differ per harness, are separated by the container.

The Adapter / Interceptor Pipeline is Interesting

Looking inside the container's nemo-evaluator, another interesting mechanism appears. LLM API requests and responses are processed through a chain of components called Interceptors. Counting the implementations lined up in packages/nemo-evaluator/src/nemo_evaluator/adapters/interceptors/, there are 10 registered classes.

This is essentially the middleware concept from web frameworks applied almost directly to LLM evaluation. What makes it clever is that the cache layer is inside the pipeline, so once-evaluated results are not recalculated. Even if the process crashes midway, only the failed samples need to be re-evaluated—this is where it pays off in actual operation.

Incidentally, CachingInterceptor has two types listed because it serves a dual role: on a cache hit, it returns the response at the request stage as a shortcut, and on a cache miss, it saves the result at the response stage. The implementation uses multiple inheritance of RequestToResponseInterceptor and ResponseInterceptor.

The PostEvalHook type at the right end of the table means "an Interceptor that handles both response processing and post-evaluation hooks." The three that qualify—ProgressTracking / ResponseReasoning / ResponseStats—handle both metrics collection during inference and summary generation after evaluation completion within the same class. It's an efficient design.

The presence of ResponseReasoningInterceptor is also quite considerate. It separates the <think>...</think> content returned by models like Nemotron or DeepSeek-R1 from the main body, so you don't need to write extra preprocessing on the scoring side.

The Launcher's Toolset

On the nel side, there's a full set of components for job management, not just container launching.

The Executor, which switches execution environments, supports three types: local execution, Slurm cluster, and Lepton. For personal DGX Spark use, Local is the norm, but it's nice that organizations with Slurm on their internal clusters can scale up directly.

The Exporter, which selects where results are sent, supports MLflow, Weights & Biases, Google Sheets, and local files. If you're running many jobs and want to compare them across the board, accumulating them in MLflow from the start makes things easier later.

Execution history is recorded in a SQLite ExecutionDB, allowing you to retrieve past jobs by invocation ID. This history is useful when you want to re-evaluate or find the best checkpoint.

And the component that resolves container images from task names is mapping.toml. It's a central registry that says "this evaluation runs on this container for this harness," and nel refers to it at startup to pull the container.

In addition, there's a standalone command called nel-watch that links training and evaluation. This is an operations topic, so I'll cover it together toward the end of the article.

23 Integrated Harnesses and 421 Tasks

The evaluation harnesses covered by NeMo Evaluator total 23 types, amounting to 421 tasks (as of nel v0.2.4). Let's first take an overview by category.

Starting from the two major general LLM harnesses lm-evaluation-harness and simple-evals, and extending to code, safety, agent, multilingual, VLM, and long context—the overall impression is that a comprehensive set of evaluation axes is covered.

Counting 421 Tasks on Real Hardware

With nel installed on DGX Spark, running ls tasks lists all available tasks.

$ nemo-evaluator-launcher ls tasks
[I 2026-04-18] Loaded external tasks from IRs total_tasks=421
[I 2026-04-18] Using merged IRs total_tasks=421 internal_tasks=0 external_tasks=421
...

Aggregating task counts by harness yields the following distribution:

Confirming the same distribution in numbers:

The Distribution Reveals Imbalances

Looking at this distribution, a few things stand out.

First, lm-eval and simple_evals alone account for 221 tasks—more than half of the total is covered by just these two major harnesses. MMLU, GSM8K, HumanEval, and the like that commonly appear in papers mostly come through one of these two.

Another thing that catches the eye is the 27 tasks in the codec harness. This is a set for contamination detection—detecting whether benchmark data has leaked into a model's training data—serving the role of ensuring the health of evaluation scores. The fact that NVIDIA is allocating significant space to this area is likely because they need to answer the question "Is that score actually due to data leakage?" when evaluating models like Nemotron and others.

There are also about 18 tasks with the adlr_ prefix. ADLR stands for NVIDIA Applied Deep Learning Research, and this is a set that allows you to reproduce the exact evaluation conditions used in the Nemotron paper. It seems like it would be very useful when you want to align benchmark results with the paper.

The multilingual coverage is also rich, with global_mmlu_full_{ja, ko, ar, ...} alone covering over 40 languages. It's straightforwardly usable for cross-model evaluation of Japanese language models as well.

On the other hand, vlmevalkit (VLM evaluation) is modest at 7 tasks, limited to things like AI2D / ChartQA / OCRBench / SlideVQA. This area still seems to have room to grow.

NVIDIA's Strategy of Not Reinventing the Wheel

When you hear "managing 23 harnesses all together," you might momentarily think NVIDIA rewrote everything from scratch. But that's not actually the case—they're using existing OSS harnesses almost as-is.

Looking inside the container nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 that nel pulls, the contents are a packaged version of EleutherAI's lm-evaluation-harness—NVIDIA hasn't rewritten it independently.

What NVIDIA has written themselves is only the adapter layer wrapping the outside of the containers, the 10-stage interceptors, and the mapping.toml and the launcher-side CLI. The clear separation between "the layer that trusts existing OSS" and "the layer that needs to be built in-house" is an interesting design from a maintainability standpoint.

Does It Actually Work on DGX Spark (ARM64)?

This is where it gets most interesting for DGX Spark users. Do the NGC containers distributed by NeMo Evaluator actually work on ARM64?

The official documentation has no explicit mention of ARM64 support, and there's a precedent where ARM64 NV-Ingest for DGX Spark was not provided, so I checked with docker manifest inspect with a bit of concern.

Looking Directly at the Manifests

Let's start by checking the lm-evaluation-harness container.

$ docker manifest inspect nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "platform": { "architecture": "amd64", "os": "linux" }
    },
    {
      "platform": { "architecture": "arm64", "os": "linux" }
    }
  ]
}

Both amd64 and arm64 are listed in the manifest list. When docker pull is run on an ARM64 host (DGX Spark), Docker automatically fetches the arm64 image.

Actual Results for 5 Major Harnesses

Repeating the same check for major harnesses and summarizing in a table:

Only vlmevalkit is distributed as amd64-only; all other major harnesses had ARM64 manifests available.

What This Means

To summarize these results: even on DGX Spark (Grace CPU = ARM64), standard harnesses work with just docker pull. Using nel's local execution mode, you can run evaluations without any additional cross-compilation.

On the other hand, for VLM evaluation, since vlmevalkit doesn't support ARM64, the options diverge. Either use a self-built image cross-compiled with BYOB's --platform linux/amd64 option, or run nel on an x86 server and point the API calls to the LLM on DGX Spark.

Either way, there are escape routes available. I'll cover the BYOB cross-build process in the next chapter.

Adding Custom Benchmarks with BYOB in a Minimal Configuration

A common situation when using LLM evaluation platforms is "the existing harnesses aren't enough." You might want to evaluate a QA set built from company data, or run a custom benchmark designed for internal documents. The feature that addresses these needs is NeMo Evaluator's BYOB (Bring Your Own Benchmark).

Old Method vs. New Method

BYOB actually has two ways to write it. Both remain due to historical reasons, and the writing experience differs quite a bit depending on which you use.

The new method arrived with v0.2.5, and the writing experience is considerably better, so it's recommended. This article proceeds with the new method as well.

Writing It in 16 Lines of Python

Let's write a simple benchmark that uses the SQuAD dataset and scores with exact match.

from nemo_evaluator.contrib.byob import benchmark, scorer, ScorerInput

@benchmark(
    name="my-qa-bench",
    dataset="hf://rajpurkar/squad?split=validation",
    prompt="Context: {context}\nQuestion: {question}\nAnswer:",
    target_field="answers",
    endpoint_type="chat",
    requirements=["datasets"],
)
@scorer
def my_qa_scorer(sample: ScorerInput) -> dict:
    predicted = sample.response.strip().lower()
    gold = sample.target["text"][0].lower() if sample.target["text"] else ""
    return {"exact_match": predicted == gold}

Just this—16 lines without comments. The @benchmark decorator handles dataset loading and prompt formatting, and the @scorer function becomes the scoring logic.

The Build and Execution Flow

The overall flow from writing the benchmark to making it callable from nel looks like this:

To make the written file callable from nel, use the nemo-evaluator-byob CLI. Start with a syntax check, then proceed to local installation and container build.

# Syntax check only (Docker not required)
$ nemo-evaluator-byob my_benchmark.py --dry-run
Validation passed. Benchmarks found:
  - my-qa-bench (normalized: my_qa_bench)
    Dataset: hf://rajpurkar/squad?split=validation
    Requirements: datasets

# Install to local environment
$ nemo-evaluator-byob my_benchmark.py
  Benchmark: my-qa-bench
  Package:   byob_my_qa_bench
  Location:  /home/morishige/.nemo-evaluator/byob_packages/byob_my_qa_bench
  Installed: byob_my_qa_bench (discoverable by nemo-evaluator)
Compiled 1 benchmark(s) successfully.

# Check the eval_type name
$ nemo-evaluator-byob --list
Installed BYOB benchmarks (/home/morishige/.nemo-evaluator/byob_packages/):
  byob_my_qa_bench.my_qa_bench

You can confirm that the bench name my-qa-bench is normalized from hyphens to underscores, and an eval_type is generated in the two-level structure byob_<name>.<name>. This byob_my_qa_bench.my_qa_bench becomes the value for the name field written in nel's evaluation configuration YAML.

Trying the Container Build

Let's build a container on DGX Spark (ARM64).

$ nemo-evaluator-byob my_benchmark.py \
    --containerize \
    --tag byob_qa:local-test
...
[I ...] Downloading HuggingFace dataset dataset=rajpurkar/squad split=validation
[I ...] Converted HuggingFace dataset to JSONL records=10570
[I ...] Building Docker image tag=byob_qa:local-test-linux-aarch64 ...
[I ...] Docker image built successfully tag=byob_qa:local-test-linux-aarch64
Docker image built: byob_qa:local-test

A 530MB ARM64 image was completed in about 36 seconds. What's nice is that during the build, it automatically fetches the SQuAD dataset from HuggingFace, converts it to JSONL (10,570 records), and bundles it into the container. Since the image becomes self-contained, evaluations can be run without external network access even when taken to a different environment later.

Another small finding: the --tag value automatically gets a platform identifier appended. For the specified byob_qa:local-test, the actual tag became byob_qa:local-test-linux-aarch64. The design intent of storing separate images per platform in the registry is apparent.

Cross-Building for amd64

The --platform linux/amd64 flag comes into play when you want to use vlmevalkit (which doesn't support ARM64) on DGX Spark (added in v0.2.5's PR #832). An image for a different platform is generated via Docker buildx.

nemo-evaluator-byob my_benchmark.py \
  --containerize \
  --platform linux/amd64 \
  --tag byob_qa:cross-amd64

Building an amd64 image from an ARM64 host goes through qemu emulation, so it takes considerably longer than native (36 seconds). In the author's environment, it was still in progress after exceeding 2 minutes, so for practical use, either build on the CI side (x86 Linux) or be prepared to wait.

This cross-build is most useful on DGX Spark for addressing the problem mentioned in the previous chapter—vlmevalkit not supporting ARM64. Having a self-built image cross-compiled with --platform linux/amd64 lets you avoid the dead end of "can't do VLM evaluation." Of course, it's also useful in the straightforward use case of building custom benchmarks, evaluating Nemotron or Gemma on internal data, and accumulating results in MLflow.

Integration with nel via YAML

When calling a built container from nel, you don't need to touch mapping.toml. With the new method, you just write the container URL directly in the container field of the evaluation configuration YAML.

evaluation:
  tasks:
    - name: byob_my_qa_bench.my_qa_bench
      container: registry.example.com/byob_qa:latest

deployment:
  target:
    api_endpoint:
      url: http://localhost:8000
      model_id: my-model
      type: chat

Managing mapping.toml was unexpectedly cumbersome with the old method, but the new method lets you consolidate everything into the evaluation configuration YAML, which is much cleaner.

Potential Pitfalls

Here are some things that are likely to trip you up when actually running BYOB.

First, a somewhat unexpected stumbling block: --containerize won't work unless nemo-evaluator is installed as an editable install from the source tree. In a wheel environment installed with pip install nemo-evaluator, the COPY nemo_eval_pkg/ ... step will produce a not found error during build. The reason is that nemo-evaluator itself needs to be baked into the container, so it looks for the source directory at build time. This can be avoided with the following steps:

git clone --depth 1 https://github.com/NVIDIA-NeMo/Evaluator.git
uv pip install -e ./Evaluator/packages/nemo-evaluator

With this, nemo-evaluator-byob --containerize will work as-is.

There's also one import pitfall: the module name is nemo_evaluator.contrib.byob, not directly under the nemo_evaluator top level. Writing from nemo_evaluator import benchmark will result in an import error that gets rejected even at the dry-run stage.

Next, if the {field} names in the prompt don't match the dataset column names, you'll get a KeyError at runtime. When column names differ, either use the field_mapping parameter for name mapping or align the prompt side.

The eval_type names that appear in --list have a byob_ prefix, and the file name and bench name are normalized (lowercased, non-alphanumeric characters converted to underscores, truncated to 50 characters). Specify this normalized name in the name field of nel YAML.

If you get an error like unknown platform during cross-build, Docker buildx's multiplatform builder may not be configured. It can be enabled with docker buildx create --use. In environments where qemu-user-static is not installed, the PLATFORMS column of buildx may stop at arm64 only, so it's reassuring to check with docker buildx ls beforehand.

Operations Tool: nel-watch

An important thing to understand for operations is the nel-watch command added in v0.2.5 (PR #857). It periodically scans a specified directory and automatically submits evaluation jobs as SLURM jobs when new checkpoints appear—a useful tool for MLOps pipelines that want to link training and evaluation. Result exports to MLflow / W&B / Google Sheets also work seamlessly through the exporters written in the evaluation configuration YAML.

However, nel-watch is SLURM-only and currently does not support Local Executor or Lepton Executor. For personal use on DGX Spark alone, it's a somewhat overkill feature, and there are many situations where directly running nel run manually is faster. It's best thought of as a feature for organizations that are already running SLURM clusters.

Summary

We've dissected the internals of NeMo Evaluator and also got our hands dirty with BYOB. Looking back, the "containerized evaluation platform" skeleton is thoroughly implemented, and I find it to be a quite well-designed system from the perspectives of reproducibility and dependency isolation.

The 2-layer structure of launcher (nel) and core (nemo-evaluator), the 10-stage Interceptor pipeline, and the catalog scale of 23 harnesses and 421 tasks. With all of this in place, and even BYOB available to write custom benchmarks in 16 lines of Python and containerize them—it's quite a substantial offering.

From the perspective of DGX Spark users, here is what was confirmed in this verification:

• 4 out of the 5 harnesses checked (lm-eval / bigcode / simple-evals / garak) support ARM64 and work as-is
• Only vlmevalkit is amd64-only, but it can be worked around by cross-building with BYOB's --platform linux/amd64
• Custom benchmarks via BYOB take about 36 seconds for a native ARM64 build, and SQuAD's 10,570 records are automatically converted to JSONL and baked into the container at build time
• The only caveat when using --containerize is that you need to reinstall from the source tree with pip install -e
• Since nel-watch requires SLURM, directly calling nel run is more practical for personal DGX Spark use