Compared Nemotron 3 Nano Omni / Gemma 4 / Cosmos-Reason2 on a Japanese multimodal benchmark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

This is a follow-up to the article I published the other day about running Nemotron 3 Nano Omni with 4 modalities on DGX Spark. Fulfilling the "full-scale evaluation of Japanese multimodal" promised at the end of the previous article, I benchmarked Nemotron 3 Nano Omni / Gemma 4 / Cosmos-Reason2 side by side on two benchmarks: Heron-Bench and JMMMU.

The verification was completed on a single DGX Spark unit, and the source code and result data are also available on GitHub. To summarize the conclusion in one sentence: Gemma 4 stood out ahead in open-ended tasks, while the 3 models were nearly equivalent in multiple-choice tasks — a somewhat surprising result.

The 3 Models Being Compared

Here is a brief summary of the characteristics of the 3 models.

I initially tried Gemma 4 31B Dense, but it was 8 times slower than Omni MoE and was not practical at the scale of this article, so I switched to the same MoE architecture 26B-A4B (Active 3.8B) (the 31B Dense numbers are in the latency footnote at the end).

About the Benchmarks

Two benchmarks were used in this article.

Heron-Bench

turing-motors/Japanese-Heron-Bench is a Japanese multimodal evaluation benchmark published by Turing. It consists of 102 questions across 3 categories — "conversation," "detailed description," and "complex reasoning" — covering 21 images related to Japan (anime, Japanese food, landmarks, calligraphy, etc.).

Evaluation uses the LLM-as-a-Judge method, where model responses are compared against reference answers pre-generated by GPT-4, and another LLM scores them on a scale of 1 to 5. In this article, Claude Haiku 4.5 was used as the judge, scoring on a 5-point scale. This approach is well-suited for measuring expressive ability in open-ended writing and understanding of Japan-specific contexts.

JMMMU

JMMMU/JMMMU is the Japanese version of MMMU, a standard benchmark for image reasoning. A total of 1,320 questions are drawn from 28 specialized fields, including chemistry, mechanical engineering, Japanese history, design, and world history. It is a multiple-choice format where the model selects the correct answer from options A–D based on an image and question.

Scoring is based on exact match with the correct label, so no judge is needed. It is well-suited for measuring diagram comprehension and logical reasoning grounded in specialized knowledge.

Verification Environment

All 3 models were run under the same conditions: --max-model-len 8192 / --gpu-memory-utilization 0.4 / --enforce-eager. Please refer to GitHub for each launch script.

Heron-Bench Scores (Open-ended)

Here are the results from running 102 questions across 3 models.

Gemma 4 26B-A4B won across all categories with a margin of +0.8 to 1.1 points over the other 2 models. In the breakdown by image category as well, Gemma 4 topped all areas including anime, food (Japanese cuisine), and landmark (Japanese landmarks). Omni's struggle is consistent with its "English-first" nature mentioned in the previous article, but Gemma 4 being this strong despite sharing the same Active 3–4B MoE architecture was beyond expectations — it may be that the cost of 4-modality integration has come at the expense of Japanese language performance.

JMMMU Scores (Multiple Choice)

Next, the 1,320-question JMMMU results.

In stark contrast to Heron-Bench, all 3 models were nearly equivalent here (within a 3.2pt difference). Surprisingly, the smallest model, Cosmos-Reason2-8B, took the top spot, with no gap commensurate with the parameter scale difference compared to the two Active 3–4B class MoE models.

Looking at individual fields, history and culture categories such as World_History (0.68–0.76 for all models) and Japanese_Heritage (Gemma 4 stood out at 0.687) were strengths across all models, while Music and Mechanical_Engineering were challenging for all three. It's also interesting to note that only Gemma 4 dropped extremely low in Energy_and_Power at 0.067, showing a significant collapse on technical diagram questions.

The finding that "open-ended tasks show large differences while multiple-choice tasks show little" suggests a practical guideline: Gemma 4 is the safe choice for tasks where models generate text, such as image captioning, tagging, and summarization, while 8B-class models are sufficient for specific-domain multiple-choice tasks.

Inference Latency

Here is the inference time per question for each model (wall-clock time with parallel execution at max_concurrency=2).

Omni was overwhelmingly fast, with Gemma 4 26B-A4B about 5 times slower and CR2 falling in between. Looking at active parameter scale alone, Omni (3B) and Gemma 4 (3.8B) are nearly equivalent, so this difference is likely mainly due to MoE backend optimization differences (Omni uses flashinfer_cutlass, Gemma 4 uses vLLM auto-pick) and activation function differences (SiLU-based vs. GELU-based), though this warrants further investigation.

Output Sample Comparison

For reference, here are responses from all 3 models to the same 2 Heron-Bench questions.

Simple factual response (heron-101: Distance to Niseko)

All 3 models answered correctly without issue on this straightforward question that only requires reading a number.

Complex reasoning (heron-005: Plot summary of an illustrated work)

The question was "Please tell me the plot of the work depicted in this illustration in detail," with the image being a famous scene from Spirited Away showing Chihiro, No-Face, and Yubaba together.

As shown here, on questions requiring complex scene description or interpretation of Japan-specific context, only Gemma 4 scored consistently high, while Omni and CR2 showed notable misidentifications and deviations.

Guidelines for Choosing a Model

Based on the results so far, here are rough guidelines for choosing between models.

Summary

I benchmarked 3 models across Heron-Bench and JMMMU on a single DGX Spark unit. The conclusion this time is: "Gemma 4 26B-A4B stands out ahead in Japanese open-ended writing, but the 3 models are roughly on par for multiple-choice tasks." Omni holds a unique position with its 4-modality integration, but takes a step back from Gemma 4 in pure Japanese benchmarks.

Next time, I plan to try a configuration that uses Omni as a perception sub-agent in combination with NeMo Agent Toolkit. Please look forward to it as a continuation of the previous Agent Toolkit article NAT1.

Behind the Verification

Run management for 3 models × 1,422 questions was done using the environment built in my previous Langfuse Self-host article. Each model was turned into a Dataset Run with langfuse.run_experiment(), judge scores and API costs were automatically aggregated via Langfuse Trace Cost, and graphs for the blog were separately finalized with matplotlib — a hybrid setup. The actual verification took about 3.5 hours, and the judge cost on the Anthropic side was just under $2.

Reference Links

• Part 1: Running Nemotron 3 Nano Omni with 4 Modalities on DGX Spark
• Hugging Face: Nemotron 3 Nano Omni NVFP4
• Hugging Face: Gemma 4 26B-A4B NVFP4
• Hugging Face: Cosmos-Reason2-8B
• Hugging Face: Japanese-Heron-Bench
• Hugging Face: JMMMU
• Langfuse Self-host LLM Observability Hands-on
• GitHub: Verification scripts and result data