Compared Nemotron 3 Nano Omni / Gemma 4 / Cosmos-Reason2 on a Japanese multimodal benchmark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

This is a follow-up to the article I published the other day about running Nemotron 3 Nano Omni with 4 modalities on DGX Spark. Fulfilling the "full-scale evaluation of Japanese multimodal" promised at the end of the first part, I ran side-by-side benchmarks of Nemotron 3 Nano Omni / Gemma 4 / Cosmos-Reason2 on two benchmarks: Heron-Bench and JMMMU.

The verification was completed on a single DGX Spark unit, and source code and result data are also available on GitHub. To summarize the conclusion in one sentence: Gemma 4 stood out ahead in open-ended tasks, while the 3 models were nearly equivalent in multiple-choice tasks — a somewhat surprising result.

The 3 Models Being Compared

Here is a brief summary of the characteristics of the 3 models.

I initially tried Gemma 4 31B Dense, but it was 8 times slower than Omni MoE and not practical at the scale of this article, so I switched to the same MoE architecture 26B-A4B (Active 3.8B) (the 31B Dense numbers are in the latency footnote at the end).

About the Benchmarks

In this article, I used two benchmarks.

Heron-Bench

turing-motors/Japanese-Heron-Bench is a Japanese multimodal evaluation benchmark published by Turing. It consists of 102 questions across 3 categories — "conversation," "detailed description," and "complex reasoning" — for 21 Japan-related images (anime, Japanese food, landmarks, calligraphy, etc.).

Evaluation uses LLM-as-a-Judge, where model responses are compared against reference answers pre-generated by GPT-4, and a separate LLM scores them on a 1–5 scale. In this article, Claude Haiku 4.5 was used as the judge for 5-level scoring. This benchmark is well-suited for measuring expressive ability in open-ended writing and understanding of Japan-specific context.

JMMMU

JMMMU/JMMMU is the Japanese version of MMMU, a standard image reasoning benchmark. A total of 1,320 questions are drawn from 28 specialized fields, including chemistry, mechanical engineering, Japanese history, design, and world history. It is a multiple-choice format where the model selects the correct answer from options A–D based on an image and question.

Scoring is based on exact match with the correct label, requiring no judge. It is well-suited for measuring chart/figure comprehension and logical reasoning based on specialized knowledge.

Verification Environment

All 3 models were run under the same conditions: --max-model-len 8192 / --gpu-memory-utilization 0.4 / --enforce-eager. See GitHub for each launch script.

Heron-Bench Scores (Open-Ended)

Here are the results from running 102 questions on 3 models.

Gemma 4 26B-A4B led in all categories with a +0.8–1.1 point margin over the other 2 models. Looking at the breakdown by image category, Gemma 4 also topped all areas including anime, food (Japanese cuisine), and landmark (Japanese landmarks). Omni's struggles are consistent with the "English-first" nature discussed in the first part, but it was beyond expectations to see Gemma 4 — which shares the same Active 3–4B MoE architecture — perform this strongly, possibly suggesting that Japanese language capability was sacrificed for the 4-modality integration.

JMMMU Scores (Multiple Choice)

Next, the JMMMU results for 1,320 questions.

In stark contrast to Heron-Bench, the 3 models were nearly equivalent here (within a 3.2pt gap). Surprisingly, the smallest model, Cosmos-Reason2-8B, took first place, with no gap proportional to parameter scale visible against the two Active 3–4B class MoE models.

Looking at individual fields, all models performed well in history and culture areas like World_History (0.68–0.76 across all models) and Japanese_Heritage (Gemma 4 notably high at 0.687), while all 3 models struggled with Music and Mechanical_Engineering. It's also interesting to note that Gemma 4 alone collapsed dramatically in Energy_and_Power at 0.067, apparently struggling significantly with technical chart-based questions.

The fact that "large gaps appear in open-ended tasks, but little difference shows in multiple-choice" provides a practical guideline: Gemma 4 is the safe choice for text-generating tasks like image captioning, tagging, or summarization, while an 8B-class model is sufficient for multiple-choice tasks in specific domains.

Inference Latency

Here is the inference time per question for each model (wall-clock time with max_concurrency=2 parallel execution).

Omni was overwhelmingly fast, with Gemma 4 26B-A4B approximately 5x slower and CR2 in between. Looking at active parameter scale alone, Omni (3B) and Gemma 4 (3.8B) are nearly equivalent, so this gap is likely mainly due to optimization differences in the MoE backend (Omni uses flashinfer_cutlass, Gemma 4 uses vLLM auto-pick) and activation functions (SiLU-type vs. GELU-type), though this warrants further investigation.

Output Sample Comparison

For reference, here are 2 examples of responses from the 3 models to the same Heron-Bench questions.

Simple Factual Response (heron-101: Distance to Niseko)

For a straightforward question that only requires reading a number, all 3 models answered correctly without any issues.

Complex Reasoning (heron-005: Synopsis of an illustrated work)

The question is "Please tell me in detail the synopsis of the work shown in this illustration," and the image is a famous scene from Spirited Away featuring Chihiro, No-Face, and Yubaba together.

As shown here, for questions requiring complex scene description or interpretation of Japan-specific context, only Gemma 4 consistently scored high, while Omni and CR2 showed notable misidentifications and deviations.

Guideline for Choosing Models

Here is a rough guideline for choosing models based on the results so far.

Summary

I ran Heron-Bench and JMMMU across 3 models on a single DGX Spark unit. The conclusion of this evaluation is: "Gemma 4 26B-A4B stands a head above the rest for Japanese open-ended writing, while the 3 models are roughly on par for multiple-choice tasks." Omni has a unique position with 4-modality integration, but falls one step behind Gemma 4 in pure Japanese benchmarks.

Next time, I plan to try a configuration using NeMo Agent Toolkit combined with Omni as a perception sub-agent. Please look forward to it as a follow-up to the earlier Agent Toolkit article NAT1.

Behind the Verification

Run management for 3 models × 1,422 questions was done by reusing the environment built in the previous Langfuse Self-host article. Each model was turned into a Dataset Run with langfuse.run_experiment(), judge scores and API costs were automatically aggregated via Langfuse Trace Cost, and only the graphs for the blog were separately finalized with matplotlib — a hybrid setup. The actual verification took approximately 3.5 hours, with judge costs coming to just under $2 on the Anthropic side.

Reference Links

• Part 1: Running Nemotron 3 Nano Omni with 4 Modalities on DGX Spark
• Hugging Face: Nemotron 3 Nano Omni NVFP4
• Hugging Face: Gemma 4 26B-A4B NVFP4
• Hugging Face: Cosmos-Reason2-8B
• Hugging Face: Japanese-Heron-Bench
• Hugging Face: JMMMU
• Langfuse Self-host LLM Observability Hands-on
• GitHub: Verification scripts and result data