I tried NVIDIA's official Japanese-enhanced LLM Nemotron 9B-v2-Japanese in various cases

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

On February 17, 2026, NVIDIA released a Japanese-enhanced LLM called NVIDIA-Nemotron-Nano-9B-v2-Japanese. It ranked 1st in the sub-10B category on Nejumi Leaderboard 4, offers up to 6x the throughput compared to Qwen3-8B, and comes with a commercially usable license. The fact that NVIDIA officially released a Japanese-enhanced model is big news for developers who want to use local LLMs in their work.

https://huggingface.co/blog/nvidia/nemotron-nano-9b-v2-japanese-ja

I happened to be experimenting with Japanese fine-tuning of Nemotron 3 Nano 4B on DGX Spark at the time, so I felt a slight pang of sadness when "the official version released something even better," but I went ahead and ran it on actual hardware to try various cases.

In this article, I'll share the results of everything I tried, from an overview of the model to setup on DGX Spark, Japanese benchmarks, and practical use cases like Tool Calling and RAG.

What is Nemotron 9B-v2-Japanese?

It is a Japanese-enhanced version of Nemotron Nano 9B, a model that has undergone additional pre-training on a Japanese corpus and fine-tuning with NVIDIA's proprietary Japanese dialogue data. It ranked 1st in the sub-10B category on Nejumi Leaderboard 4, has a commercially usable license (no MAU restrictions), and also supports Tool Calling. Details on the architecture and training process are summarized in the NVIDIA official blog post, so this article will focus on "what it can actually do."

In this article, in addition to setup on DGX Spark and Japanese benchmarks, I tested the following 4 use cases on actual hardware.

Running It on DGX Spark

pip version of vLLM Didn't Work

First, I tried using vLLM installed via pip (v0.12.0). The model downloaded and loaded without issues, but an error occurred during Triton PTX code generation at inference time.

ptxas fatal   : Value 'sm_121a' is not defined for option 'gpu-name'

DGX Spark's GB10 GPU has CUDA Compute Capability 12.1 (sm_121a), and the version of Triton used internally by the pip version of vLLM doesn't support this generation. Even when bypassing Triton compilation with the --enforce-eager flag, the same error recurred on the first inference.

NGC Container Version of vLLM Works

While the pip version didn't work, using the NGC official vLLM container (based on v0.13.0, aarch64-compatible build) allows normal inference on DGX Spark. Since the FLASH_ATTN backend is used instead of Triton, the sm_121a error does not occur.

docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host -p 8000:8000 \
    nvcr.io/nvidia/vllm:26.01-py3 \
    python3 -m vllm.entrypoints.openai.api_server \
        --model nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
        --trust-remote-code \
        --dtype bfloat16 \
        --max-model-len 4096

Model loading took about 2 minutes, and memory usage was approximately 16.6 GiB. After startup, inference is available via the OpenAI-compatible /v1/chat/completions endpoint.

Converting to GGUF and Running on Ollama

If you want to try it without Docker, you can also convert it to GGUF and run it on Ollama. Use convert_hf_to_gguf.py to convert from HuggingFace safetensors to GGUF.

# Reuse the vLLM container's PyTorch environment for GGUF conversion
docker run --gpus all --entrypoint bash \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -v /tmp:/tmp --ipc=host \
    vllm/vllm-openai:v0.12.0 -c \
    "pip install -q gguf && python /tmp/llama.cpp/convert_hf_to_gguf.py \
        /root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-Nano-9B-v2-Japanese/snapshots/*/ \
        --outtype bf16 --outfile /tmp/nemotron-9b-jp-bf16.gguf"

A BF16 GGUF file of 341 tensors and 17.8GB was generated. With DGX Spark's 128GB unified memory, this size fits comfortably at full BF16 precision.

Register it with Ollama.

cat <<'EOF' > /tmp/Modelfile-nemotron-9b-jp
FROM /tmp/nemotron-9b-jp-bf16.gguf
PARAMETER num_ctx 4096
PARAMETER temperature 0
EOF

ollama create nemotron-9b-jp -f /tmp/Modelfile-nemotron-9b-jp

Controlling Thinking Mode

9B-v2-Japanese has thinking mode (a feature that outputs the thought process with <think> tags) enabled by default.

Looking at the HuggingFace chat template, it's controlled by the enable_thinking variable. When set to False, an empty <think></think> tag is inserted in the assistant's response, telling the model "don't think."

In Ollama, this control is achieved through the Modelfile template.

# Thinking OFF (for benchmark evaluation)
ollama create nemotron-9b-jp-nothink -f /tmp/Modelfile-9b-nothink

# Thinking ON (for answers with reasoning)
ollama create nemotron-9b-jp-think -f /tmp/Modelfile-9b-think

The template content describes the chat template's special tokens (<SPECIAL_10> = System, <SPECIAL_11> = User/Assistant, <SPECIAL_12> = EOS) written in Go template syntax.

TEMPLATE """{{- range .Messages }}
{{- if eq .Role "system" }}<SPECIAL_10>System
{{ .Content }}
{{ else if eq .Role "user" }}<SPECIAL_11>User
{{ .Content }}
{{ else if eq .Role "assistant" }}<SPECIAL_11>Assistant
<think></think>{{ .Content }}
<SPECIAL_12>
{{ end }}
{{- end }}<SPECIAL_11>Assistant
<think></think>"""

TEMPLATE """{{- range .Messages }}
{{- if eq .Role "system" }}<SPECIAL_10>System
{{ .Content }}
{{ else if eq .Role "user" }}<SPECIAL_11>User
{{ .Content }}
{{ else if eq .Role "assistant" }}<SPECIAL_11>Assistant
<think>
{{ .Content }}
<SPECIAL_12>
{{ end }}
{{- end }}<SPECIAL_11>Assistant
<think>
"""

What Kind of Specs Are Needed to Run It?

This time I ran it at full BF16 precision on DGX Spark (128GB unified memory), but with quantization, it can run in more accessible environments. With 9B parameters, here are rough estimates for file size and required memory by quantization level.

Thanks to the Mamba-2 hybrid architecture, only 4 out of all 56 layers are Attention layers (verifiable in hybrid_override_pattern in config.json). Since the KV cache only needs to cover 4 layers rather than the full 56 layers of a standard Transformer, memory consumption increases gradually even with long contexts.

For choosing quantization levels, Q8_0 is recommended if prioritizing accuracy (the difference from BF16 is barely perceptible), while Q4_K_M is practical if prioritizing memory. Q5_K_M sits in between and is a good balance of accuracy and memory.

Looking at it by GPU, VRAM 12GB class (RTX 4060 Ti / RTX 3060) runs Q4_K_M comfortably. VRAM 16GB and above (RTX 4070 Ti Super / RTX 3090) can handle Q8_0 practically. For Apple Silicon Macs, 16GB unified memory should handle up to Q5_K_M, and 32GB should be fine even with full BF16 precision.

If you don't want to convert GGUF yourself, quantized models published by the community are available on HuggingFace. mmnga-o's repository offers various quantizations using a Japanese imatrix, and you can try downloading and loading them directly into Ollama or LM Studio.

Japanese Benchmark (JCommonsenseQA)

Evaluation Conditions

Measurements were taken under the same conditions as the previous Nemotron 3 Nano 4B evaluation.

Thinking mode was set to OFF consistently for a fair comparison with the existing scores from the 4B model. The evaluation script used was a custom script that sends 3-shot prompts to Ollama's REST API and extracts the answer choices (choice0-4) using regular expressions. Results measured under the same conditions (3-shot, temperature 0, thinking OFF, BF16) with the NGC vLLM container are described later.

Results

9B-v2-Japanese achieved 91.2%. This is +2.9% better than the 4B model I fine-tuned myself with 1,000 samples (88.3%).

With 2.5x more parameters and additional professional training on Japanese data, this result was as expected. What's worth noting is that the 4B base model (no FT) achieved 87.7%. With only 3.6B active parameters, the gap from the 9B is just 3.5%, showing that Nemotron 3 Nano's MoE architecture has quite high parameter efficiency.

Average latency was 0.98 seconds per question, and evaluating all 1,119 questions completed in about 18 minutes. At full BF16 precision (17.8GB), DGX Spark's 128GB unified memory has plenty of room.

Accuracy Differences by Inference Engine

JCommonsenseQA was also measured under the same conditions (3-shot, temperature 0, thinking OFF, BF16) with the NGC vLLM container. This is a comparison with only the inference engine changed, using the same model on the same hardware. For reference, results on SageMaker (ml.g5.2xlarge, A10G GPU) with vLLM are also included.

Comparing Ollama (llama.cpp) and vLLM on the same DGX Spark hardware, there was a 7.7 percentage point difference in accuracy. Since the hardware is identical, this difference stems from differences in the inference engine implementations. Between the two vLLM setups (DGX Spark 83.5% and SageMaker 84.8%), the difference is only 1.3 percentage points despite different versions and GPUs, suggesting a trend of "inference engine differences > hardware differences."

Nemotron 9B-v2-Japanese uses a Transformer + Mamba-2 hybrid architecture, with only 4 Attention layers out of 56 total, and the remaining 52 layers alternating between Mamba-2 (27 layers) and MLP (25 layers). Mamba-2's state-space model operations have more implementation variation compared to Transformer Attention, making it prone to differences in internal numerical processing between llama.cpp and vLLM. This implementation difference in the Mamba-2 layers may be responsible for the accuracy differences.

On the other hand, vLLM is about 2.6x faster than Ollama in terms of latency (0.38 sec vs 0.98 sec), and processing all 1,119 questions completed in about 7 minutes (compared to about 18 minutes for Ollama). This suggests using Ollama when prioritizing accuracy, and vLLM when prioritizing throughput.

Trying Various Cases

Now that I understand the model's basic performance, let me try some cases that might be useful in actual work.

Tool Calling

When you ask a company chatbot "What's the weather like tomorrow?", it calls a weather API; when you say "Tell me my schedule for next week," it references a calendar API. Tool Calling is a mechanism that delegates to an LLM the task of "choosing the appropriate API from natural language instructions and calling it." External functions are defined with JSON schemas, and the model decides which function to call with which arguments.

The model uses a proprietary format, receiving tool definitions via <AVAILABLE_TOOLS> tags and returning calls via <TOOLCALL> tags. Since Ollama's custom Modelfile doesn't support the native tools API, I verified by embedding tool definitions in the system message.

I defined two functions—weather retrieval and calendar lookup—and when asked "Please tell me tomorrow's weather in Tokyo," it correctly called get_weather with city: "Tokyo".

To retrieve tomorrow's weather forecast for Tokyo, I need date information.
I will calculate "tomorrow's" date based on the current date.

<TOOLCALL>[{"name": "get_weather", "arguments": {"city": "東京"}}]</TOOLCALL>

For cases requiring multiple tools, such as "Check my schedule for February 20th and also tell me the weather in Osaka that day," it handled it with a sequential execution pattern: first calling get_calendar_events, receiving the response, and then calling get_weather. Rather than parallel calling, it processes one at a time, but the result is a response that correctly integrates both pieces of information.

<TOOLCALL>[{"name": "get_calendar_events", "arguments": {"date": "2026-02-20"}}]</TOOLCALL>

Your schedule for February 20, 2026 is as follows:
- 10:00-11:00: Team meeting
- 14:00-17:00: Osaka business trip

Additionally, I will call the weather API to retrieve Osaka's weather information.
<TOOLCALL>[{"name": "get_weather", "arguments": {"city": "大阪"}}]</TOOLCALL>

I also tested structured JSON output. When asked to extract 7 fields—name / age / gender / address / company / position / email—from a text like "Taro Tanaka (35 years old, male) lives in Shibuya, Tokyo...," all fields were extracted accurately. Combined with Ollama's format: "json" option, stable JSON output is obtained.

{
  "name": "田中太郎",
  "age": 35,
  "gender": "男性",
  "address": "東京都渋谷区",
  "company": "株式会社テック",
  "position": "主任エンジニア",
  "email": "tanaka@example.com"
}

RAG (Retrieval-Augmented Generation)

"According to this internal policy, can contract employees work remotely?"—answering such inquiries by having people re-read the regulations each time is burdensome. RAG is a pattern that passes documents as context to an LLM and has it respond based on that content. I tested this with a scenario of using it as a QA bot for internal regulations and product manuals.

I passed a fictional remote work company policy (5 sections on eligible employees, usage days, application procedures, security requirements, and expenses, approximately 400 characters) as context and asked 3 questions.

All 3 questions were answered accurately. In particular, the correct judgment of "not stated" for question 3 is practically valuable. Whether a model can correctly say "no" about something not written in the document directly affects the reliability of RAG. Response time was about 10 seconds.

1. Contract employees cannot work remotely in principle, but with approval
   from their department head, up to 2 days per week is permitted.

2. If you want to suddenly change to remote work, you need to notify your
   supervisor by email by 5 PM the previous day. Also, since the deadline
   for advance applications in the attendance system is the 25th of the
   previous month, this email procedure applies for sudden changes.

3. Not stated

Summarization and Translation

Wanting to summarize a long meeting memo in 3 lines, or translate an email to send to an overseas office into English—these are common text processing tasks that, when handled by a local LLM, have the advantage of not sending data to external services.

When I instructed the model to "summarize in 3 lines" a Japanese text about generative AI and data governance (3 paragraphs, approximately 500 characters), it returned a summary that accurately compressed the three points: common principles of GDPR and Japan's Personal Information Protection Act, the use of on-premises and region-specified cloud services, and hybrid operations. The structure of the original text was preserved, and there were no missing pieces of information.

The proliferation of generative AI has made corporate data governance increasingly important,
with the handling of personal information in LLM training data becoming a particular challenge.
Regulations across countries regarding cross-border data transfers (GDPR, Japanese law, etc.)
share "transparency" as a common principle, and the use of on-premises and region-specified cloud
services for domestic data retention is growing. Hybrid operations tailored to the level of
confidentiality are gaining attention as a practical solution.

For Japanese-to-English translation, I had a business document about handling personal information (starting with "Our company, in order to properly manage customer personal information...") translated into English. The result was natural English suitable for a business document, like "Our company processes customer personal information exclusively at our domestic data centers." Technical terms like "closed network" (閉域ネットワーク) were appropriately translated as "closed network environment." Translation completed in about 4.5 seconds.

Our company processes customer personal information exclusively at
our domestic data centers to ensure proper management. We do not
transfer any data to overseas servers, and security is maintained
through encrypted communication and a closed network environment.
If you have any questions or concerns, please contact the
Privacy Office.

Code Generation

"I want to aggregate this data a bit, but it's annoying to look up pandas syntax every time." "I want to quickly write a SQL query for this condition." These are cases where repeatedly generated scripts for data analysis and report creation can be delegated to an LLM.

I requested "a Python script that reads customer data from a CSV and aggregates the number of people and average purchase amount by age group." Practical code using pandas was generated, including age group classification with pd.cut(), aggregation with groupby().agg(), and error handling for FileNotFoundError and empty data. Japanese comments were also natural. However, it took about 92 seconds, so you'll need to wait a bit when code volume is large.

# Script to aggregate customer data by age group
# CSV file format required: name, age, purchase_amount
# Output: Display number of people and average purchase amount by age group

import pandas as pd

def main():
    # 1. Specify the CSV file path
    file_path = 'customers.csv'

    try:
        # 2. Load CSV data with pandas
        df = pd.read_csv(file_path)

        # 3. Check if required columns exist (error prevention)
        required_columns = ['name', 'age', 'purchase_amount']
        if not all(col in df.columns for col in required_columns):
            missing = [col for col in required_columns if col not in df.columns]
            raise ValueError(f"Required columns are missing: {missing}")

        # 4. Verify that age is numeric
        if not pd.api.types.is_numeric_dtype(df['age']):
            raise TypeError("The 'age' column must be a numeric type")

        if not pd.api.types.is_numeric_dtype(df['purchase_amount']):
            raise TypeError("The 'purchase_amount' column must be a numeric type")

        # 5. Create age group as a new column (20s, 30s...)
        df['age_group'] = pd.cut(
            df['age'],
            bins=[0, 19, 29, 39, 49, 100],
            labels=['19歳以下', '20代', '30代', '40代', '50代以上'],
            right=False
        )

        # 6. Aggregate by age group: count and average purchase amount
        result = df.groupby('age_group').agg(
            人数=('name', 'count'),
            平均購入金額=('purchase_amount', 'mean')
        ).reset_index()

        # 7. Display results (up to 2 decimal places)
        print(result.to_string(
            index=False,
            float_format=lambda x: f"{x:.2f}"
        ))

    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
    except pd.errors.EmptyDataError:
        print("Error: The CSV file is empty.")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

if __name__ == "__main__":
    main()

For SQL, I had it generate a query with a JOIN of an orders table and a customer table + WHERE conditions (period specification and member rank filtering) + GROUP BY + ORDER BY.

SELECT c.都道府県, SUM(o.数量 * o.単価) AS 売上合計
FROM orders o
JOIN customers c ON o.顧客ID = c.顧客ID
WHERE YEAR(o.注文日) = 2026 AND MONTH(o.注文日) = 1
  AND c.会員ランク = 'ゴールド'
GROUP BY c.都道府県
ORDER BY 売上合計 DESC;

It handled Japanese column names as-is, and the logical structure of the query is correct. It was generated in about 34 seconds.

Comparing Deployment Options

When actually using Nemotron 9B-v2-Japanese in business, the next question is where to run it. Here I'll organize 3 options.

When dealing with personal information or confidential data, it may be necessary to consider cross-border data transfers. Using SageMaker in the Tokyo Region allows you to set up a closed configuration within a VPC, achieving a configuration where data never leaves Japan. Since Bedrock model availability was unconfirmed as of February 2026, SageMaker is the practical choice when a reliably closed configuration is required.

DGX Spark enables fully local operation without network connectivity, making it suitable for prototyping during the development phase or for verifications where you want zero risk of data leaking externally. However, since it is limited to the processing capacity of a single machine, migration to the cloud may be considered if production throughput requirements are strict.

Summary

I ran the NVIDIA official Japanese-enhanced model Nemotron 9B-v2-Japanese on DGX Spark and tested it with benchmarks and practical use cases.

To summarize my impressions from practical use cases: Tool Calling and JSON structured output are at a production-ready level, RAG was accurate including "not stated" judgments, and summarization/translation was of sufficient quality for business documents. Code generation also produced practical pandas and SQL scripts.

Within the scope of this verification, what I found particularly interesting was that even running the same model on the same hardware, there was a 7.7 percentage point difference in accuracy depending on the inference engine. Whether Mamba-2 hybrid architecture tends to produce differences in internal numerical processing between llama.cpp and vLLM is worth noting—it may make sense to use Ollama (llama.cpp) when prioritizing accuracy, and vLLM when prioritizing throughput.

The pip version of vLLM doesn't work on DGX Spark, but the NGC container version (26.01-py3) allows vLLM to be used as well. Initially this article stated "vLLM didn't work," but failing to check NGC was a lesson learned. When running something on DGX Spark, I should check the NGC catalog first, before pip packages.

Personally, this was an experience where "having tried FT through trial and error myself, I could truly appreciate the quality of the official model." Seeing the official version achieve +3.5% against the 4B model where I squeezed out +0.6% through FT with 1,000 samples shows that professional work with both the quality and quantity of data is truly powerful.

On the other hand, when it comes to using it in business, the question of "where to run it" becomes important. I'd like to try deploying to SageMaker Tokyo Region and domain-specific fine-tuning as well.

Extra: Having Nemotron Review Its Own Article

Just for fun, I had Nemotron 9B-v2-Japanese itself read a summary of this article and asked for its impressions.

It may be a bit too self-congratulatory, but the improvement suggestions pointing out "there are few mentions of inference speed" and "there are no comparisons with other models" are quite on point. It was also an interesting experiment in what an LLM says when reviewing itself.

Reference Links

• NVIDIA-Nemotron-Nano-9B-v2-Japanese（HuggingFace）
• Nemotron-H Technical Report（arXiv）
• NVIDIA Official Blog（Japanese）
• Nejumi Leaderboard 4
• Japanese Fine-tuning of Nemotron 3 Nano on DGX Spark
• NGC vLLM Container
• llama.cpp（GGUF conversion）