I tried NVIDIA's official Japanese-enhanced LLM Nemotron 9B-v2-Japanese in various cases

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

On February 17, 2026, NVIDIA released NVIDIA-Nemotron-Nano-9B-v2-Japanese, an LLM with enhanced Japanese language capabilities. It ranked 1st in the sub-10B category on Nejumi Leaderboard 4, achieves up to 6x the throughput of Qwen3-8B, and comes with a commercially usable license. The fact that NVIDIA officially released a Japanese-enhanced model is major news for developers who want to use local LLMs in their work.

https://huggingface.co/blog/nvidia/nemotron-nano-9b-v2-japanese-ja

I happened to be experimenting with Japanese fine-tuning of Nemotron 3 Nano 4B on DGX Spark at the time, so I felt a slight pang of bittersweet emotion at the situation where "the official team released something even better," but I ran it on actual hardware and tested various cases.

In this article, I'll share the results of testing everything from an overview of the model and setup on DGX Spark, to Japanese benchmarks, and practical use cases such as Tool Calling and RAG.

What is Nemotron 9B-v2-Japanese

This is a Japanese-enhanced version of Nemotron Nano 9B, a model that has undergone additional pre-training on Japanese corpora and fine-tuning with NVIDIA's proprietary Japanese conversational data. It ranks 1st in the sub-10B category on Nejumi Leaderboard 4, comes with a commercially usable license (no MAU restrictions), and also supports Tool Calling. The architecture and training process details are summarized in NVIDIA's official introduction article, so this article will focus on "what it can actually do."

In addition to setup on DGX Spark and Japanese benchmarks, I tested the following 4 use cases on actual hardware.

Running on DGX Spark

pip version of vLLM didn't work

First, I tried in an environment where vLLM was installed via pip (v0.12.0), as recommended officially. The model download and loading completed without issues, but an error occurs during inference in Triton's PTX code generation.

ptxas fatal   : Value 'sm_121a' is not defined for option 'gpu-name'

The DGX Spark's GB10 GPU has CUDA Compute Capability 12.1 (sm_121a), and the version of Triton used internally by the pip version of vLLM doesn't support this generation. Even when bypassing Triton compilation with the --enforce-eager flag, the same error recurred on the first inference.

NGC container version of vLLM works

While vLLM didn't work with the pip version, using the official NGC vLLM container (based on v0.13.0, aarch64-compatible build) allows normal inference on DGX Spark. Since the FLASH_ATTN backend is used instead of Triton, the sm_121a error doesn't occur.

docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host -p 8000:8000 \
    nvcr.io/nvidia/vllm:26.01-py3 \
    python3 -m vllm.entrypoints.openai.api_server \
        --model nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
        --trust-remote-code \
        --dtype bfloat16 \
        --max-model-len 4096

Model loading took about 2 minutes, with memory usage of approximately 16.6 GiB. After startup, inference is available via the OpenAI-compatible /v1/chat/completions endpoint.

Converting to GGUF and running with Ollama

If you want to try it without using Docker, you can also convert to GGUF and run it with Ollama. Use convert_hf_to_gguf.py to convert from HuggingFace safetensors to GGUF.

# Reusing the vLLM container's PyTorch environment for GGUF conversion
docker run --gpus all --entrypoint bash \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -v /tmp:/tmp --ipc=host \
    vllm/vllm-openai:v0.12.0 -c \
    "pip install -q gguf && python /tmp/llama.cpp/convert_hf_to_gguf.py \
        /root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-Nano-9B-v2-Japanese/snapshots/*/ \
        --outtype bf16 --outfile /tmp/nemotron-9b-jp-bf16.gguf"

A BF16 GGUF file with 341 tensors and 17.8GB was generated. With DGX Spark's 128GB unified memory, this size fits comfortably at BF16 full precision.

Registering with Ollama:

cat <<'EOF' > /tmp/Modelfile-nemotron-9b-jp
FROM /tmp/nemotron-9b-jp-bf16.gguf
PARAMETER num_ctx 4096
PARAMETER temperature 0
EOF

ollama create nemotron-9b-jp -f /tmp/Modelfile-nemotron-9b-jp

Controlling thinking mode

9B-v2-Japanese has thinking mode enabled by default (a feature that outputs the thought process with <think> tags).

Looking at the HuggingFace chat template, it's controlled by the enable_thinking variable. When set to False, empty <think></think> tags are inserted in the assistant's response, telling the model "don't think."

In Ollama, this control is achieved through the Modelfile template.

# Thinking OFF (for benchmark evaluation)
ollama create nemotron-9b-jp-nothink -f /tmp/Modelfile-9b-nothink

# Thinking ON (for responses with reasoning)
ollama create nemotron-9b-jp-think -f /tmp/Modelfile-9b-think

The content of the template is the special tokens from the chat template (<SPECIAL_10> = System, <SPECIAL_11> = User/Assistant, <SPECIAL_12> = EOS) written in Go template syntax.

TEMPLATE """{{- range .Messages }}
{{- if eq .Role "system" }}<SPECIAL_10>System
{{ .Content }}
{{ else if eq .Role "user" }}<SPECIAL_11>User
{{ .Content }}
{{ else if eq .Role "assistant" }}<SPECIAL_11>Assistant
<think></think>{{ .Content }}
<SPECIAL_12>
{{ end }}
{{- end }}<SPECIAL_11>Assistant
<think></think>"""

TEMPLATE """{{- range .Messages }}
{{- if eq .Role "system" }}<SPECIAL_10>System
{{ .Content }}
{{ else if eq .Role "user" }}<SPECIAL_11>User
{{ .Content }}
{{ else if eq .Role "assistant" }}<SPECIAL_11>Assistant
<think>
{{ .Content }}
<SPECIAL_12>
{{ end }}
{{- end }}<SPECIAL_11>Assistant
<think>
"""

What kind of specs are needed to run it

This time I ran it at BF16 full precision on DGX Spark (128GB unified memory), but quantization allows it to run in more accessible environments. With 9B parameters, the estimated file sizes and required memory for different quantization levels are as follows:

Thanks to the Mamba-2 hybrid architecture, out of all 56 layers, only 4 are Attention layers (verifiable in config.json's hybrid_override_pattern). Since the KV cache only needs 4 layers worth rather than the full 56 layers of a normal Transformer, memory consumption increases more gradually with longer contexts.

As for choosing quantization level, Q8_0 is recommended when prioritizing accuracy (the difference from BF16 is barely perceptible), while Q4_K_M is realistic when prioritizing memory. Q5_K_M is in between, offering a good balance of accuracy and memory.

Looking at GPU categories, VRAM 12GB class (RTX 4060 Ti / RTX 3060) can run comfortably with Q4_K_M. VRAM 16GB or more (RTX 4070 Ti Super / RTX 3090) makes Q8_0 practical. For Apple Silicon Macs, up to Q5_K_M with 16GB unified memory, and BF16 full precision should be fine with 32GB.

Without converting GGUF yourself, there are quantized models published by the community on HuggingFace. mmnga-o's repository offers various quantizations using Japanese imatrix, and you may be able to download and load them directly into Ollama or LM Studio to try.

Japanese Benchmark (JCommonsenseQA)

Evaluation Conditions

Measured under the same conditions as the previous evaluation of Nemotron 3 Nano 4B.

Thinking mode was set to OFF uniformly for fair comparison with existing scores from the 4B model. The evaluation script used was a custom script that sends 3-shot prompts to Ollama's REST API and extracts choices (choice0-4) from responses using regular expressions. Results measured under the same conditions (3-shot, temperature 0, thinking OFF, BF16) using the NGC vLLM container are described later.

Results

9B-v2-Japanese achieved 91.2%. This exceeds the self-fine-tuned 4B model with 1,000 examples (88.3%) by +2.9%.

With 2.5x more parameters and additional professional training on Japanese data, this result is as expected. What I find more noteworthy is that the 4B base model (without FT) scored 87.7%. The gap between 3.6B active parameters and 9B is only 3.5%, showing that the MoE architecture of Nemotron 3 Nano was quite parameter-efficient.

Average latency was 0.98 seconds per question, with all 1,119 questions completing in about 18 minutes. At BF16 full precision (17.8GB), DGX Spark's 128GB unified memory has plenty of headroom.

Accuracy differences by inference engine

I also measured JCommonsenseQA with the NGC vLLM container under the same conditions (3-shot, temperature 0, thinking OFF, BF16). This is a comparison with the same model and same hardware but different inference engines. For reference, I've also included results from vLLM on SageMaker (ml.g5.2xlarge, A10G GPU).

Comparing Ollama (llama.cpp) and vLLM on the same DGX Spark, there was a 7.7 percentage point difference in accuracy. Since the hardware is identical, this difference stems from differences in the inference engine implementations. Between the two vLLM environments (DGX Spark 83.5% vs. SageMaker 84.8%), despite different versions and GPUs, the difference is only 1.3 points, suggesting a trend of "inference engine differences > hardware differences."

Nemotron 9B-v2-Japanese uses a Transformer + Mamba-2 hybrid architecture with only 4 Attention layers out of 56, the remaining 52 layers alternating between Mamba-2 (27 layers) and MLP (25 layers). The state space model computations of Mamba-2 have more implementation variation compared to Transformer Attention, making it a structure where numerical processing differences can more easily arise between llama.cpp and vLLM. This difference in Mamba-2 layer implementation may be reflected in the accuracy gap.

On the other hand, in terms of latency, vLLM is about 2.6x faster than Ollama (0.38 sec vs. 0.98 sec), and processing all 1,119 questions completed in about 7 minutes (Ollama took about 18 minutes). A clear use case distinction emerges: Ollama if prioritizing accuracy, vLLM if prioritizing throughput.

Testing Various Cases

Now that the model's basic performance is understood, let's test cases that are likely useful in practice.

Tool Calling

When you ask an internal chatbot "What's the weather tomorrow?" it calls the weather API; when you say "Tell me my schedule for next week" it references the calendar API. Tool Calling is the mechanism of having the LLM serve as the bridge that "selects and calls the appropriate API from natural language instructions." External functions are defined with JSON schemas, and the model judges which function to call with which arguments.

The model uses a proprietary format, receiving tool definitions in <AVAILABLE_TOOLS> tags and returning calls in <TOOLCALL> tags. Since Ollama's custom Modelfile doesn't support the native tools API, I tested by embedding tool definitions in the system message.

With two functions defined — weather retrieval and calendar reference — when asked "Please tell me tomorrow's weather in Tokyo," it correctly called get_weather with city: "Tokyo".

To get the weather forecast for Tokyo tomorrow, I need date information.
I'll calculate tomorrow's date based on the current date.

<TOOLCALL>[{"name": "get_weather", "arguments": {"city": "東京"}}]</TOOLCALL>

For cases requiring multiple tools, such as "Check the schedule for February 20th and also tell me the weather in Osaka that day," it handled this with a sequential execution pattern: first calling get_calendar_events, then after receiving the response, calling get_weather. Rather than parallel calls, it processes one at a time, but ultimately returned a response that correctly integrated both pieces of information.

<TOOLCALL>[{"name": "get_calendar_events", "arguments": {"date": "2026-02-20"}}]</TOOLCALL>

The schedule for February 20, 2026 is as follows:
- 10:00-11:00: Team meeting
- 14:00-17:00: Osaka business trip

I'll also call the weather API to get weather information for Osaka.
<TOOLCALL>[{"name": "get_weather", "arguments": {"city": "大阪"}}]</TOOLCALL>

I also tested structured JSON output. When asked to extract 7 fields — name / age / gender / address / company / position / email — from text describing "Taro Tanaka (35 years old, male) lives in Shibuya Ward, Tokyo…", all fields were accurately extracted. Combined with Ollama's format: "json" option, stable JSON output was obtained.

{
  "name": "田中太郎",
  "age": 35,
  "gender": "男性",
  "address": "東京都渋谷区",
  "company": "株式会社テック",
  "position": "主任エンジニア",
  "email": "tanaka@example.com"
}

RAG (Retrieval-Augmented Generation)

"According to these internal regulations, can temporary staff work remotely?" — Answering such inquiries by having people read through the regulations each time is time-consuming. RAG is a pattern where documents are passed as context to the LLM, which then responds based on that content. I tested this with the use case of a QA bot for internal regulations and product manuals in mind.

Passing a fictional remote work internal regulation (5 sections covering eligible employees, number of days, application procedure, security requirements, and expenses, approximately 400 characters) as context, I asked 3 questions.

All 3 questions were answered accurately. The fact that the "not stated" determination for question 3 worked correctly is particularly practical. Whether the model can say "no" about something not written in the document is directly linked to the reliability of RAG. Response time was about 10 seconds.

1. Temporary staff cannot in principle work remotely, but with the
   approval of their department head, they may use it up to 2 days per week.

2. If you want to suddenly change to remote work, you need to contact
   your supervisor by email by 5pm the previous day. Also, since the
   deadline for advance application in the attendance system is the 25th
   of the previous month, this email procedure applies for sudden changes.

3. Not stated

Summarization and Translation

Wanting to summarize a long meeting memo in 3 lines, wanting to translate an email to be sent to an overseas branch — these are everyday text processing tasks that can be handled entirely with a local LLM, with the advantage that data doesn't need to be sent to external services.

When instructed to "summarize in 3 lines" a Japanese text about generative AI and data governance (3 paragraphs, approximately 500 characters), a summary came back that accurately compressed the 3 key points: common principles of GDPR and Japan's Personal Information Protection Act, utilization of on-premises and region-specified cloud, and hybrid operations. The structure of the original text was preserved without information loss.

The spread of generative AI has made corporate data governance important,
with the handling of personal information in LLM training data becoming
a key challenge. International regulations on cross-border data transfers
(GDPR, Japanese law, etc.) share "transparency" as a common principle,
and on-premises or region-specified cloud usage is advancing to keep
domestic data within the country. Hybrid operations based on sensitivity
levels are attracting attention as a realistic solution.

For Japanese-to-English translation, I had it translate a business document about personal information handling ("Our company, in order to properly manage customer personal information…"). The result was natural English for a business document, such as "Our company processes customer personal information exclusively at our domestic data centers." Technical terms were handled appropriately, such as translating "閉域ネットワーク" (closed network) as "closed network environment." Translation completed in about 4.5 seconds.

Our company processes customer personal information exclusively at
our domestic data centers to ensure proper management. We do not
transfer any data to overseas servers, and security is maintained
through encrypted communication and a closed network environment.
If you have any questions or concerns, please contact the
Privacy Office.

Code Generation

"I want to aggregate this data a bit, but I always have to look up how to write pandas," "I want to quickly write SQL for these conditions" — this is the use case of having the LLM handle the repetitive script creation that arises in data analysis and report generation.

I requested "a Python script that reads customer data from CSV and aggregates the number of people and average purchase amount by age group." Practical code was generated using pandas, including age group classification with pd.cut(), aggregation with groupby().agg(), and error handling for FileNotFoundError and empty data. The Japanese comments were also natural. However, it took about 92 seconds, so there's a noticeable wait when the code volume is large.

# Script to perform aggregation by age group from customer data
# CSV file: requires format of name, age, purchase_amount
# Output: Displays number of people and average purchase amount per age group

import pandas as pd

def main():
    # 1. Specify the CSV file path
    file_path = 'customers.csv'

    try:
        # 2. Read CSV data with pandas
        df = pd.read_csv(file_path)

        # 3. Check if required columns exist (to prevent errors)
        required_columns = ['name', 'age', 'purchase_amount']
        if not all(col in df.columns for col in required_columns):
            missing = [col for col in required_columns if col not in df.columns]
            raise ValueError(f"Required columns are missing: {missing}")

        # 4. Verify that age is numeric
        if not pd.api.types.is_numeric_dtype(df['age']):
            raise TypeError("The 'age' column must be numeric type")

        if not pd.api.types.is_numeric_dtype(df['purchase_amount']):
            raise TypeError("The 'purchase_amount' column must be numeric type")

        # 5. Create age groups as a new column (20s, 30s...)
        df['age_group'] = pd.cut(
            df['age'],
            bins=[0, 19, 29, 39, 49, 100],
            labels=['Under 19', '20s', '30s', '40s', '50s and above'],
            right=False
        )

        # 6. Aggregate by age group: count and average purchase amount
        result = df.groupby('age_group').agg(
            Count=('name', 'count'),
            Average_Purchase=('purchase_amount', 'mean')
        ).reset_index()

        # 7. Display results (to 2 decimal places)
        print(result.to_string(
            index=False,
            float_format=lambda x: f"{x:.2f}"
        ))

    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
    except pd.errors.EmptyDataError:
        print("Error: CSV file is empty.")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

if __name__ == "__main__":
    main()

For SQL, I had it generate a query combining a JOIN of the orders table and customers table + WHERE conditions (date range and member rank filtering) + GROUP BY + ORDER BY.

SELECT c.都道府県, SUM(o.数量 * o.単価) AS 売上合計
FROM orders o
JOIN customers c ON o.顧客ID = c.顧客ID
WHERE YEAR(o.注文日) = 2026 AND MONTH(o.注文日) = 1
  AND c.会員ランク = 'ゴールド'
GROUP BY c.都道府県
ORDER BY 売上合計 DESC;

It handled Japanese column names directly, and the logical structure of the query is correct. It was generated in about 34 seconds.

Comparing Deployment Options

When actually using Nemotron 9B-v2-Japanese in business, where to run it becomes the next question. Here I'll organize 3 options.

For cases involving personal or confidential data, attention to cross-border data transfer may be required. SageMaker in the Tokyo Region allows configuring a closed network within a VPC, enabling a configuration where data never leaves Japan. Since Bedrock model availability was unconfirmed as of February 2026, SageMaker is the practical choice when a closed network configuration is definitely required.

DGX Spark supports fully local operation without network connectivity, making it suitable for prototyping in development stages or verification where you want to eliminate the risk of data leaking externally. On the other hand, since it's limited to the processing capacity of one machine, if production throughput requirements are strict, you may need to consider migrating to the cloud.

Summary

I ran NVIDIA's official Japanese-enhanced model Nemotron 9B-v2-Japanese on DGX Spark and tested it with benchmarks and practical use cases.

Summarizing impressions from the practical use cases: Tool Calling and JSON structured output are at a practical level, RAG was accurate including the "not stated" determination, and summarization/translation was of sufficient quality for business documents. Code generation also produced practical pandas and SQL scripts.

Within the scope of this verification, what I found particularly interesting was that running the same model on the same hardware, accuracy differed by 7.7 percentage points depending on the inference engine. Whether the Mamba-2 hybrid architecture tends to produce numerical processing differences between llama.cpp and vLLM, one possible distinction might be: Ollama (llama.cpp) if prioritizing accuracy, vLLM if prioritizing throughput.

The pip version of vLLM doesn't work on DGX Spark, but the NGC container version (26.01-py3) allows vLLM to be used. Initially this article stated that "vLLM didn't work," but not checking NGC was a reflection point. When trying to run something on DGX Spark, I should check the NGC catalog before pip packages.

Personally, this was an experience where "because I tried fine-tuning myself through trial and error, I can understand how impressive the official model is." Seeing the official release achieve +3.5% over the 4B model where I had squeezed out +0.6% through FT on 1,000 examples, I was reminded that professional work with both quality and quantity of data is really powerful.

On the other hand, when it comes to actually using it in business, "where to run it" is the next concern. I'd also like to try deployment to SageMaker Tokyo Region and domain-specific fine-tuning.

Bonus: Having Nemotron Review Its Own Article

Since the opportunity presented itself, I had Nemotron 9B-v2-Japanese itself read a summary of this article and asked for its thoughts.

It may be a bit too self-congratulatory, but the improvement suggestions pointing out "insufficient mention of inference speed" and "no comparison with other models" are quite on target. It was also an interesting experiment to see what an LLM says when reviewing itself.

Reference Links

• NVIDIA-Nemotron-Nano-9B-v2-Japanese (HuggingFace)
• Nemotron-H Technical Report (arXiv)
• NVIDIA Official Blog (Japanese)
• Nejumi Leaderboard 4
• Tried Japanese Fine-tuning Nemotron 3 Nano on DGX Spark
• NGC vLLM Container
• llama.cpp (GGUF conversion)