I tried fine-tuning Nemotron 3 Nano in Japanese on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In my previous article "A Summary of the Local LLM Landscape in 2026," I wrote that Nemotron 3 Nano had "no Japanese benchmark results," but after trying it on DGX Spark, I found it performed quite well even out of the box. That got me curious, so I decided to try fine-tuning it on Japanese using NVIDIA's publicly released synthetic persona dataset Nemotron-Personas-Japan.

I'll say upfront: I naively assumed that training on persona dialogue data would also improve Japanese benchmarks, but the capabilities measured by the training data (descriptions of Japanese people's lives and occupations) and the evaluation benchmarks (multiple-choice general knowledge questions) are quite different. I'll discuss this miscalculation in detail later in the article, but since the results weren't catastrophic, I'm sharing the whole trial-and-error process.

This article summarizes the results of QLoRA fine-tuning Nemotron 3 Nano on Japanese data on DGX Spark, comparing performance before and after both quantitatively and qualitatively.

What is Nemotron 3 Nano?

Nemotron 3 Nano is a MoE (Mixture of Experts) model released by NVIDIA in December 2025. Of its 31.6B total parameters, only 3.6B are active at any given time.

The architecture is interesting: roughly 92% of its 52 layers use the Mamba-2 State Space Model, while the remaining 8% use Transformer self-attention — a hybrid configuration (the nemotron_h architecture). Mamba-2 can process long contexts with constant computation, enabling a maximum context length of 1 million tokens. Each layer has 128 routed experts, with only 6 activated per token.

It also pairs well with DGX Spark. In a unified memory environment, MoE models incur no PCIe transfer penalty, so expert switching runs smoothly. With Q4 quantization, it fits in about 24GB — roughly 1/5 of the 128GB memory — leaving plenty of room for fine-tuning.

The license is the NVIDIA Open Model License, which permits commercial use, modification, and redistribution.

Baseline Evaluation

First, let's check the model's Japanese language performance out of the box.

Measuring Commonsense Reasoning with JCommonsenseQA

On DGX Spark, I evaluated all 1,119 questions from the JCommonsenseQA v1.3 validation set using 3-shot prompting. Here's a comparison with other models:

Achieving 92.5% with only 3.6B active parameters comes very close to Gemma 3, a 27B Dense model. Even out of the box, it seems sufficiently practical for Japanese commonsense reasoning.

Test Environment

Hardware

Software

I used NVIDIA's official PyTorch container as the base and installed the necessary libraries for fine-tuning.

Since Unsloth does not support the nemotron_h architecture, I used HuggingFace PEFT + TRL directly.

Fine-Tuning Preparation

Docker Environment Setup

Since DGX Spark is an aarch64 environment, packages that only have x86_64 wheels can be troublesome. Using NVIDIA's official PyTorch container avoids CUDA compatibility issues.

# Pull the NVIDIA PyTorch container
docker pull nvcr.io/nvidia/pytorch:25.11-py3

# Create working directory
mkdir -p ~/nemotron-ft && cd ~/nemotron-ft

# Start container (GPU access + working directory mount)
docker run -it --gpus all \
  -v $(pwd):/workspace/nemotron-ft \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --shm-size=16g \
  --name nemotron-ft \
  nvcr.io/nvidia/pytorch:25.11-py3

Install the fine-tuning libraries inside the container.

pip install peft trl bitsandbytes datasets hf_transfer accelerate

Since the NVIDIA PyTorch container on DGX Spark already comes with PyTorch optimized for aarch64, only a few additional packages need to be installed.

Exploring the Nemotron-Personas-Japan Dataset

The Nemotron-Personas-Japan dataset published by NVIDIA is a synthetic persona dataset based on Japan's census and labor statistics.

Generated with NeMo Data Designer, the dataset includes diverse personas reflecting Japan's demographics — age, gender, region, occupation, education level, and more. It is fully synthetic for privacy protection and contains no PII (Personally Identifiable Information).

Let's start by examining the data.

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-Personas-Japan", split="train")
print(f"Total records: {len(ds):,}")  # 1,000,000
print(f"Columns: {ds.column_names}")

Each record has 22 columns, consisting of 6 persona fields (professional_persona, sports_persona, arts_persona, travel_persona, culinary_persona, persona) and metadata such as age, occupation, and prefecture. Each persona field contains Japanese text describing a person's background — for example, a career history or thoughts about local food culture.

Converting to SFT Format

We convert the persona data into instruction-following QA pairs.

def make_professional_prompt(example):
    """Generate career-related QA pairs"""
    loc = example.get("prefecture", "Japan")
    occ = example.get("occupation", "office worker")
    user_msg = (
        f"We'd like to ask someone working as {occ} in {loc}. "
        f"Could you tell us about your career and work experience?"
    )
    return user_msg, example["professional_persona"]

I prepared 4 conversion templates and applied them in round-robin fashion to a subset of 1,000 records.

Here is one sample after conversion:

[USER]
We'd like to ask someone working as a mid-level construction industry professional in Saga Prefecture.
Could you tell us about your career and work experience?

[ASSISTANT]
Sosuke Ohata is a mid-level manager who prioritizes on-site safety and schedule transparency,
combining simplified CAD design with mobile log data visualization
to encourage his team toward systematic, reliable execution....

One characteristic of the persona data is that responses are written in the third person. First person would be more natural for SFT data, but since the purpose here was to verify "how much effect a small dataset of 1,000 samples can have," I kept the conversion logic simple.

Running Fine-Tuning

Loading the Model and Configuring LoRA

Using HuggingFace PEFT, we load Nemotron 3 Nano with 4-bit (NF4) quantization and add a LoRA adapter.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    quantization_config=bnb_config,
    device_map={"": 0},
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

The key point here is the LoRA target modules. Since Nemotron 3 Nano is a hybrid of Transformer and Mamba-2 layers, I targeted the projection layers of both.

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=[
        # Transformer self-attention
        "q_proj", "k_proj", "v_proj", "o_proj",
        # MoE feed-forward
        "gate_proj", "up_proj", "down_proj",
        # Mamba-2 projections
        "in_proj", "out_proj",
    ],
    lora_dropout=0,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

Let's check the LoRA adapter configuration.

Only 1.38% of the parameters are trained. The key refinement here is including not just the Transformer layers, but also the in_proj and out_proj of the Mamba-2 layers. Since Mamba layers make up 48 of the 52 total layers, ignoring them would significantly limit the scope of learning.

Running Training

We run 1 epoch of training using TRL's SFTTrainer.

from trl import SFTConfig, SFTTrainer

sft_config = SFTConfig(
    output_dir="./outputs/nemotron-ft-1k",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=10,
    num_train_epochs=1,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_steps=100,
    optim="adamw_8bit",
    max_length=2048,
    packing=False,
)

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset,
    args=sft_config,
)

trainer.train()

Training Results

Looking at the loss curve, it dropped sharply in the first 30 steps (15.78 → 6.65) and then gradually converged — a typical curve. This shows that even with just 1,000 samples, the model was absorbing Japanese language patterns.

Evaluation Pipeline

Converting to GGUF and Loading into Ollama

One concern here is the inference method.

Using HuggingFace's model.generate() is the simplest approach, but the HuggingFace implementation of Nemotron 3 Nano (modeling_nemotron_h.py) has several bugs in the forward pass, making it impossible to obtain correct output.

I therefore switched to reusing Ollama (ggml backend), which I had used for the baseline evaluation. llama.cpp natively supports NemotronHForCausalLM, and this implementation runs stably.

To use the LoRA adapter with Ollama, convert it to GGUF format using the following steps:

# Clone the llama.cpp repository
git clone --depth 1 https://github.com/ggml-org/llama.cpp /tmp/llama-cpp

# Prepare the Python environment for conversion
uv venv /tmp/gguf-env
source /tmp/gguf-env/bin/activate
uv pip install numpy sentencepiece gguf safetensors protobuf \
  transformers torch --index-url https://download.pytorch.org/whl/cpu

# Convert LoRA adapter to GGUF
python /tmp/llama-cpp/convert_lora_to_gguf.py \
  ~/nemotron-ft/outputs/nemotron-ft-1k/lora/ \
  --outfile /tmp/nemotron-ft-1k-lora.gguf \
  --outtype f32

The converted LoRA GGUF file was 1.77GB, containing 324 tensors (LoRA A/B pairs for each module).

Next, we create an inference model in Ollama combining the base model and LoRA adapter.

FROM nemotron-3-nano:latest
ADAPTER /tmp/nemotron-ft-1k-lora.gguf
PARAMETER num_ctx 4096
PARAMETER temperature 0

ollama create nemotron-ft-1k -f Modelfile

Post-Fine-Tuning Evaluation

Before/After Comparison on JCommonsenseQA

I evaluated the fine-tuned model under the same conditions as the baseline (3-shot, all 1,119 questions).

Even with QLoRA using just 1,000 persona samples, we saw a +1.1% improvement.

Comparison with Other Models

Here's a comparison against the major models introduced in the local LLM article.

The fine-tuned model's 93.6% is just 0.3% behind Gemma 3 27B (93.9%), a 27B Dense model. It's quite interesting to get this close with only 3.6B active parameters. However, as discussed below, the benchmark and training data aren't well-aligned, so it's more appropriate to view this score as confirmation that "FT didn't break anything" rather than placing too much weight on the number itself.

Qualitative Evaluation: Observing Changes in Japanese

Since numbers alone don't tell the whole story, I compared outputs by sending the same prompts to both the pre- and post-FT models.

Natural Japanese

Prompt: "Please suggest a travel plan from Tokyo to Osaka."

The pre-FT (BASE) output begins with "Below, we'll use two representative transportation options, 'Shinkansen' and 'Highway Bus,' as a base..." — a slightly stiff style. The post-FT output opens more naturally with "I'll introduce several routes and highlights to suit your travel purpose and available time." The comparison table of transportation options also felt more readable in the post-FT version, with clearer "pros/cons" contrasts.

Formal Language and Business Documents

Prompt: "Please write an email to your supervisor requesting a schedule change for next week's meeting."

This is where a clear difference emerged. The pre-FT output includes the English word scheduled mixed into the Japanese body text. The post-FT output is entirely in Japanese, and was formatted as a complete business email including a signature block (phone number, email address, and postal address). Japanese business customs present in the persona data appear to have been reflected.

Japanese Cultural Knowledge

Prompt: "Please explain Japanese New Year (Oshogatsu) traditions to a foreign friend."

Both the pre- and post-FT outputs covered major New Year events (year-end cleaning, kagami mochi, hatsumode, etc.), but the style of explanation differs. The pre-FT output organizes everything into a large table listing "what to do" and "why it's important." The post-FT output follows a chronological flow — "year-end cleaning → preparing kagami mochi → osechi cuisine" — which is easier for a foreign friend to follow. However, the post-FT output incorrectly refers to Japanese New Year as "Seollal" (the Korean Lunar New Year), revealing the limits of knowledge gained from persona data.

Overall, the post-FT model tended to produce more natural Japanese, with less English mixing in, and outputs more aligned with Japanese business customs. Personally, I was surprised to see this much change from just 1,000 persona samples.

Alignment Between Training Data and Benchmark

As mentioned at the start, I naively assumed that training on persona dialogue data would improve Japanese benchmarks across the board. In reality, this assumption was too optimistic — JCommonsenseQA is not a particularly appropriate benchmark for measuring the effects of this fine-tuning.

The training data consists of texts about the lives and occupations of Japanese people, such as "the career of a mid-level construction manager in Saga Prefecture" or "a persona about Shizuoka Prefecture's food culture." JCommonsenseQA, on the other hand, tests general knowledge through multiple-choice questions like "What is the title of the supreme commander of an occupied territory? → Governor-General" or "What are rare earth elements called? → Rare earth elements." The domains of knowledge being tested are quite different. Ideally, the evaluation metric should have been decided based on the training data before starting.

The +1.1% improvement is likely not a direct result of persona data strengthening commonsense reasoning, but rather an indirect effect of the additional Japanese text training broadening reading comprehension and vocabulary coverage. The difference of +12 out of 1,119 questions is borderline in terms of statistical significance, so it's best to avoid over-interpreting the score.

The changes observed in the qualitative evaluation — "English mixing in business emails disappeared" and "introductory sentences became more natural" — are more straightforward evidence of the fine-tuning effect from persona data. It's more appropriate to view JCommonsenseQA as a check that "commonsense reasoning ability wasn't broken by fine-tuning."

Pitfalls Encountered

There were several unexpected issues during this experiment. I'm documenting them here in hopes they'll be useful to anyone attempting the same thing.

Unsloth Does Not Support Nemotron 3 Nano

DGX Spark Playbooks introduces a QLoRA procedure using Unsloth, but Nemotron 3 Nano's nemotron_h architecture (Mamba-2 + Transformer hybrid MoE) is not on Unsloth's supported model list. FalconH1, another hybrid model, is supported by Unsloth, but its internal layer structure is different enough that it couldn't be reused.

I ended up switching to using HuggingFace PEFT + TRL directly, which doesn't require much more code. It's unfortunate to miss out on Unsloth's optimizations (improved memory efficiency and training speed), but with DGX Spark's 128GB unified memory, bare HuggingFace didn't run into out-of-memory issues.

There are Bugs in the HuggingFace Nemotron 3 Nano Inference Implementation

This was the most painful part of the whole process. Training completes normally with transformers' Trainer, but calling model.generate() for inference produces broken output.

Digging into the cause, I found multiple bugs in HuggingFace's modeling_nemotron_h.py (the file implementing the forward pass for Nemotron 3 Nano). There are issues with the state management of Mamba-2 layers and the routing logic of MoE, which prevent correct logits from being obtained even in a single forward pass.

Training itself was running fine since the loss was decreasing, but the broken inference meant I got stuck at the seemingly basic step of "checking the output of the trained model." I didn't realize this at first and wasted a lot of time repeatedly reviewing the LoRA configuration and dataset.

merge_and_unload() Corrupts the Weights

Since HuggingFace inference wasn't working, I thought of converting to GGUF and evaluating with Ollama — but there was a trap here too.

The typical LoRA-to-GGUF conversion flow is "merge the adapter into the base model with merge_and_unload() → convert to GGUF," but with 4-bit QLoRA, the dequantization from NF4 to BF16 that merge_and_unload() performs causes significant precision loss, making the weights unusable. The conversion completes without errors, but the output becomes a string of commas and unrelated words.

The solution is to use llama.cpp's convert_lora_to_gguf.py to convert the LoRA adapter to GGUF separately, then apply it to the base model using Ollama's ADAPTER directive. This approach leaves the base model weights completely untouched, so no precision degradation from quantization occurs.

It took me a full day to get to this point...

Summary

Leveraging DGX Spark's 128GB unified memory, I was able to complete Japanese fine-tuning of Nemotron 3 Nano (31.6B MoE) entirely on a local machine.

Here is a summary of the results:

With QLoRA on just 1,000 persona samples, the qualitative evaluation clearly showed a reduction in English mixing and an improvement in business document quality. JCommonsenseQA also showed a +1.1% improvement, but I believe this is more of an indirect effect from an overall boost in Japanese language processing ability, rather than a direct result of the persona data.

On the other hand, the Nemotron 3 Nano ecosystem is still maturing. Between Unsloth not being supported, bugs in HuggingFace's inference implementation, and the merge_and_unload() trap, the situation was very much "training works, but building the evaluation pipeline is a struggle." Since the llama.cpp (Ollama) implementation is stable, I think going through GGUF for evaluation and inference is the practical solution for now.

Given how much changed with just 1,000 samples, I'm curious what would happen if trained on the full 1 million-record dataset. Also, as a lesson learned, if the goal is to measure the conversational quality and naturalness of Japanese as trained through persona data, I should have incorporated a dialogue quality evaluation like Japanese MT-Bench from the start, rather than a knowledge-based benchmark like JCommonsenseQA — even though it's more costly...

I hope this is helpful for anyone looking to try the same thing on DGX Spark.

Reference Links

• Nemotron 3 Nano Official Blog
• Nemotron 3 Nano (HuggingFace)
• Nemotron-Personas-Japan (HuggingFace)
• DGX Spark Playbooks - Unsloth
• DGX Spark Playbooks - PyTorch Fine-Tune
• llama.cpp (GGUF conversion)
• HuggingFace PEFT
• A Summary of the Local LLM Landscape in 2026