I tried fine-tuning Nemotron 3 Nano in Japanese on DGX Spark

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In my previous article "A Look at the State of Local LLMs in 2026," I wrote that Nemotron 3 Nano had "no Japanese benchmark results," but after trying it on a DGX Spark, I found it was quite excellent even in its raw state. That got me curious, and I decided to try fine-tuning it for Japanese using the NVIDIA-published synthetic persona dataset Nemotron-Personas-Japan.

To be upfront about this: I naively assumed that training on persona dialogue data would improve Japanese benchmark scores, but the capabilities measured by the training data (descriptions of Japanese people's lives and occupations) and the evaluation benchmarks (multiple-choice general knowledge questions) are quite different. I'll go into more detail about this mismatch in the latter half of the article, but since things didn't completely fall apart, I'm sharing the process including the trial and error.

This article summarizes the results of QLoRA fine-tuning Nemotron 3 Nano on Japanese data on a DGX Spark and comparing performance before and after, both quantitatively and qualitatively.

What is Nemotron 3 Nano?

Nemotron 3 Nano is a MoE (Mixture of Experts) model released by NVIDIA in December 2025. Of its 31.6B total parameters, only 3.6B are actually active at any time.

The architecture is interesting: of 52 layers, approximately 92% are Mamba-2 State Space Models and the remaining 8% are Transformer self-attention mechanisms, making it a hybrid architecture (nemotron_h architecture). Mamba-2 can process long contexts with constant computation, enabling support for context lengths of up to 1 million tokens. Each layer has 128 routed experts, with only 6 activated per token.

It also works well with DGX Spark. In a unified memory environment, MoE models don't incur PCIe transfer penalties, so expert switching runs smoothly. At Q4 quantization it uses about 24GB, roughly 1/5 of the 128GB memory, leaving plenty of room for fine-tuning.

The license is the NVIDIA Open Model License, which permits commercial use, modification, and redistribution.

Baseline Evaluation

First, let's confirm the Japanese language performance of the raw model.

Measuring Commonsense Reasoning with JCommonsenseQA

I evaluated all 1,119 questions from the JCommonsenseQA v1.3 validation set on DGX Spark using 3-shot prompting. Here is a comparison with other models.

Achieving 92.5% with only 3.6B active parameters is a score that rivals the 27B Dense Gemma 3. Even in its raw state, it seems sufficiently practical for Japanese commonsense reasoning.

Verification Environment

Hardware

Software

Using NVIDIA's official PyTorch container as the base, I installed the necessary libraries for fine-tuning.

Since Unsloth doesn't support the nemotron_h architecture, I used HuggingFace PEFT + TRL directly.

Preparing for Fine-Tuning

Setting Up the Docker Environment

Since DGX Spark is an aarch64 environment, packages that only have x86_64 wheels can be a struggle. Using NVIDIA's official PyTorch container helps avoid CUDA compatibility issues.

# Pull the NVIDIA PyTorch container
docker pull nvcr.io/nvidia/pytorch:25.11-py3

# Create working directory
mkdir -p ~/nemotron-ft && cd ~/nemotron-ft

# Start container (GPU access + working directory mount)
docker run -it --gpus all \
  -v $(pwd):/workspace/nemotron-ft \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --shm-size=16g \
  --name nemotron-ft \
  nvcr.io/nvidia/pytorch:25.11-py3

Install the fine-tuning libraries inside the container.

pip install peft trl bitsandbytes datasets hf_transfer accelerate

The NVIDIA PyTorch container on DGX Spark already has PyTorch optimized for aarch64 installed, so fewer additional packages need to be installed.

Exploring the Nemotron-Personas-Japan Dataset

Nemotron-Personas-Japan, published by NVIDIA, is a synthetic persona dataset based on Japan's national census and labor statistics.

Generated with NeMo Data Designer, it contains diverse personas reflecting Japan's demographics including age, gender, region, occupation, and education level. It is entirely synthetic data for privacy protection and contains no PII (personally identifiable information).

Let's first take a look at the data contents.

from datasets import load_dataset

ds = load_dataset("nvidia/Nemotron-Personas-Japan", split="train")
print(f"Total records: {len(ds):,}")  # 1,000,000
print(f"Columns: {ds.column_names}")

Each record has 22 columns, consisting of 6 types of persona fields (professional_persona, sports_persona, arts_persona, travel_persona, culinary_persona, persona) and metadata such as age, occupation, and prefecture. Each persona field contains Japanese text describing a person's background, such as their career history or attachment to local food culture.

Converting to SFT Format

We convert the persona data into instruction-following format QA pairs.

def make_professional_prompt(example):
    """Generate career-related QA pairs"""
    loc = example.get("prefecture", "Japan")
    occ = example.get("occupation", "office worker")
    user_msg = (
        f"I'd like to ask someone working as a {occ} in {loc}. "
        f"Could you tell me about your career and work experience?"
    )
    return user_msg, example["professional_persona"]

I prepared 4 conversion templates and applied them in round-robin to a subset of 1,000 records.

Here is one converted sample.

[USER]
I'd like to ask someone working as a mid-level construction worker in Saga Prefecture.
Could you tell me about your career and work experience?

[ASSISTANT]
Sosuke Ohata is a mid-level manager who prioritizes on-site safety and process transparency,
combining simple CAD design with mobile log data visualization
to encourage his team toward planned and reliable execution...

A characteristic of persona data is that responses are written in the third person. First person would be more natural for SFT data, but since the goal this time was to verify "how effective a small dataset of 1,000 items can be," I kept the conversion logic simple.

Executing Fine-Tuning

Loading the Model and Configuring LoRA

We load Nemotron 3 Nano with 4-bit (NF4) quantization using HuggingFace PEFT and add a LoRA adapter.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    quantization_config=bnb_config,
    device_map={"": 0},
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

The key point this time is the LoRA target modules. Since Nemotron 3 Nano is a hybrid of Transformer and Mamba-2 layers, we target the projection layers of both.

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=[
        # Transformer self-attention
        "q_proj", "k_proj", "v_proj", "o_proj",
        # MoE feed-forward
        "gate_proj", "up_proj", "down_proj",
        # Mamba-2 projections
        "in_proj", "out_proj",
    ],
    lora_dropout=0,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

Let's check the LoRA adapter settings.

Only 1.38% of the total parameters are trained, but the key innovation this time was including not just Transformer layers but also Mamba-2 layers' in_proj and out_proj. Since Mamba layers account for 48 of the 52 total layers, ignoring them would significantly limit the scope of learning.

Running the Training

We run 1 epoch of training using TRL's SFTTrainer.

from trl import SFTConfig, SFTTrainer

sft_config = SFTConfig(
    output_dir="./outputs/nemotron-ft-1k",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=10,
    num_train_epochs=1,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_steps=100,
    optim="adamw_8bit",
    max_length=2048,
    packing=False,
)

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset,
    args=sft_config,
)

trainer.train()

Training Results

Looking at the loss curve, it dropped sharply in the first 30 steps (15.78 → 6.65) and then gradually converged — a typical curve. This shows that even with a small dataset of 1,000 items, the model is absorbing Japanese language patterns.

Evaluation Pipeline

Converting to GGUF and Loading into Ollama

One thing to consider here is the inference method.

Using HuggingFace's model.generate() is the simplest approach, but the HuggingFace implementation of Nemotron 3 Nano (modeling_nemotron_h.py) has several bugs in the forward pass, making it impossible to get correct output.

So I switched to reusing Ollama (ggml backend) that I had used for the baseline evaluation. llama.cpp natively supports NemotronHForCausalLM, and this implementation works stably.

To use the LoRA adapter with Ollama, convert it to GGUF format using the following steps.

# Clone the llama.cpp repository
git clone --depth 1 https://github.com/ggml-org/llama.cpp /tmp/llama-cpp

# Prepare Python environment for conversion
uv venv /tmp/gguf-env
source /tmp/gguf-env/bin/activate
uv pip install numpy sentencepiece gguf safetensors protobuf \
  transformers torch --index-url https://download.pytorch.org/whl/cpu

# Convert LoRA adapter to GGUF
python /tmp/llama-cpp/convert_lora_to_gguf.py \
  ~/nemotron-ft/outputs/nemotron-ft-1k/lora/ \
  --outfile /tmp/nemotron-ft-1k-lora.gguf \
  --outtype f32

The converted LoRA GGUF file was 1.77GB. It contains 324 tensors (LoRA A/B pairs for each module).

Next, we create an inference model in Ollama that combines the base model + LoRA adapter.

FROM nemotron-3-nano:latest
ADAPTER /tmp/nemotron-ft-1k-lora.gguf
PARAMETER num_ctx 4096
PARAMETER temperature 0

ollama create nemotron-ft-1k -f Modelfile

Post-Fine-Tuning Evaluation

Before/After Comparison on JCommonsenseQA

I evaluated the fine-tuned model under the same conditions as the baseline (3-shot, all 1,119 questions).

Even with QLoRA using just 1,000 persona data items, we saw a +1.1% improvement.

Comparison with Other Models

Let's line it up against the major models introduced in the local LLM article.

The post-FT score of 93.6% is only 0.3% behind Gemma 3 27B (93.9%), a 27B Dense model. It's interesting how close it gets with only 3.6B active parameters. However, as I'll explain below, the compatibility between this benchmark and the training data isn't great, so it's more appropriate to view the score as confirming "FT didn't break anything" rather than placing too much weight on the number itself.

Observing Japanese Language Changes Through Qualitative Evaluation

Since numbers alone don't tell the whole story, I compared outputs by feeding the same prompts to the models before and after FT.

Natural Japanese

Prompt: "Please suggest a travel plan from Tokyo to Osaka"

The pre-FT (BASE) response opens with "Below, we'll use two representative transportation options, 'Shinkansen' and 'highway bus,' as the basis for..." in a somewhat stiff style. Post-FT, it begins with "I'll introduce some routes and highlights based on your travel purpose and how much time you have," which is a more natural introduction. The transportation comparison table in the post-FT version also had clearer "pros/cons" contrasts and felt easier to read.

Honorifics and Business Documents

Prompt: "Please write an email to your boss requesting a schedule change for next week's meeting"

This is where a difference emerged. In the pre-FT version, English text such as scheduled（スケジュール） appears mixed into the body of the message. Post-FT, everything is unified in Japanese, and it outputs a complete business email format including a signature block (phone number, email address, address). It appears that Japanese business customs contained in the persona data are being reflected.

Japanese Cultural Knowledge

Prompt: "Please explain how Japanese people spend New Year's to a foreign friend"

Both pre-FT and post-FT covered the major New Year's events (year-end cleaning, kagami mochi, hatsumode, etc.), but the style of explanation differs. Pre-FT presents a large table listing "what to do" and "why it's important." Post-FT explains in chronological order: "year-end cleaning → preparing kagami mochi → osechi cuisine," making it easier to follow for foreigners. However, the post-FT output incorrectly refers to New Year's as "Seollal" (Korea's Lunar New Year), revealing the limits of the knowledge acquired through persona data.

Overall, post-FT outputs tended to be more natural in Japanese, with less English mixed in and outputs more aligned with Japanese business customs. Personally, this degree of change from just 1,000 persona data items exceeded my expectations.

Compatibility Between Training Data and Benchmarks

As I mentioned at the beginning, I started this assuming that training on persona dialogue data would improve Japanese benchmarks across the board. In reality, this expectation was overly optimistic — JCommonsenseQA is not a particularly appropriate benchmark for measuring the effects of this fine-tuning.

The training data consists of texts like "career description of a mid-level construction manager in Saga Prefecture" and "persona related to food culture in Shizuoka Prefecture." JCommonsenseQA, on the other hand, consists of multiple-choice questions on general knowledge such as "What do you call the supreme commander of an occupied territory? → Governor-General" and "What are rare earth elements called? → Lanthanides" — quite different domains of knowledge. Ideally, the evaluation metric should be decided first, aligned with the training data, before getting started.

The +1.1% improvement is likely not because the persona data directly reinforced commonsense reasoning, but rather an indirect effect of additional training on Japanese text broadening reading comprehension and vocabulary coverage. Since a difference of +12 questions out of 1,119 is borderline in terms of statistical significance, it seems best to avoid placing too much emphasis on the score.

The changes visible in qualitative evaluation — "English mixed into business emails disappeared" and "introductory sentences became more natural" — are a more direct reflection of the FT effects from persona data. It's more appropriate to view JCommonsenseQA as a confirmation that "FT did not destroy commonsense reasoning ability."

Pitfalls Encountered

There were several unexpected issues during this verification. I'm summarizing them here in the hope that they will be useful for others trying the same thing.

Unsloth Does Not Support Nemotron 3 Nano

While DGX Spark Playbooks introduces QLoRA procedures using Unsloth, Nemotron 3 Nano's nemotron_h architecture (Mamba-2 + Transformer hybrid MoE) is not included in Unsloth's list of supported models. Although FalconH1, a similar hybrid model, is supported by Unsloth, the internal layer structure is different so it couldn't be reused.

I ended up switching to using HuggingFace PEFT + TRL directly, which isn't much more code. It's a shame that Unsloth's optimizations (improved memory efficiency and training speed) aren't available, but with the 128GB unified memory of DGX Spark, there were no memory issues even with plain HuggingFace.

There Are Bugs in the HuggingFace Nemotron 3 Nano Inference Implementation

This was the most challenging part of the entire process. Training completes normally with transformers' Trainer, but calling model.generate() during inference produces corrupted output.

Tracing the cause, I found multiple bugs in HuggingFace's modeling_nemotron_h.py (the file implementing the forward pass for Nemotron 3 Nano). There are issues with the state management of Mamba-2 layers and the routing logic of MoE, and even a single forward pass doesn't yield correct logits.

Training itself works fine since the loss is decreasing, but inference is broken, causing a stumble at the very basic step of "checking the output of the trained model." I initially didn't realize this and wasted a lot of time reviewing the LoRA configuration and dataset over and over.

Weights Get Corrupted with merge_and_unload()

Since inference with HuggingFace wasn't working, I thought of converting to GGUF and evaluating with Ollama, but there was another trap here.

The typical LoRA GGUF conversion flow is "merge adapter into base model with merge_and_unload() → convert to GGUF," but with 4-bit QLoRA, merge_and_unload() causes significant precision degradation during the NF4 → BF16 dequantization process, making the weights unusable. The conversion completes normally, but the output becomes a stream of commas and unrelated words.

The solution is to individually convert the LoRA adapter to GGUF using llama.cpp's convert_lora_to_gguf.py and apply it to the base model using Ollama's ADAPTER directive. This approach doesn't touch the base model's weights at all, so no quantization-induced precision degradation occurs.

It took me a full day to get here...

Summary

By leveraging the 128GB unified memory of DGX Spark, I was able to complete Japanese fine-tuning of Nemotron 3 Nano (31.6B MoE) entirely on a local machine.

Here is a summary of this verification's results.

With just 1,000 persona data items and QLoRA, the qualitative evaluation clearly showed reductions in English mixing and improvements in business document quality. JCommonsenseQA also showed a +1.1% improvement, but I believe this is an indirect effect of improved Japanese language processing capability rather than a direct effect of the persona data.

On the other hand, the Nemotron 3 Nano ecosystem is still maturing. With Unsloth incompatibility, bugs in the HuggingFace inference implementation, and the merge_and_unload() trap, it was a situation of "training works but building an evaluation pipeline is a struggle." Since the llama.cpp (Ollama) implementation is stable, performing evaluation and inference via GGUF seems like the practical solution for now.

Since this much change was visible with 1,000 items, I'm curious what would happen with training on the full dataset of 1 million items. Also, as a lesson learned from this time, if I'm evaluating conversation quality and natural Japanese that persona data helps develop, I should have incorporated a dialogue quality evaluation like Japanese MT-Bench from the start, rather than a knowledge benchmark like JCommonsenseQA — even though it costs more...

I hope this is useful for those who want to try the same thing on DGX Spark.

Reference Links

• Nemotron 3 Nano Official Blog
• Nemotron 3 Nano (HuggingFace)
• Nemotron-Personas-Japan (HuggingFace)
• DGX Spark Playbooks - Unsloth
• DGX Spark Playbooks - PyTorch Fine-Tune
• llama.cpp (GGUF conversion)
• HuggingFace PEFT
• A Look at the State of Local LLMs in 2026