I tried teaching Classmethod's culture CLP to Nemotron 9B-v2

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Division.

In my previous article, I experimented with RAFT (Retrieval Augmented Fine Tuning) to inject knowledge from the National Tax Agency's FAQ data into Nemotron 9B to improve RAG accuracy. This time, I'm shifting direction slightly to try "value injection."

Classmethod has a code of conduct called CLP (Classmethod Leadership Principle), consisting of 10 items. Leadership, Partnership, Just Try It... these behavioral guidelines and values that every employee knows — can we teach them to an LLM as well?

https://classmethod.jp/company/culture/

I tried combining Constitutional AI (CAI) with SimPO (Simple Preference Optimization).

What is Constitutional AI

Constitutional AI is an alignment method announced by Anthropic in 2022. Its distinguishing feature is that instead of human feedback, the AI provides feedback to itself based on pre-defined "principles" (a Constitution).

Here's the rough flow:

1. Feed a prompt to the model and generate an initial response (R0)
2. Self-critique R0 against the principles of the Constitution
3. Generate an improved response (R1) based on the critique
4. Create a preference pair with R0 as rejected and R1 as chosen
5. Perform preference optimization (DPO or SimPO) on the preference pairs

The original paper uses a two-stage process: SFT phase (supervised learning on improved responses) → RL phase (PPO), but recently a configuration that skips SFT and goes directly to DPO or SimPO preference optimization is commonly used. I adopted this simplified configuration this time.

It's normally used to "reduce harmful responses," but if you change the content of the principles, it should also work for corporate codes of conduct.

Making CLP into a Constitution

The CLP consists of 10 items.

I convert these into "principles" for Constitutional AI. For each item, I defined a pair of critique prompts and revision prompts.

For example, for "Leadership," it works like this:

• The critique prompt asks: "Is this response passive and leaving things to others? Please check whether the content proactively shows actions and proposals."
• The revision prompt instructs: "Revise the response to proactively propose specific actions on your own initiative."

I prepared similar pairs for all 10 items and defined them as constitution.json.

Data Generation Pipeline

Data generation proceeds in 4 steps.

Generating Business Scene Prompts

First, I generated diverse business scenarios where CLP would be tested. Against a target of 300, I actually obtained 292.

I used the Claude Haiku API to generate the prompts.

Initial Response → Critique → Revision

For each of the 292 prompts, I executed the following process:

1. Have the plain Nemotron 9B v2 Japanese running on Ollama generate an initial response (R0) (average 1,394 characters)
2. Randomly select 2 principles from the 10 CLP items
3. Have Claude Haiku critique R0 (Self-Critique)
4. Generate an improved version based on the critique (Revision)

Since 2 principles are applied per prompt, 292 x 2 = 584 pairs are generated. After passing through a quality filter (excluding pairs where the difference between R0 and the improved version is too small), 0 pairs were filtered out, and all 584 chosen/rejected pairs became training data.

Training with SimPO

Why SimPO

DPO is the most well-known method for preference optimization, but I chose SimPO this time.

DPO requires retaining a reference model (a copy of the model before training) during training, using 18GB x 2 = 36GB of memory for Nemotron 9B in BF16. This fits within the DGX Spark's 128GB, but running simultaneous forward passes on 2 models with the Mamba-2 Hybrid architecture carries risks.

SimPO requires no reference model and can train with just a single model. It also reduces GPU memory consumption to about 60% compared to DPO. It can be used simply by specifying loss_type="simpo" in TRL's CPOTrainer, making implementation concise.

LoRA Constraints (53% Coverage)

As in my previous article, because Nemotron 9B uses the Mamba-2 Transformer Hybrid architecture, HF PEFT's LoRA can only be applied to the Attention + FFN layers (the in_proj / out_proj of Mamba-2 layers are not supported). LoRA coverage is 53%.

Honestly, I wasn't sure whether "value alignment" would work at 53% coverage until I tried it.

Running Training

I ran SimPO training on the NGC NeMo 25.11.01 container.

# Run inside NGC container
python n5-simpo-train.py train \
  --data-file ./data/n5/train.jsonl \
  --output-dir ./data/n5/adapter \
  --method simpo

The loss started at 1.6017, dropped to 1.5273, then stabilized around 1.57. SimPO loss doesn't drop as dramatically as DPO, so this progression was as expected.

GGUF Conversion and Ollama Registration

I converted the trained LoRA adapter to GGUF format using llama.cpp and registered it with Ollama.

# LoRA → GGUF conversion (36.1 MB, 132 tensors)
python convert_lora_to_gguf.py ./data/n5/adapter \
  --base-model-id nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
  --outtype f16

# Register with Ollama
ollama create nemotron-9b-jp-cai -f Modelfile

The Modelfile is just 2 lines: base model + ADAPTER.

FROM nemotron-9b-jp-nothink
ADAPTER ./nemotron-9b-cai-lora.gguf

Evaluation Results

CLP Compliance (LLM-as-Judge)

I prepared 5 test prompts per CLP principle, 50 prompts in total. Model responses are scored by Claude Haiku on a 1-5 scale. To account for variability in LLM-as-Judge evaluation, I ran it twice under identical conditions and used the average.

The baseline already scores very high at 4.76/5.0. This shows that the plain Nemotron 9B v2 Japanese is already capable of providing high-quality responses to business scene questions.

The CAI model improved Partnership from 4.80 → 5.00. On the other hand, Feedback declined from 4.70 → 4.30. Overall, it was nearly flat at -0.01.

JCQ Regression Check

I checked for regression in general performance using JCommonsenseQA (1,119 questions).

The regression was -0.4pp, within the acceptable range (within -2pp). This confirms that SimPO preference optimization barely impairs general Japanese language ability.

Qualitative Comparison

Let me compare actual responses side by side. I'll cover 4 scenarios.

Across all 4 scenarios, the CAI model showed a tendency to mention "consideration for others" and "building trust relationships" at an earlier stage in the responses. The +0.20 improvement in Partnership score is thought to reflect this change. In the "Just Try It" scenario, there are also signs that the model has learned the practical side of CLP, proposing concrete actions before abstract analysis.

Discussion

Does value alignment work with 53% LoRA?

To put it plainly, my honest impression is "it partially worked."

Partnership showed a consistent +0.20 improvement in both evaluations, and the qualitative change of mentioning "consideration for others" earlier is stable. On the other hand, Feedback worsened by -0.40. Under the constraint of 53% LoRA coverage, the model can only learn from the behavior of Attention and FFN layers. Since the long-term context processing handled by Mamba-2 layers cannot be changed, the effect may have been concentrated on some CLP principles within the training scope.

The fact that "Partnership" showed a change with the small dataset of 584 pairs is itself an interesting result. If the LoRA coverage rate could be increased, more uniform improvement might be expected.

Cost-effectiveness of Constitutional AI + SimPO

Here is a summary of the costs for the entire pipeline.

API costs are very inexpensive at $1.72. The time bottleneck is R0 generation (dependent on Ollama inference speed), which takes approximately 80 seconds per item on the DGX Spark's GB10 GPU. Since there's ample GPU memory headroom, switching to an inference server like vLLM would likely enable further speedups.

Possibilities and Limitations of "Internalizing Corporate Culture into LLMs"

A baseline of 4.76/5.0 means that the plain Nemotron 9B v2 Japanese has already sufficiently learned business etiquette and communication best practices.

In other words, the aspects of "corporate culture" that overlap with general business etiquette are already present in the LLM, leaving little room for improvement through Constitutional AI. On the other hand, improvement was seen in areas with company-specific nuances, such as "Partnership."

There is also a structural challenge in that codes of conduct like CLP are highly abstract and therefore difficult to reflect in LLM responses. With more concrete norms (for example, "in code reviews, always mention at least one positive point before giving feedback"), the effect of CAI might become clearer.

Summary

In my previous article I tried "knowledge injection" (RAFT), and this time I tried "value injection" (Constitutional AI + SimPO).

"Knowledge injection" tends to yield clear improvements, whereas "value injection" is more susceptible to ceiling effects the higher the base model's capabilities — that difference has become apparent. That said, it's an interesting result that Partnership showed a consistent change across 2 evaluations after just ~40 minutes of training.

This was an experiment under the constraint of 53% LoRA, but as an attempt to internalize company-specific codes of conduct into an LLM, the Constitutional AI approach still seems worth continuing to explore.

The scripts and data used this time are published on GitHub.

https://github.com/himorishige/dgx-spark-blog/tree/main/n5-constitutional-ai-simpo

Reference Links

• Constitutional AI: Harmlessness from AI Feedback (Anthropic original paper)
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• DPO: Direct Preference Optimization
• TRL CPOTrainer Documentation
• Constitutional AI with Open LLMs (HuggingFace Blog)
• NVIDIA Nemotron Nano 9B v2 Japanese
• Classmethod Leadership Principle