I tried teaching Classmethod's culture CLP to Nemotron 9B-v2

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

In my previous article, I used RAFT (Retrieval Augmented Fine Tuning) to inject National Tax Agency FAQ data into Nemotron 9B to improve RAG accuracy — a "knowledge injection" experiment. This time, I'm shifting direction slightly to try "value injection."

Classmethod has a set of 10 behavioral principles called CLP (Classmethod Leadership Principles): Leadership, Partnership, Just Try It... Every employee knows these guidelines and values. Can we teach them to an LLM as well?

https://classmethod.jp/company/culture/

I tried combining Constitutional AI (CAI) and SimPO (Simple Preference Optimization).

What is Constitutional AI

Constitutional AI is an alignment method announced by Anthropic in 2022. Its key feature is having the AI itself provide feedback based on pre-defined "principles" (a Constitution), rather than relying on human feedback.

Here's a rough overview of the process:

1. Feed the model a prompt and generate an initial response (R0)
2. Self-critique R0 against the principles in the Constitution
3. Generate an improved response (R1) based on the critique
4. Create a preference pair with R0 as rejected and R1 as chosen
5. Perform preference optimization (DPO or SimPO) using the preference pairs

The original paper uses a two-phase approach: SFT phase (supervised learning with improved responses) → RL phase (PPO), but recently a simplified configuration that skips SFT and goes directly to DPO or SimPO preference optimization seems to be widely used. I adopted this simplified configuration this time as well.

It's typically used to "reduce harmful responses," but by changing the content of the principles, it should also work with corporate codes of conduct.

Turning CLP into a Constitution

CLP consists of 10 principles.

I converted these into "principles" for Constitutional AI. For each item, I defined a pair of prompts: one for critique (Self-Critique) and one for revision.

For example, in the case of "Leadership":

• The critique prompt asks: "Is this response passive and leaving things to the other party? Please check whether it presents content that proactively takes action and makes proposals."
• The revision prompt instructs: "Revise the response to make it content that proactively proposes specific actions."

I prepared similar pairs for all 10 principles and defined them as constitution.json.

Data Generation Pipeline

Data generation proceeds in 4 steps.

Generating Business Scene Prompts

First, I generated diverse business scenarios that would put CLP to the test. Against a target of 300, I actually obtained 292.

I used the Claude Haiku API to generate the prompts.

Initial Response → Critique → Revision

For each of the 292 prompts, I performed the following processing:

1. Have the base Nemotron 9B v2 Japanese running on Ollama generate an initial response (R0) (average 1,394 characters)
2. Randomly select 2 principles from the 10 CLP items
3. Have Claude Haiku critique R0 (Self-Critique)
4. Generate a revised version based on the critique (Revision)

Since 2 principles are applied per prompt, 292 x 2 = 584 pairs are generated. After passing through a quality filter (excluding cases where the difference between R0 and the improved version is too small), 0 pairs were filtered out, and all 584 chosen/rejected pairs became training data.

Training with SimPO

Why SimPO

DPO is the most well-known method for preference optimization, but I chose SimPO this time.

DPO requires maintaining a reference model (a copy of the model before training) during training, which uses 18GB x 2 = 36GB of memory with Nemotron 9B in BF16. It fits within the DGX Spark's 128GB, but running forward passes simultaneously with 2 models on the Mamba-2 Hybrid architecture carries risks.

SimPO doesn't require a reference model and can train with just a single model. GPU memory consumption is about 60% compared to DPO. It can be used simply by specifying loss_type="simpo" in TRL's CPOTrainer, making the implementation concise.

LoRA Constraints (53% Coverage)

As in the previous article, because Nemotron 9B uses the Mamba-2 Transformer Hybrid architecture, HF PEFT's LoRA can only be applied to Attention + FFN layers (the in_proj / out_proj of the Mamba-2 layers are not supported). LoRA coverage is 53%.

Honestly, whether "value alignment" would work with 53% coverage was something I couldn't know without trying.

Running Training

I ran SimPO training on the NGC NeMo 25.11.01 container.

# Run inside NGC container
python n5-simpo-train.py train \
  --data-file ./data/n5/train.jsonl \
  --output-dir ./data/n5/adapter \
  --method simpo

Loss started at 1.6017, dropped to 1.5273, then stabilized around 1.57. SimPO loss doesn't drop as dramatically as DPO, so this trend was as expected.

GGUF Conversion and Ollama Registration

I convert the trained LoRA adapter to GGUF format using llama.cpp and register it in Ollama.

# LoRA → GGUF conversion (36.1 MB, 132 tensors)
python convert_lora_to_gguf.py ./data/n5/adapter \
  --base-model-id nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese \
  --outtype f16

# Register with Ollama
ollama create nemotron-9b-jp-cai -f Modelfile

The Modelfile is just 2 lines: base model + ADAPTER.

FROM nemotron-9b-jp-nothink
ADAPTER ./nemotron-9b-cai-lora.gguf

Evaluation Results

CLP Compliance (LLM-as-Judge)

I prepared 5 test prompts for each of the CLP principles, 50 prompts in total. Model responses are scored by Claude Haiku on a scale of 1-5. To account for variability in LLM-as-Judge evaluation, I ran the evaluation twice under identical conditions and used the average.

The baseline already scores very high at 4.76/5.0. This shows that the base Nemotron 9B v2 Japanese can already provide high-quality responses to business-scene questions.

In the CAI model, Partnership improved from 4.80 to 5.00. On the other hand, Feedback declined from 4.70 to 4.30. Overall, it's essentially flat at -0.01.

JCQ Regression Check

I checked for regression in general performance using JCommonsenseQA (1,119 questions).

The regression is -0.4pp, within the acceptable range (within -2pp). It was confirmed that SimPO preference optimization barely impairs general Japanese language ability.

Qualitative Comparison

Let me compare actual responses side by side. I'll cover 4 scenarios.

Across all 4 scenarios, the CAI model showed a tendency to mention "consideration for the other party" and "building trust" earlier in the response. It can be considered that the +0.20 improvement in Partnership score reflects this change. In the "Just Try It" scenario, there's a sign that the model has also learned the practical aspect of CLP, proposing concrete actions before abstract analysis.

Discussion

Does Value Alignment Work with 53% LoRA?

To give the conclusion upfront: "it partially worked" is my honest impression.

Partnership showed a consistent +0.20 improvement in both evaluations, and qualitatively, the change to mentioning "consideration for the other party" first has been stable. On the other hand, Feedback worsened by -0.40. Under the constraint of 53% LoRA coverage, the model can only learn the behavior of Attention and FFN layers. Since the long-term context processing handled by the Mamba-2 layers cannot be modified, there's a possibility that the effect was biased toward some of the CLP principles in the training targets.

The fact that a change in "Partnership" emerged with the small-scale data of 584 pairs was itself an interesting result. If LoRA coverage could be increased, more uniform improvement might be expected.

Cost-Effectiveness of Constitutional AI + SimPO

Here's a summary of the costs for the entire pipeline.

API costs are very low at $1.72. The time bottleneck is R0 generation (dependent on Ollama inference speed), which takes about 80 seconds per item on the DGX Spark's GB10 GPU. There's room in GPU memory, so switching to an inference server like vLLM should enable further speedup.

Possibilities and Limitations of "Internalizing Corporate Culture into LLMs"

A baseline of 4.76/5.0 means that the base Nemotron 9B v2 Japanese has already sufficiently learned business etiquette and communication best practices.

In other words, the parts of "corporate culture" that overlap with general business etiquette are already built into the LLM, leaving little room for improvement through Constitutional AI. On the other hand, improvement was seen in areas with company-specific nuances, such as "Partnership."

There is also a structural challenge that codes of conduct like CLP are highly abstract, making them difficult to reflect in LLM responses. For more concrete norms (for example, "always mention at least one positive point before giving feedback in a code review"), the effect of CAI might become clearer.

Summary

In the previous article I tried "knowledge injection" (RAFT), and this time I tried "value injection" (Constitutional AI + SimPO).

While "knowledge injection" tends to produce clear improvements, "value injection" reveals that the higher the base model's capability, the more likely a ceiling effect occurs. That said, it's an interesting result that consistent change in Partnership appeared across 2 evaluations with just ~40 minutes of training.

This was an experiment under the constraint of 53% LoRA, but as an attempt to internalize company-specific codes of conduct into an LLM, the Constitutional AI approach seems worth continuing to explore.

The scripts and data used this time are published on GitHub.

https://github.com/himorishige/dgx-spark-blog/tree/main/n5-constitutional-ai-simpo

Reference Links

• Constitutional AI: Harmlessness from AI Feedback (Anthropic original paper)
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• DPO: Direct Preference Optimization
• TRL CPOTrainer Documentation
• Constitutional AI with Open LLMs (HuggingFace Blog)
• NVIDIA Nemotron Nano 9B v2 Japanese
• Classmethod Leadership Principle