I attended NVIDIA Japan's NPN Partner Model Customization Bootcamp
This page has been translated by machine translation. View original
Introduction
Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.
On August 5, 2026, I attended the "Japan NVIDIA NPN Partner Model Customization Bootcamp" hosted by NVIDIA Japan and Macnica. Following the Agentic AI Bootcamp in June, this was the second hands-on session for NPN (NVIDIA Partner Network) partners, with model customization as the theme. It was an even more in-depth experience than the previous one, covering synthetic data generation and reinforcement learning in a single day to train a small model for agent use.
The report from the previous session is here (article from June 2026). At the end of that article I wrote that "a sequel focusing on reinforcement learning had been announced," and that sequel is this session.
The slides and teaching materials repository from this session are not public, but the reference implementation that formed the basis of the hands-on is publicly available on GitHub as NVIDIA GenerativeAIExamples. Reading it alongside this article should allow you to reconstruct the flow of the day quite well.
This article summarizes the day's agenda, the key points covered in each session, and the actual measurements and data pitfalls I observed firsthand, as a participation report. I hope it resonates with those who have "seen GRPO by name in papers but never actually run it themselves."
In a Nutshell
The structure was a one-day pipeline of three steps — synthetic data generation → reinforcement learning → execution — to build an agent that receives natural language instructions and outputs LangGraph CLI commands as JSON. Data was machine-generated with NeMo Data Designer (500 samples), rewards were machine-scored by a NeMo Gym validation server, and training was run by TRL's GRPOTrainer. In other words, it was a full end-to-end experience of a Reinforcement Learning from Verifiable Rewards (RLVR) pipeline, with no human annotation and no human evaluation.
If the previous Agentic AI Bootcamp was the "building an agent" session, this one was the "training the model inside the agent" session. The choice of subject matter was also bold — rather than SFT, the go-to for fine-tuning introductions that even beginners can pick up quickly, it tackled GRPO head-on, a topic few have hands-on experience with. The base model, Qwen3-0.6B-Base, is a raw pre-trained model with no SFT at all, and to borrow the instructor's words, we were making it able to output JSON "from the same state as DeepSeek-R1-Zero," using only rewards.
Event Overview
The event was structured in two parts: Day 0 (online Dry Run) and Day 1 (in-person main session).
- Date: August 5, 2026, 9:30–19:00 (JST); Day 0 was held online on August 3
- Venue: Macnica Shinagawa Office
- Target participants: NVIDIA NPN partner employees
- Organizers: NVIDIA Japan + Macnica + OpenACC organization
The reference implementation that formed the basis of the hands-on is here.
The execution environment was a shared GPU cluster distributed in advance; the flow was to SSH in, run a shell script to launch a JupyterLab container, and connect via browser using port forwarding. The GPUs were A100s. As a side note, the venue network had port 22 (SSH) blocked and required explicitly specifying port 443, and almost all connection issues people ran into seemed to stem from forgetting to specify this. Multiple TAs were on hand, and issues like this were resolved quickly.
Day's Agenda
The main session agenda was roughly as follows.
| Time | Session |
|---|---|
| 10:00 - 10:30 | Introduction and Cluster setup |
| 10:30 - 11:00 | Tutorial 1: Overview & Synthetic Data Generation |
| 11:15 - 12:15 | Lab 1: Synthetic Data Generation |
| 13:15 - 14:15 | Tutorial 2: Introduction to GRPO |
| 14:30 - 15:30 | Lab 2: GRPO |
| 15:45 - 16:45 | Lab 3: Run Bash Agent |
| 16:45 - 17:00 | Wrap up |
Connecting the three Labs forms the following pipeline.
From here, I'll walk through what was covered in each session and the key points I observed.
Tutorial 1: Getting the Difference Between SFT and RL Straight
The first lecture was a review of LLM fundamentals. It started with a rundown that the architecture has barely changed since the 2017 Transformer — it's just multiple stacked blocks of Attention and MLP — and that the current mainstream is replacing the MLP portion, which accounts for most of the parameters, with MoE (Mixture of Experts). The treatment of reasoning models was equally blunt: "They're just trained to output <think> tags; the structure is the same. The output tokens are just longer." That kind of no-nonsense explanation was refreshing to hear.
The main topic was the three stages of training — pre-training, SFT, and RL. As the instructor summarized it: SFT "increases the likelihood of correct answers in the dataset; the training data is fixed," while RL "gives rewards to outputs generated by the model itself; the training data is sampled from the model itself." This difference in "where the training data comes from" turned out to be foreshadowing that would matter in the later Labs.
The contextual explanation was also interesting: OpenAI o1's arrival opened a sudden gap between closed and open models, then DeepSeek-R1 appeared and published — hyperparameters included — that reasoning could be acquired through RL, allowing open models to catch up. The GRPO we'd be doing today is exactly that DeepSeek-R1 algorithm. A show-of-hands survey of the audience revealed that many had SFT experience, while only a few had RL experience — a distribution that made the bootcamp's target audience very clear.
Lab 1: Running Synthetic Data Generation with NeMo Data Designer
Lab 1 was the part where we create training data. Using a synthetic data generation library called NeMo Data Designer, we generated 500 pairs of Japanese instruction sentences for LangGraph CLI and their corresponding correct JSON tool calls.
The generation model was a Nemotron-3-Nano series 30B reasoning model served locally with vLLM and registered with Data Designer as an OpenAI-compatible endpoint. Using a reasoning model for data generation causes thinking traces from <think> to leak into the generated text, so disabling thinking with enable_thinking: False was the first key point.
A key design decision was not having the LLM create the correct labels. Command names and port numbers were first assigned mechanically by a sampler.
# Source: Bootcamp material 01_synthetic_data_generation_ja.ipynb (excerpt)
config_builder.add_column(SamplerColumnConfig(name="command", sampler_type=SamplerType.CATEGORY,
params=CategorySamplerParams(values=["new", "dev", "up", "build", "dockerfile"])))
config_builder.add_column(SamplerColumnConfig(name="port", sampler_type=SamplerType.UNIFORM,
params=UniformSamplerParams(low=3000, high=9000), convert_to="int"))
The only thing the LLM generates is the natural language instruction with those values embedded; the correct JSON is assembled deterministically from the sampler values. Having the LLM write labels risks errors, but with this design the labels are always correct. Post-generation validation only needs to check that the sampler values appear verbatim in the instruction and verify required arguments. In my run, 500 records were generated and 459 passed validation — a yield of about 92%. Generated records looked like this:
{
"input": "Please start the server container while monitoring changes on port 8807.",
"output": {
"command": "up",
"template": null,
"path": null,
"port": 8807,
"no_browser": null,
"watch": true,
"tag": null,
"output_path": null
}
}
On the other hand, I also noticed a pitfall when I looked through the generated data afterward. Lines like this were mixed in:
{
"input": "dev 서버をポート 3989 で起動してください。",
"output": {
"command": "dev",
"template": null,
"path": null,
"port": 3989,
"no_browser": null,
"watch": null,
"tag": null,
"output_path": null
}
}
Korean has crept into what should be a Japanese instruction. When I counted, about 16% of the 413 training records contained Hangul, and almost all of them were for the dev command. It looks like the generation model was slipping into Korean in contexts mixing the English word "dev server." The validation filter only checked for value inclusion and arguments, with no language check, so these slipped through. The fact that the generation prompts were written in English also seems to be a contributing factor. SDG is not done just because generation is complete and the count is met — there needs to be a phase where you actually read through the contents. That was the biggest lesson from Lab 1.
Tutorial 2: Deriving the GRPO Formula from REINFORCE
The afternoon lecture was the highlight of the bootcamp. It walked through deriving the GRPO objective function step by step, starting from REINFORCE and passing through PPO. The instructor opened with "sorry for all the math," but having this one hour meant that every hyperparameter in the subsequent Lab became readable with meaning — personally, I think this was the most valuable session of the day.
At a high level, the progression was as follows. REINFORCE, dating back to around 1992, is a simple policy gradient that "reinforces the generation path taken if the reward is positive, and weakens it if negative," with high reward variance that makes training unstable. From there, subtracting a baseline from the reward was introduced to reduce variance, and using a state value function as the baseline led to PPO. However, PPO requires an additional model to predict value. GRPO's approach was to "throw out the value model entirely." It generates 8 or 16 responses to the same prompt all at once, and uses the Z-score normalized by the mean and standard deviation of rewards within the group as the advantage. Because it works with group statistics, no additional model is needed.
The instructor's summary was excellent: you want to maximize advantage, but you also want KL regularization to prevent drifting too far from the original model — that is "the spirit of GRPO." After working through the derivation, the fact that placing PPO and GRPO side by side shows the only difference is the definition of advantage also sinks in smoothly.
The approach to rewards was also laid out as a contrast between RLHF and RLVR. RLHF uses a reward model trained on human preferences, so it cannot assign rewards in domains the reward model wasn't trained on. RLVR judges correctness mechanically with code, so the same output always returns the same reward, keeping training stable. The current state of the art, we were told, is using both: "teach verifiable tasks like math and coding with RLVR, then use RLHF at the end to align with human preferences." During Q&A, there was also discussion of how "training itself is now easy to run, but reward construction is very hard and reward hacking appears quickly," as well as the operational point that code execution for verification runs on CPU rather than GPU, making CPU increasingly important for RL — the kind of practical intuition you wouldn't get from just reading papers.
Lab 2: Applying GRPO to Qwen3-0.6B-Base
Now for the reinforcement learning itself. The training target was Qwen3-0.6B-Base with no SFT at all, and we would teach it JSON tool calls using only rewards. Rewards were returned by a FastAPI validation endpoint implemented as a NeMo Gym resource server. The scoring criteria were as follows:
| Condition | Reward |
|---|---|
| JSON cannot be parsed | -1.0 |
| Command mismatch | -1.0 |
| Command matches, flags partially match | 0.0 ~ +1.0 |
| Exact match | +1.0 |
| Format bonus (JSON parseable) | +0.3 |
Training settings were passed directly to TRL's GRPOTrainer. The "group" from the lecture corresponds to num_generations.
# Source: Bootcamp material 02_grpo_training_ja.ipynb (excerpt)
training_args = GRPOConfig(
temperature=1.0,
num_generations=8, # Number of generations per prompt = group size
learning_rate=1e-5,
per_device_train_batch_size=48, # 6 prompts x 8 generations
max_steps=60,
bf16=True,
)
True to the instructor's words — "don't look at the loss at all; what you should look at is the reward" — the GRPO loss is a policy gradient objective value, so it doesn't decrease monotonically and can also go negative. In my run, average reward went from -0.869 in the first 5 steps to +0.324 in the last 5 steps. 60 steps on a single A100 took 11 minutes and 34 seconds. For roughly the first 26 steps there was a stagnation period where reward was pinned near -1, and looking at the reward server logs, the model was outputting garbled text mixing Chinese and Korean. Around step 27, when something resembling JSON began to appear, there was a sudden breakthrough and it rose steadily from there. The +0.3 format bonus appears to function as a "cheap partial reward" ladder for escaping this stagnation. Even in the early stages when no exact matches occur, just outputting the JSON shape creates a reward differential, which raises the within-group standard deviation and generates a learning signal.
Another highlight of this Lab was the reveal of the gap between the lecture content and the implementation. The Lab 2 configuration uses mostly the GRPOTrainer defaults as-is: num_iterations=1 means there's no old policy, so the probability ratio is always 1 and clipping is effectively disabled; furthermore beta=0 eliminates the KL regularization term. In other words, the default GRPOTrainer runs in a bare form where, of all the terms carefully derived in the lecture, only advantage remains. These defaults apparently differ across frameworks, and having this reveal immediately after working through the math cultivates the habit of "reading framework defaults with skepticism."
As a minor stumbling point: if the vLLM spun up in Lab 1 still holds GPU memory, you'll get an OOM when GRPO training starts. This happened frequently in the venue; the fix was to run the shutdown cell from Lab 1 or identify the relevant process with nvidia-smi and kill it.
Lab 3: Running the Trained Agent
The final part was running the model with the trained LoRA adapter loaded as an agent. It takes Japanese instructions, converts them to JSON tool calls, passes them through a command allowlist check, asks for human confirmation before execution, and then actually runs the CLI. On my machine, giving the instruction "create a new project using the agent-python template" triggered langgraph new and actually generated a directory.
What I personally liked most about the implementation was how system prompts were handled. The training notebook was designed to import the prompt from the inference agent's config file, structurally guaranteeing that the same string is used for both training and inference.
# Source: Bootcamp material 02_grpo_training_ja.ipynb (excerpt)
# To use the identical prompt for training and inference, load json_system_prompt
# from config.py as a single source of truth
# (any mismatch causes small models to collapse)
sys.path.insert(0, "bash_agent")
from config import Config
SYSTEM_PROMPT = Config().json_system_prompt
Small models in the 0.6B class collapse in output if there's even a slight mismatch between training and inference prompts, so this "single source of truth" design is a pattern I'd want to replicate in real work. The safety mechanisms were also solid for teaching material: rejecting commands containing backticks or $, splitting the command and checking the leading token against an allowlist, and finally a y/N human confirmation — a genuinely sound multi-layered defense.
At the same time, the limitations of the small model were also observable. When given instructions without specifying a path, the model fills in a path like app/agent by drawing on the training data distribution. Since REQUIRED fields in the schema must be filled, the model simply invents values. Responses after receiving tool execution results also collapsed into parroting or garbled text. Since only single-turn JSON generation was trained, multi-turn is out of distribution. It was an educationally instructive way to break — a clear demonstration that what you trained is exactly what defines the boundaries of behavior.
Trying to Induce Reasoning Acquisition in Free Time
After Lab 3 came free experimentation time. The instructor's recommended challenge was "try adding a reward for <think> tags and see if you can build a reasoning model," and I gave it a shot. I added +0.5 reward to outputs containing <think> and ran it, but the result was zero activations across 2,880 rollouts. Prompting the model to think didn't change anything.
Thinking about why: this brings us back to Tutorial 1's point that "RL training data is sampled from the model itself." Because the <think> format barely exists in the pre-training distribution of Qwen3-0.6B-Base, it was never sampled during exploration, and rewards cannot reinforce actions that are never sampled. I came to understand through the defeat of running out of time exactly why DeepSeek-R1-Zero had enforced <think> via a template. The plan for a rematch — seeding the distribution with few-shot examples or template enforcement before amplifying with rewards — is something I intend to test on my own DGX Spark.
Points That Left an Impression
Through participation, there were four points that crystallized for me.
The first is the RL principle that rewards only act on actions that are sampled. The contrast from the lecture — "SFT has fixed data, RL samples from the model itself" — became genuinely internalized through the first-hand experience of zero activations during free time. If you want to teach a new output format, you need to seed the distribution before placing rewards.
The second is the reveal that GRPOTrainer's defaults are a bare form with only advantage remaining. Clipping and KL, both present in the paper's formulas, are gone under the default settings. Since defaults differ across frameworks, when running GRPO you should confirm "which terms are currently active" before touching hyperparameters.
The third is the difficulty of reward design. The instructor's words — "training itself is now easy to run, but reward construction is very hard" — carry different weight after seeing both sides: the format bonus functioning as a ladder for cold-start, and also being a potential entry point for reward hacking if made too large, since rewards would concentrate on outputs that merely look correctly formatted. I'd want to unit-test reward functions against actual generated outputs before running them.
The fourth is the operational point that CPU is becoming more important in RL. The "verification" in verifiable rewards is code execution, which runs on CPU, not GPU. The NeMo Gym reward server in this session ran entirely on CPU. When estimating RL infrastructure, counting only GPUs could leave you blindsided.
Looking at these together, none of them are about flashy accuracy improvements — they're all about the messy parts: rewards not going up, strange languages mixed into data, GPU memory not being released. But when model customization comes up as an option in real projects, these are exactly the things that matter. I think the structure of the bootcamp — not just walking through a clean success path, but building in the sticking points and design intuitions as well — was valuable. And choosing to commit to GRPO for a full day rather than defaulting to SFT, which is easier to get working even for beginners, might be an expression of the intent to have participants "take home the messy parts too."
The venue atmosphere was similar to last time, with multi-person teams from various companies progressing while asking each other questions. Just as the instructor noted in Q&A that "RL requires many experimental iterations," this is a domain where you run, adjust settings, run again, and repeat — it doesn't resolve in one shot. The fact that the reward curves looked completely different between adjacent seats despite using the same materials was a scene unique to this subject matter.
Summary
The Model Customization Bootcamp was a hands-on session that let you experience the full pipeline — from synthetic data generation through reinforcement learning with verifiable rewards to agent execution — in a single day. Having the derivation from REINFORCE to GRPO and the reveal of TRL's default values as a paired package meant we were brought not to "running GRPO somehow" but to "running it while understanding which terms are active" — and that was enormously valuable. At the same time, challenges that were only visible because I was hands-on — Korean language contamination, zero activations for <think> — were also things I could take home.
Reference Links
- NVIDIA GenerativeAIExamples - Bash Computer Use Agent — Reference implementation for this hands-on session
- NeMo Data Designer
- NeMo Gym
- TRL GRPOTrainer
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Original paper introducing GRPO)
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Previous participation report
Code snippets in this article are cited in brief excerpts from bootcamp materials, with attribution. The materials are based on NVIDIA GenerativeAIExamples' bash_computer_use_agent (Apache License 2.0).
